SlowGuess/NeurIPS_2023_OCR huggingface dataset:
At online iteration  $n$ , an intuitive way to solve the bi-level optimization problem (3)-(4) is to solve it in a double-loop way [9, 26, 28]. In specific, the double-loop method first solves the inner-level problem (4) to find a close approximation of  $\omega^{*}(\theta)$  and then uses the obtained result to get the gradient (5), thus solving the outer-level problem (3). However, it requires multiple gradient descent for the inner-level problem (4) to get  $\omega^{*}(\theta)$  before updating the decision variable of the outer-level problem where each gradient descent of the inner-level problem needs to solve a constrained RL problem in our case. Therefore, the double-loop method is not suitable to be an online algorithm when the data is revealed in a fast speed. To design an algorithm suitable for the online fashion, we adopt the spirit of [29, 30] and solve the outer-level and inner-level problems in a single loop. However, our method is not a simple extension of [29, 30] to online settings because they are centralized and require the inner objective function to be strongly convex but our problem does not have these properties.
In specific, at each online iteration, the decision variables of both the outer-level and inner-level problems update only once where the gradient of the inner objective function is given in Lemma 3. For the outer-level problem, as  $\omega^{*}(\theta)$  is inaccessible, we cannot get the exact gradient defined in
(5). Therefore, we propose the following gradient approximation whose derivation can be found in Appendix:  $\bar{\nabla} L(\theta ,\omega) = N_{L}E_{S,A}^{\pi_{\omega ;\theta}}[\sum_{t = 0}^{T}\gamma^{t}\nabla_{\theta}r_{\theta}(S_{t},A_{t})] - \sum_{v = 1}^{N_{L}}\nabla_{\theta}\hat{J}_{r_{\theta}}(\zeta^{[v]}),$  where  $\pi_{\omega ;\theta}$  is the constrained soft Bellman policy with parameters  $(\omega ,\theta)$ . Compared to (5), this gradient approximation does not include the second-order term  $M(\theta ,\omega)$  and thus is much more computationally efficient. It is shown in Appendix that the approximation error  $||\nabla L(\theta ,\omega^{*}(\theta)) - \bar{\nabla} L(\theta ,\omega)||\leq C_{\theta}||\omega^{*}(\theta) - \omega ||$  where  $C_\theta$  is a positive constant whose expression is in Appendix. Intuitively, as the inner decision variable  $\omega$  approaches  $\omega^{*}(\theta)$  in the learning process, this gradient approximation error can be sufficiently small.
Lemma 3. The gradient of  $G^{[v]}(\omega; \theta)$  is  $\mu(\pi_{\omega; \theta}) - \hat{\mu}(\zeta^{[v]})$  where  $\pi_{\omega; \theta}$  is the constrained soft Bellman policy with parameters  $(\theta, \omega)$ .
Lemma 4. The global likelihood function  $L(\theta, \omega)$  is  $C_L$ -lipschitz continuous and  $\bar{C}_L$ -smooth in  $(\theta, \omega)$ , where  $C_L$  and  $\bar{C}_L$  are positive constants.
4.2 Consensus-based distributed learning
In our distributed learning setting, each learner  $v$  only knows its local information (e.g.,  $L^{[v]}$  and  $G^{[v]}$ ), so that it cannot directly solve the problem (3)-(4). Therefore, each learner  $v$  updates the decision variables using the gradients of its local outer-level and inner-level objective functions and uses communications to collaboratively solve the problem (3)-(4). Similar to the global gradient approximation  $\bar{\nabla} L(\theta, \omega)$ , learner  $v$  has its local gradient approximation:  $\bar{\nabla} L^{[v]}(\theta, \omega) = E_{S,A}^{\pi_{\omega; \theta}}[\sum_{t=0}^{T} \gamma^{t} \nabla_{\theta} r_{\theta}(S_{t}, A_{t})] - \nabla_{\theta} \hat{J}_{r_{\theta}}(\zeta^{[v]})$ . Learner  $v$  uses samples to estimate  $\bar{\nabla} L^{[v]}(\theta, \omega)$  via  $\hat{\nabla} L^{[v]}(\theta, \omega) \triangleq \frac{1}{m^{[v]}} \sum_{j=1}^{m^{[v]}} \nabla_{\theta} \hat{J}_{r_{\theta}}(\zeta^{j}) - \nabla_{\theta} \hat{J}_{r_{\theta}}(\zeta^{[v]})$  where  $\zeta^{j}$  is generated by rolling out  $\pi_{\omega; \theta}$  and  $m^{[v]}$  is the number of the samples. Moreover, learner  $v$  also uses the same samples to estimate  $\nabla G^{[v]}(\omega; \theta)$  in Lemma 3 via  $\hat{\nabla} G^{[v]}(\omega; \theta) \triangleq \frac{1}{m^{[v]}} \sum_{j=1}^{m^{[v]}} \hat{\mu}(\zeta^{j}) - \hat{\mu}(\zeta^{[v]})$ .
Algorithm 1 Multi-agent behavior inference from distributed and streaming demonstrations
Input:  $\{\theta^{[v]}(1)\}_{v = 1}^{N_L},\{\omega^{[v]}(1)\}_{v = 1}^{N_L},W(n)$    
Output:  $\theta^{[v]}(N),\omega^{[v]}(N),\pi_{\omega^{[v]}(N);\theta^{[v]}(N)},\quad \forall v\in \mathcal{V}$    
1: for  $n = 1,\dots ,N$  do   
2: for  $v\in \mathcal{V}$  do   
3: Finds policy  $\pi_{\omega^{[v]}(n);\theta^{[v]}(n)}$  using soft Q-learning [41] or soft actor-critic [42].   
4: Simulates samples  $\{\zeta^j\}_{j = 1}^{m^{[v]}}$  using  $\pi_{\omega^{[v]}(n);\theta^{[v]}(n)}$    
5: Receives  $\theta^{[v']}(n),\omega^{[v']}(n)$  from the neighbors  $v^{\prime}\in \mathcal{N}^{[v]}(n)$  and observes  $\zeta^{[v]}(n)$    
6:  $\omega^{[v]}(n + 1) = \sum_{v' = 1}^{N_L}W^{[vv']}(n)\omega^{[v']}(n) - \frac{\alpha(n)}{l}\sum_{i = 0}^{l - 1}\hat{\nabla} G^{[v]}(\omega^{[v]}(n);\theta^{[v]}(n),n - i)$    
7:  $\theta^{[v]}(n + 1) = \sum_{v' = 1}^{N_L}W^{[vv']}(n)\theta^{[v']}(n) - \frac{\beta(n)}{l}\sum_{i = 0}^{l - 1}\hat{\nabla} L^{[v]}(\theta^{[v]}(n),\omega^{[v]}(n),n - i)$    
8: end for   
9: end for
In Algorithm 1, at online iteration  $n$ , each learner  $v$  sequentially executes the following two steps: (i) uses current reward and cost function parameters to get the corresponding policy and generate samples (Lines 3-4); (ii) communicates with neighbors and updates its parameters for both outer-level and inner-level problems in a single loop using the samples generated in last step (Lines 5-7). In the update process (Lines 6-7), the first term (convex combination) encourages consensus among different learners and the second term (gradient) drives to the set of stationary points. Notice that even though soft Q-learning and soft actor-critic are designed for unconstrained RL, as shown in Appendix, we can revise them to approximate the constrained soft Bellman policy.
Theorem 1. Suppose Assumptions 1, 2, 3 hold. Let the step sizes  $\alpha(n) = \frac{\bar{\alpha}}{n^{\eta_1}}$  and  $\beta(n) = \frac{\bar{\beta}}{n^{\eta_2}}$  where  $\eta_1, \eta_2 \in (\frac{1}{2}, 1)$  and  $\bar{\alpha}, \bar{\beta} \in (0, \frac{1}{N_L(2\bar{C}_L + 1)})$ , then for any  $v, v' \in \mathcal{V}$  in Algorithm 1:
(consensus):  $||\theta^{[v]}(N) - \theta^{[v']}(N)|| \leq O(\frac{1}{N^{\eta_1}} + \bar{\epsilon}^N)$ ,  $||\omega^{[v]}(N) - \omega^{[v']}(N)|| \leq O(\frac{1}{N^{\eta_2}} + \bar{\epsilon}^N)$ , and  $\sup_{(s,a) \in S \times \mathcal{A}} \left\{ |\pi_{\omega^{[v]}(N); \theta^{[v]}(N)}(a|s) - \pi_{\omega^{[v']} (N); \theta^{[v']} (N)}(a|s)| \right\} \leq O\left(\frac{1}{N^{\eta_1}} + \frac{1}{N^{\eta_2}} + \bar{\epsilon}^N\right)$ , where  $\sup_{(s,a) \in S \times \mathcal{A}} \{\cdot\}$  outputs the supremum value over  $S \times \mathcal{A}$  and the expression of  $\bar{\epsilon} \in (0,1)$  can be
found in Appendix.
$$
\begin{array}{l l} (d e c r e a s i n g a v e r a g e l o c a l r e g r e t): & \frac {1}{N} \sum_ {n = 1} ^ {N} E \big [ \frac {1}{l} | | \sum_ {i = 0} ^ {l - 1} \nabla L (\theta^ {[ v ]} (n), \omega^ {[ v ]} (n), n - i) | | ^ {2} \big ] \leq \\ O \big (\frac {1}{N ^ {1 - \eta_ {1}}} + \frac {1}{N ^ {1 - \eta_ {2}}} + \frac {1}{N} \big) + \frac {2 (C _ {L}) ^ {2}}{l}. \end{array}
$$
(sub-linear cumulative constraint violation):  $\sum_{n=1}^{N} E\left[J_{c_E}^2\left(\pi_{\omega^{[v]}(n); \theta^{[v]}(n)}\right)\right] \leq O(N^{\eta_2} + 1)$ , where the expectation is taken over randomly demonstrated trajectories at each online iteration.
Theorem 1 shows that the learners will achieve consensus on the reward and cost function parameters, and policy. The average local regret decreases to an upper bound  $2(C_L)^2 /l$ . Notice that this upper bound has similar property with that in [22, 23]: (i) they are both less than the uniform upper bound  $C_L^2$  of the global loss function  $L$  when  $l > 2$ ; (ii) they are both diminishing if  $l$  is dependent on  $n$  and  $l(n) = \omega (1)$  [23]. The number of total outer gradient steps taken by each learner in Algorithm 1 is  $O(N)$  which is smaller than  $O(2Nl + l^2)$  in [22] and  $O(Nl^2)$  in [23]. The reason of fewer gradient steps in Algorithm 1 is that it is an online gradient descent algorithm, which does not require multiple gradient calls at each online iteration as follow-the-leader-based methods [22, 23] do.
Moreover, if the reward functions are linear as in [20, 21], we have the following stronger result:
Corollary 1. If the reward functions are linear, the average difference of cumulative reward between the learned policy and expert policy diminishes:  $\frac{1}{N}\sum_{n = 1}^{N}E\big[(J_{r_E}(\pi_{\omega^{[v]}(n)};\theta^{[v]}(n)) - J_{r_E}(\pi_E))^2\big]\leq O(\frac{1}{N^{1 - \eta_1}} +\frac{1}{N})$  for any  $v\in \mathcal{V}$
This section shows that Algorithm 1 is effective to both discrete and continuous environments. We use four centralized baselines for comparisons: (i) Behavior inference from centralized and streaming demonstrations (BICS): This is the centralized counterpart of MA-BIRDS where a central learner obtains all the demonstrations at each online iteration. (ii) Follow the leader (FTL): This method [22] uses a follow-the-leader-based method to solve online non-convex optimization problems where a (near)-stationary solution of the sum of all the previous loss functions is found at each online iteration. As this method only solves single-level optimization problems, we revise it to a single-loop method to solve bi-level optimization problems. (iii) Double-loop method (DLM): This method extends D-ICRL [9] to online centralized settings where the inner-level problem is first solved and then the result is used to solve the outer-level problem. At each online iteration, DLM only updates the decision variable of the outer-level problem once. (iv) ME-greedy: This method is an online extension of [18] which assumes the access to the ground truth reward and uses a greedy method to estimate the constraints based on maximum entropy (ME) IRL [11].
5.1 Evasion from patrolled area
We consider the evader-patroller setting (Figure 2a) introduced in [21]. The experts (E1 and E2) are the patrollers patrolling around the area and they aim to switch their positions. They have four actions (i.e., moving up, down, left, and right) and need to avoid collisions with each other and the obstacles (i.e., constraints). The experts are programmed to follow the optimal policy. The learners (L1 and L2) are the evaders who want to learn the behavior model of the experts in order to reach the goal G without being caught. The red crosses in Figure 2 represent the constraints and each state is colored according to the scaled visitation frequency.
(a) Ground truth environment
Figure 2: Evader-patroller environment.
(b) Learned environment
Figure 2b shows the results learned by Algorithm 1 (each learner recovers the same constraints). Two constraints are not recovered as the experts' policy will not change even these constraints are absent. Three constraints are falsely learned because the experts (optimal policy) do not visit those states but the learned policy (constraint soft Bellman policy) have a high chance of visiting the states if the
states are not prohibited as the constrained soft Bellman policy has non-zero probability of choosing any action and this probability can be large if the corresponding action is not heavily penalized.
To reason about the performance of our algorithm, we use four metrics: false positive rate (FPR), false negative rate (FNR), constraint violation rate (CVR), and success rate (SR). The FPR [18], is the proportion of learned constraints that are not the ground truth constraints, FNR is the proportion of the ground truth constraints that are not learned, CVR introduced in [19] is the percentage of the learned policy violating any constraint, and SR is the percentage of the learned policy reaching the destinations and avoiding collisions.
(a) FPR (lower is better) L1
(b) FNR (lower is BICS
Figure 3: Algorithm performance. The L1 and L2 are the distributed learners in MA-BIRDS.
(c) CVR (lower is better) FTL DLM
(d) SR (higher is better) ME-greedy
Figure 3 shows that Algorithm 1 is on par with and even outperforms the baselines even if it is distributed and does not have the access to the ground truth reward. For the baselines, the centralized learners receive two demonstrations at each online iteration. We can see that the gradient-based methods (i.e., MA-BIRDS, BICS, FTL, and DLM) have much better performance than the greedy-based method (i.e., ME-greedy). The reason is that the greedy-based method can only learn one constraint at each online iteration while the gradient-based methods can learn multiple constraints at each online iteration because the gradient-based methods update  $\omega$  which works on all the possible constraints indicated by the cost feature vector  $\phi$ .
Table 1: Performance comparisons. Here, D means distributed and NATR means no access to the (ground truth) rewards.
D	NATR	FPR	FNR	CVR	SR
MA-BIRDS	L1	✓	✓	0.053 ± 0.008	0.013 ± 0.013	0.040 ± 0.010	0.960 ± 0.010
L2	✓	✓	0.053 ± 0.008	0.013 ± 0.013	0.045 ± 0.015	0.955 ± 0.015
BICS	×	✓	0.038 ± 0.019	0.013 ± 0.013	0.040 ± 0.005	0.960 ± 0.005
FTL	×	✓	0.047 ± 0.009	0.032 ± 0.032	0.030 ± 0.010	0.970 ± 0.010
DLM	×	✓	0.050 ± 0.009	0.022 ± 0.022	0.015 ± 0.005	0.985 ± 0.005
ME-greedy	×	×	0.019 ± 0.000	0.753 ± 0.000	1.000 ± 0.000	0.000 ± 0.000
Table 1 shows the final results after the 40 demonstrations are revealed. Notice that even if the gradient-based methods have different results in FPR and FNR, they achieve similar performance in CVR and SR. The reason is that the different constraints they learn will not affect the learned policy as those constraints are either blocked by other constraints or occupy the states that the experts will barely visit (as shown in Figure 2b).
Moreover, to show that MA-BIRDS is suitable for online learning when the streaming data arrives at a fast speed, we propose the following three metrics: scaled time per online iteration (STPOI),  $50\%$  SR scaled time  $(50\%)$  ST), and  $90\%$  SR scaled time  $(90\%)$  ST). The STPOI is the scaled time that an algorithm needs to finish the computation of an online iteration,  $50\%$  ST is the scaled time that an algorithm needs to reach  $50\%$  SR, and  $90\%$  ST is the scaled time to
Table 2: Computation time comparisons.
STPOI	50% ST	90% ST
L1	1.000	1.000	1.000
L2	1.000	1.000	1.000
BICS	1.157	1.155	1.158
FTL	7.500	3.753	5.769
DLM	7.482	3.740	7.481
ME-greedy	63.615	>318.075	>97.869
reach  $90\%$  SR. We use scaled time instead of actual time because actual time varies a lot on different
hardwares and different problems. The time is scaled in the way that the fastest algorithm has scaled time 1.000.
Table 2 shows that MA-BIRDS and BICS are much faster than the baselines in each online iteration, and thus are more suitable for the online setting with fast streaming data. The MA-BIRDS is slightly faster than BICS in each online iteration because the centralized learner needs to process two new demonstrations at each online iteration while each distributed learner only needs to process one. Moreover, we can see that MA-BIRDS achieves  $50\%$  and  $90\%$  SR with the shortest time.
5.2 Drone motion planning with obstacles
In this example, we analyze MA-BIRDS on a real-world problem with continuous state and action spaces introduced in [9]. We build a simulator in Gazebo (Figure 4a) where we (humans) control the two drones to their diagonal doors while avoiding collisions. We reveal each of the four learners one demonstration at each online iteration. In this experiment, the learners are the computers recording the demonstrations of the experts (humans) controlling the drones. As the state and action spaces are continuous, the potential constraints can also be continuous. Therefore, we cannot use the metrics FPR and FNR, and we cannot use the baseline ME-greedy. As ME-greedy is designed for discrete state-action space, we replace it with ME-gradient [19] which extends ME-greedy to
(a) Gazebo simulator
Figure 4: Drone motion planning with obstacles.
continuous settings where a cost function is learned using a gradient-based method. Similar to ME-greedy, ME-gradient is centralized and assumes access to the ground truth reward. Figure 4b shows the learned constraints on which the learners reach consensus and the trajectories of the learned policy. Each of the four trajectories is generated by one of the four learners L1-L4. Each trajectory consists of the paths of the two drones (i.e., a red path and a blue path).
Table 3: Performance and computation time comparisons (drone motion planning).
D	NATR	CVR	SR	STPOI	50% ST	90% ST
L1	✓	✓	0.025 ± 0.025	0.974 ± 0.025	1.000	1.000	1.000
L2	✓	✓	0.030 ± 0.010	0.970 ± 0.010	1.000	1.000	1.000
L3	✓	✓	0.025 ± 0.025	0.973 ± 0.025	1.000	1.000	1.000
L4	✓	✓	0.025 ± 0.025	0.972 ± 0.015	1.000	1.000	1.000
BICS	×	✓	0.015 ± 0.015	0.979 ± 0.020	1.029	1.025	1.027
FTL	×	✓	0.015 ± 0.010	0.981 ± 0.018	10.481	7.860	5.240
DLM	×	✓	0.020 ± 0.015	0.975 ± 0.015	9.658	6.767	6.837
ME-gradient	×	×	0.013 ± 0.013	0.980 ± 0.018	1.013	1.010	1.012
Table 3, together with Table 2, shows that MA-BIRDS and BICS can reach the same good performance with the baselines and use much shorter time or have fewer requirements. In specific, compared to FTL [22], MA-BIRDS only requires about  $20\%$  of the time to reach the same performance and is more than six times faster in each iteration. Compared to DLM [9], MA-BIRDS only requires about  $15\%$  of the time to reach the same performance and is also more than six times faster in each iteration. Compared to ME-gradient, MA-BIRDS can reach the same performance within the same amount of time even if it is distributed and has no access to the ground truth reward.
6 Discussion and future work
We propose MA-BIRDS, the first IRL framework that is effective in learning multi-agent behaviors from distributed and streaming demonstrations under continuous and discrete environments. We formulate a distributed online bi-level optimization problem and propose a fast distributed online gradient descent single-loop algorithm with theoretical guarantees that is suitable to online settings.
Experimental results show that MA-BIRDS is effective in both continuous and discrete environments. Despite its benefits, one limitation is the assumption of linear cost functions. We will explore approaches to relax this assumption in the future.
Potential negative social impact. Since MA-BIRDS can infer the experts' behaviors, potential negative social impact may occur if the learners are malicious. Take the evader-patroller setting as an example; the patrollers could be safeguards or park rangers, and the evaders could be poachers. The malicious poachers may use MA-BIRDS to escape. To avoid this situation, the experts should take additional strategies such as regularly demonstrating misleading behaviors so that MA-BIRDS will learn a wrong behavior model of the experts.
This work is partially supported by the National Science Foundation through grants ECCS 1846706 and CNS 1830390. We would like to thank the reviewers for their insightful and constructive suggestions.
[37] S. Arora and P. Doshi, "A survey of inverse reinforcement learning: Challenges, methods and progress," Artificial Intelligence, vol. 297, p. 103500, 2021.
[38] B. D. Ziebart, J. A. Bagnell, and A. K. Dey, "Modeling interaction via the principle of maximum causal entropy," in International Conference on Machine Learning, pp. 1255-1262, 2010.
[39] S. P. Boyd and L. Vandenberghe, Convex Optimization. Cambridge university press, 2004.
[40] B. D. Ziebart, Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, 2010.
[41] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement learning with deep energy-based policies,” in International Conference on Machine Learning, pp. 1352–1361, 2017.
[42] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor," in International Conference on Machine Learning, pp. 1861-1870, 2018.