Title: Learning Multi-agent Behaviors from Distributed and Streaming Demonstrations
Venue: NeurIPS 2023
Accepted: yes
Keywords: inverse reinforcement learning; distributed online bi-level optimization
Abstract: This paper considers the problem of inferring the behaviors of multiple interacting experts by estimating their reward functions and constraints where the distributed demonstrated trajectories are sequentially revealed to a group of learners. We formulate the problem as a distributed online bi-level optimization problem where the outer-level problem is to estimate the reward functions and the inner-level problem is to learn the constraints and corresponding policies. We propose a novel ``multi-agent behavior inference from distributed and streaming demonstrations" (MA-BIRDS) algorithm that allows the learners to solve the outer-level and inner-level problems in a single loop through intermittent communications. We formally guarantee that the distributed learners achieve consensus on reward functions, constraints, and policies, the average local regret (over $N$ online iterations) decreases at the rate of $O(1/N^{1-\eta_1}+1/N^{1-\eta_2}+1/N)$, and the cumulative constraint violation increases sub-linearly at the rate of $O(N^{\eta_2}+1)$ where $\eta_1,\eta_2\in (1/2,1)$.

Learning Multi-agent Behaviors from Distributed and Streaming Demonstrations

Abstract

This paper considers the problem of inferring the behaviors of multiple interacting experts by estimating their reward functions and constraints where the distributed demonstrated trajectories are sequentially revealed to a group of learners. We formulate the problem as a distributed online bi-level optimization problem where the outer-level problem is to estimate the reward functions and the inner-level problem is to learn the constraints and corresponding policies. We propose a novel 'multi-agent behavior inference from distributed and streaming demonstrations" (MA-BIRDS) algorithm that allows the learners to solve the outer-level and innerlevel problems in a single loop through intermittent communications. We formally guarantee that the distributed learners achieve consensus on reward functions, constraints, and policies, the average local regret (over N online iterations) decreases at the rate of O (1 /N 1 -η 1 +1 /N 1 -η 2 +1 /N ) , and the cumulative constraint violation increases sub-linearly at the rate of O ( N η 2 +1) where η 1 , η 2 ∈ (1 / 2 , 1) .

1 Introduction

Multi-agent systems (MASs) are an effective tool to model networked systems where multiple entities interact with each other to reach certain goals. Due to the lack of a centralized authority, the data in MASs is usually distributed [1, 2]. Therefore, distributed learning is desired for MASs where machine learning models are trained over distributed data sets. Current works on distributed learning include distributed supervised learning [3, 4], distributed unsupervised learning [5, 6], distributed reinforcement learning [7, 8], etc. Recently, distributed learning is applied to learn the behaviors in MASs from distributed experts' demonstrations via inverse reinforcement learning (IRL) [9].

In IRL [10, 11, 12, 13, 14], a learner aims to learn a policy that imitates the expert behaviors in the demonstrations by first learning a reward function. Multi-agent IRL [15, 16, 17] extends IRL to MASs where the reward functions and policies of a group of experts are learned. However, current works on multi-agent IRL have the following two limitations: (i) As mentioned, the demonstration data in MASs is usually distributed while these works assume that there is a centralized learner which can obtain all the demonstrations. (ii) There are usually many underlying constraints in MASs, e.g., avoiding collision with each other and obstacles, and thus learning a reward function combined with a set of constraints is better than learning a single reward function in terms of explaining the experts' behaviors [18, 19]. Therefore, distributed inverse constrained reinforcement learning (D-ICRL) [9] is proposed where a group of learners cooperatively learn the behaviors in an MAS by estimating the experts' reward functions and constraints and each learner can only access a local demonstration set.

While D-ICRL performs over pre-collected distributed data, recent applications [20, 21] of IRL motivate the need for algorithms that can learn from sequentially revealed demonstrations and continuously improve the learned models. For example, inferring a person's intent by observing her

ongoing daily routine [20] and updating evasion strategies by continuously observing the patrollers' behaviors [21]. However, D-ICRL is not efficient to deal with streaming data because it has a double-loop learning structure where the outer loop is to update the constraints and the inner loop needs to find a corresponding reward estimate. While the computation overhead of this double-loop structure is reluctantly acceptable in offline settings, it is too time-consuming for streaming data since the computation may not be finished before the next data arrives. As D-ICRL is the most important baseline of our work, we include a section in Appendix to summarize our improvements from it.

Related works and our improvements . Current theoretical works on online IRL [20, 21] only consider linear reward functions and thus cast the problem as online convex optimization. While these approaches achieve sub-linear regret, their analysis does not hold for non-linear reward functions (e.g., neural networks) as the corresponding objective functions can be non-convex. To quantify the algorithmic performance on online non-convex optimization, 'local regret" [22, 23] is proposed which, at each online iteration, quantifies the gradient norms of the average sum of some already-revealed loss functions. Current state-of-the-arts [22, 23] use follow-the-leader-based methods to minimize the local regret. While their methods can achieve the tight regret bound, the follow-the-leader-based methods require multiple gradient descent steps until reaching a near-stationary point at each online iteration. However, if the data arrival rate is fast, the computation may not be done before the next data arrives. To alleviate the computation burden at each online iteration, we use an online gradient descent (OGD) method which only updates the decision variable by one gradient descent step at each online iteration. To the best of our knowledge, no works can quantify the local regret of OGD.

Inspired by [9, 24], we formulate a distributed online bi-level optimization problem where the learners cooperatively learn the reward functions in the outer-level problem and the constraints and corresponding policies in the inner-level problem. Papers [9, 24, 25] use double-loop methods [26, 27, 28] to solve their bi-level optimization problems where they first find a (sub)optimal solution of the inner-level problem in a faster loop and then solve the outer-level problem in a slower loop. However, when it comes to streaming data, the double-loop method needs multiple steps for the inner-level problem before updating the decision variable of the outer-level problem, which can be too slow to finish the update of the outer decision variable when the data is revealed in a fast speed. Therefore, we use a single-loop method which updates both the outer and inner decision variable only once at each iteration. Notice that the state-of-the-arts on single-loop bi-level optimization [29, 30] cannot be directly applied to our problem because they are centralized and require the inner objective function to be strongly convex while our problem does not have these properties.

Contribution statement . Our contributions are threefold. First, we consider the problem where a group of learners cooperatively recover the policies by estimating the reward functions and constraints from distributed streaming demonstrations of cooperative experts. We formulate this 'multi-agent behavior inference from distributed and streaming demonstrations" (MA-BIRDS) problem as a distributed online bi-level optimization problem. Second, we propose a novel distributed online gradient descent algorithm for the learners to learn the reward functions, constraints, and the corresponding policies in a single loop where the decision variables of both the outer-level and inner-level problems are updated only once at each online iteration. Third, we prove that the distributed learners achieve consensus in reward functions, constraints, and policies, respectively, at the rate of O (1 /N η 1 +¯ ϵ N ) , O (1 /N η 2 +¯ ϵ N ) , and O (1 /N η 1 +1 /N η 2 +¯ ϵ N ) where ¯ ϵ ∈ (0 , 1) and η 1 , η 2 ∈ (1 / 2 , 1) . The local regret averaged over N iterations decreases at the rate of O (1 /N 1 -η 1 +1 /N 1 -η 2 +1 /N ) and the cumulative constraint violation grows sub-linearly at the rate of O ( N η 2 +1) . Moreover, if the reward functions are linear, we prove that the average cumulative reward difference between the experts and learners diminishes at the rate of O (1 /N 1 -η 1 +1 /N ) .

2 Model

This section presents the models of the experts and learners.

Experts . There are N E experts whose decision making is modeled as a constrained Markov game (CMG) [31]. A CMG ( S , A , γ, T, P 0 , P, r E , c E , b ) consists of a state set S ≜ ∏ N E i =1 S ( i ) , an action set A ≜ ∏ N E i =1 A ( i ) , a discount factor γ , a time horizon T , and an initial state distribution P 0 . The state transition function is P and P ( s ′ | s, a ) represents the probability of transitioning to state s ′ from s by taking action a ≜ ( a (1) , · · · , a ( N E ) ) . Expert i 's reward function is r ( i ) E : S × A → R and the

experts are cooperative, i.e., r E ≜ ∑ N E i =1 r ( i ) E . The cost function of expert i is c ( i ) E ≜ ( ω ( i ) E ) ⊤ ϕ ( i ) where ϕ ( i ) : S × A → [0 , d 1 ] l ( i ) is an l ( i ) -dimensional cost feature vector, d 1 is a bounded constant, and ω ( i ) E ∈ R l ( i ) + is the weight. The cost function of all the experts is c E ≜ ∑ N E i =1 c ( i ) E . Expert i 's policy π ( i ) E ( a ( i ) | s ) represents the probability of expert i taking action a ( i ) at state s and the joint policy of all the experts is π E ( a | s ) ≜ ∏ N E i =1 π ( i ) E ( a ( i ) | s ) . We define J r E ( π ) ≜ E π S,A [ ∑ T t =0 γ t r E ( S t , A t )] as the expected cumulative reward under policy π where the initial state is drawn from P 0 and J c E ( π ) ≜ E π S,A [ ∑ T t =0 γ t c E ( S t , A t )] as the expected cumulative cost. The experts' policy π E wants to maximize J r E ( π ) subject to J c E ( π ) ≤ b where b is the budget. Following [9, 18, 19], we study hard constraints (i.e., b = 0 ). These experts use π E to demonstrate N L trajectories { ζ [ v ] ( n ) } N L v =1 at each online iteration n where each trajectory ζ [ v ] ( n ) ≜ s [ v ] 0 ( n ) , a [ v ] 0 ( n ) , · · · , s [ v ] T ( n ) , a [ v ] T ( n ) is a state-action sequence of all the experts. This distributed online data style can also be found in [32].

Learners . There are N L learners where each learner v knows ( γ, T, { ϕ ( i ) } N E i =1 , ζ [ v ] ( n )) and ζ [ v ] ( n ) is a demonstration observed by learner v at online iteration n . Each learner wants to use communications to learn the cost functions by estimating ω E ≜ [( ω (1) E ) ⊤ , · · · , ( ω ( N E ) E ) ⊤ ] ⊤ and reward functions using parameterized models { r ( i ) θ ( i ) } N E i =1 where θ ( i ) ∈ R d ( i ) is d ( i ) -dimensional. Here we relax the linear reward assumption in [9] but keep its linear cost assumption because non-linear cost functions can make the problem ill-defined (explained in Appendix).

Assumption 1. The reward function r θ satisfies the following: | r θ ( s, a ) | ≤ C , ||∇ θ r θ ( s, a ) || ≤ ¯ C , and ||∇ 2 θθ r θ ( s, a ) || ≤ ˜ C for any ( s, a ) ∈ S ×A and any θ where C , ¯ C , and ˜ C are positive constants.

Notice that Assumption 1 is standard in RL [33, 34, 35].

The communication network is modeled as a time-varying directed graph G ( n ) ≜ ( V , E ( n )) where V ≜ { 1 , · · · , N L } is the node (learner) set and E ( n ) ⊆ V × V is the set of directed edges (communication links) at time n . The edge ( v, v ′ ) ∈ E ( n ) means that learner v receives information from learner v ′ at time n and ( v, v ) ∈ E ( n ) for all n ≥ 0 . The adjacency matrix of the graph at time n is W ( n ) ≜ [ W [ vv ′ ] ( n )] v,v ′ ∈V ∈ R N L × N L where W [ vv ′ ] ( n ) = 0 if and only if ( v, v ′ ) / ∈ E ( n ) . The set of neighbors of learner v at time n is N [ v ] ( n ) ≜ { v ′ ∈ V| ( v, v ′ ) ∈ E ( n ) } .

Assumption 2. There exists an integer B ≥ 1 such that the graph ( V , E ( n ) ∪ · · · ∪ E ( n + B -1)) is strongly connected for all n ≥ 0 .

Assumption 3. The adjacency matrix W ( n ) has the following properties: (i) 1 ⊤ W ( n ) = 1 ⊤ and W ( n ) 1 = 1 where 1 is the column vector whose entries are all ones. (ii) There is an ϵ ∈ (0 , 1) such that W [ vv ] ( n ) ≥ ϵ for all v ∈ V and W [ vv ′ ] ( n ) ≥ ϵ if ( v, v ′ ) ∈ E ( n ) .

Notice that these two assumptions are standard in distributed learning [3, 9, 36].

Figure 1 shows that the learners stand outside the MAS observing the sequential data revealed by the experts and collaboratively learn the reward functions and constraints. Each learner aims to learn the reward functions and constraints of all the experts where θ [ v ] and ω [ v ] are the reward and cost function parameter es-

timates of all the experts learned by learner v .

Notions and Notations . Define θ ≜ [( θ (1) ) ⊤ , · · · , ( θ ( N E ) ) ⊤ ] ⊤ and r θ ≜ ∑ N E i =1 r ( i ) θ ( i ) . We use θ [ v ] to represent learner v 's learned θ . Therefore, the dimension of each θ [ v ] is ∑ N E i =1 d ( i ) . Given a trajectory ζ = s 0 , a 0 , · · · , s T , a T , the empirical cumulative reward under r θ is ˆ J r θ ( ζ ) ≜ ∑ T t =0 γ t r θ ( s t , a t ) , the empirical cumulative cost feature is defined as ˆ µ ( ζ ) ≜ ∑ T t =0 γ t ϕ ( s t , a t ) where ϕ ≜ [( ϕ (1) ) ⊤ , · · · , ( ϕ ( N E ) ) ⊤ ] ⊤ . The expectation of cumulative cost feature under a given policy π is µ ( π ) ≜ E π S,A [ ∑ T t =0 γ t ϕ ( S t , A t )] and the expectation of cumulative reward r θ under a

given policy π is J r θ ( π ) ≜ E π S,A [ ∑ T t =0 γ t r θ ( S t , A t )] . The set of all stochastic policies is Π where every π ∈ Π satisfies π ( a | s ) ≥ 0 for any ( s, a ) ∈ S × A and ∫ a ∈A π ( a | s ) da = 1 for any s ∈ S .

3 Problem Formulation

In MA-BIRDS, the learners collaboratively learn the experts' policy π E by estimating the reward function r E and cost function c E , and each learner v does not share its private data such as local trajectory ζ [ v ] and local estimates ˆ J r θ ( ζ [ v ] ) and ˆ µ ( ζ [ v ] ) . While it seems that learning a well-structured reward function is enough to prevent 'bad" movements by assigning negative reward, we include a section (in Appendix) to further discuss the benefits of learning both reward and cost functions.

Many IRL works [9, 11, 24, 37 ? ] have a bi-level learning structure where the outer level is to learn a reward function and the inner level is to learn a corresponding policy by solving an RL problem under the current learned reward function. Inspired by their bi-level structure, we formulate a bi-level optimization problem where the outer level is to learn a reward function r θ and the inner level is to find the cost function and policy corresponding to r θ . In what follows, we first define the inner-level optimization problem and then introduce the bi-level optimization problem.

The inner-level optimization . Given a learned reward function r θ , the corresponding policy π r θ is the optimal solution of the following constrained RL problem where the reward function is r θ :

where H ( π ) ≜ ∑ T t =0 E S,A [ -γ t ln π ( A t | S t )] is causal entropy [38]. The constraint in problem (1) is cost feature expectation matching similar to the spirit of 'feature expectation matching" in [11, 38].

However, the policy π r θ is hard to get because problem (1) is non-convex. A standard way to tackle this difficulty is dual methods [39], therefore, we introduce the dual function of problem (1): G ( ω ; θ, n ) ≜ max π ∈ Π H ( π ) + J r θ ( π ) + ω ⊤ ( µ ( π ) -1 N L ∑ N L v =1 ˆ µ ( ζ [ v ] ( n ))) where the dual variable ω is used to estimate ω E . Notice that the dual function G ( ω ; θ, n ) is convex in ω [39].

Lemma 1. The optimal solution of problem (1) is the constrained soft Bellman policy π ω ∗ ( θ,n ); θ and its parameter ω ∗ ( θ, n ) is the optimal solution of the dual problem min ω G ( ω ; θ, n ) .

The expression of constrained soft Bellman policy is in Appendix. Lemma 1 shows that π ω ∗ ( θ,n ); θ = π r θ and ω ∗ ( θ, n ) = arg min ω G ( ω ; θ, n ) . Therefore, we can solve problem (1) by solving its dual problem. We use the dual problem to be the inner-level problem where ω ∗ ( θ, n ) is the learned cost function and π ω ∗ ( θ,n ); θ is the learned policy corresponding to the current learned reward function r θ .

The bi-level optimization . Given a reward function r θ , the inner-level problem arg min ω G ( ω ; θ, n ) can find the corresponding cost function and policy. The outer level aims to learn r θ via minimizing the following loss function over θ :

ω

where L ( θ, ω ∗ ( θ, n ) , n ) ≜ -∑ N L v =1 ∑ T t =0 γ t ln π ω ∗ ( θ,n ); θ ( a [ v ] t ( n ) | s [ v ] t ( n )) is the negative log likelihood of the trajectories { ζ [ v ] ( n ) } N L v =1 received at time n under the policy π ω ∗ ( θ,n ); θ [9, 40] and ζ [ v ] ( n ) = { ( s [ v ] t ( n ) , a [ v ] t ( n )) } 0 ≤ t ≤ T . The likelihood function is widely used in IRL [9 ? , 40] to learn the reward function.

Notice that (2) requires all the demonstrations { ζ [ v ] ( n ) } N L v =1 at time n . However, each learner v can only observe ζ [ v ] ( n ) and formulate its local negative log likelihood function L [ v ] ( θ, ω ∗ ( θ, n ) , n ) ≜ -∑ T t =0 γ t ln π ω ∗ ( θ,n ); θ ( a [ v ] t ( n ) | s [ v ] t ( n )) and local dual function G [ v ] ( ω ; θ, n ) ≜ max π ∈ Π H ( π ) + J r θ ( π ) + ω ⊤ ( µ ( π ) -ˆ µ ( ζ [ v ] ( n ))) . Notice that L = ∑ N L v =1 L [ v ] and G = 1 N L ∑ N L v =1 G [ v ] .

As the demonstrations are streaming, we have a sequence of loss functions { L ( · , ω ∗ ( · , n ) , n ) } n ≥ 1 . We use this sequence of loss functions to formulate an online learning problem and a common problem for online learning is to minimize the regret: ∑ N n =1 L ( θ ( n ) , ω ∗ ( θ ( n ) , n ) , n ) -∑ N n =1 L ( θ ∗ , ω ∗ ( θ ∗ , n ) , n )

which quantifies the difference of the accumulative losses between the learned parameter θ and the best parameter θ ∗ in hindsight. However, it is too challenging to minimize the regret in our case because L is non-convex [22]. Therefore, we use local regret [22, 23] which is widely used in online non-convex optimization. It quantifies the general stationarity of a sequence of loss functions. In specific, given a sequence of loss functions { f ( · , n ) } n ≥ 1 , the local regret [22, 23] at online iteration n is defined as || 1 l ∑ l -1 i =0 ∇ f ( x ( n ) , n -i ) || 2 which quantifies the gradient norms of the average of l previously received loss functions under the current learned parameter x ( n ) . The total local regret is defined as the sum of the local regret at every online iteration n , i.e., ∑ N n =1 || 1 l ∑ l -1 i =0 ∇ f ( x ( n ) , n -i ) || 2 . In our case, we replace f with our loss function (2) and thereby formulate the local regret (3)-(4) which has a distributed bi-level formulation. We want to minimize the local regret (3)-(4).

The time window length is 1 ≤ l ≤ N and L [ v ] ( θ, ω, i ) = 0 if i ≤ 0 [22]. The outer-level problem (3) is to learn the reward parameter θ and the inner-level problem (4) is to learn the cost parameter ω given θ . The learned policy is the constrained soft Bellman policy π ω ; θ with parameters ( θ, ω ) .

4 Algorithm and Performance Guarantee

This section consists of two subsections where the first one introduces an approximation method to solve the bi-level optimization problem (3)-(4) in a single loop and the second one introduces a consensus-based method for the multiple learners to solve the problem in a distributed way.

4.1 Approximation-based single-loop method

In this part, we develop an approximation-based single-loop method which (i) does not use the exact gradient of the outer-level problem (3) but an approximation of the gradient; (ii) solves the outer-level and inner-level problems in a single loop. In the following analysis, for simple notations, we omit the time index n and imply that the analysis holds for all n .

Lemma 2. The problem min ω G ( ω ; θ ) has a unique optimal solution ω ∗ ( θ ) for any θ .

Since the inner-level problem (4) is unconstrained and ω ∗ ( θ ) is its optimal solution, then we have ∇ ω G ( ω ∗ ( θ ); θ ) = 0 . Taking derivative with respect to θ on both sides renders:

where M ( θ, ω ) ≜ ∇ 2 G ( ω ; θ )[ ∇ 2 G ( ω ; θ )] -1 . Then, using the chain rule, we have:

At online iteration n , an intuitive way to solve the bi-level optimization problem (3)-(4) is to solve it in a double-loop way [9, 26, 28]. In specific, the double-loop method first solves the inner-level problem (4) to find a close approximation of ω ∗ ( θ ) and then uses the obtained result to get the gradient (5), thus solving the outer-level problem (3). However, it requires multiple gradient descent for the inner-level problem (4) to get ω ∗ ( θ ) before updating the decision variable of the outer-level problem where each gradient descent of the inner-level problem needs to solve a constrained RL problem in our case. Therefore, the double-loop method is not suitable to be an online algorithm when the data is revealed in a fast speed. To design an algorithm suitable for the online fashion, we adopt the spirit of [29, 30] and solve the outer-level and inner-level problems in a single loop. However, our method is not a simple extension of [29, 30] to online settings because they are centralized and require the inner objective function to be strongly convex but our problem does not have these properties.

In specific, at each online iteration, the decision variables of both the outer-level and inner-level problems update only once where the gradient of the inner objective function is given in Lemma 3. For the outer-level problem, as ω ∗ ( θ ) is inaccessible, we cannot get the exact gradient defined in

(5). Therefore, we propose the following gradient approximation whose derivation can be found in Appendix: ¯ ∇ L ( θ, ω ) = N L E π ω ; θ S,A [ ∑ T t =0 γ t ∇ θ r θ ( S t , A t )] -∑ N L v =1 ∇ θ ˆ J r θ ( ζ [ v ] ) , where π ω ; θ is the constrained soft Bellman policy with parameters ( ω, θ ) . Compared to (5), this gradient approximation does not include the second-order term M ( θ, ω ) and thus is much more computationally efficient. It is shown in Appendix that the approximation error ||∇ L ( θ, ω ∗ ( θ )) -¯ ∇ L ( θ, ω ) || ≤ C θ || ω ∗ ( θ ) -ω || where C θ is a positive constant whose expression is in Appendix. Intuitively, as the inner decision variable ω approaches ω ∗ ( θ ) in the learning process, this gradient approximation error can be sufficiently small.

Lemma 3. The gradient of G [ v ] ( ω ; θ ) is µ ( π ω ; θ ) -ˆ µ ( ζ [ v ] ) where π ω ; θ is the constrained soft Bellman policy with parameters ( θ, ω ) .

Lemma 4. The global likelihood function L ( θ, ω ) is C L -lipschitz continuous and ¯ C L -smooth in ( θ, ω ) , where C L and ¯ C L are positive constants.

4.2 Consensus-based distributed learning

In our distributed learning setting, each learner v only knows its local information (e.g., L [ v ] and G [ v ] ), so that it cannot directly solve the problem (3)-(4). Therefore, each learner v updates the decision variables using the gradients of its local outer-level and inner-level objective functions and uses communications to collaboratively solve the problem (3)-(4). Similar to the global gradient approximation ¯ ∇ L ( θ, ω ) , learner v has its local gradient approximation: ¯ ∇ L [ v ] ( θ, ω ) = E π ω ; θ S,A [ ∑ T t =0 γ t ∇ θ r θ ( S t , A t )] - ∇ θ ˆ J r θ ( ζ [ v ] ) . Learner v uses samples to estimate ¯ ∇ L [ v ] ( θ, ω ) via ˆ ∇ L [ v ] ( θ, ω ) ≜ 1 m [ v ] ∑ m [ v ] j =1 ∇ θ ˆ J r θ ( ζ j ) - ∇ θ ˆ J r θ ( ζ [ v ] ) where ζ j is generated by rolling out π ω ; θ and m [ v ] is the number of the samples. Moreover, learner v also uses the same samples to estimate ∇ G [ v ] ( ω ; θ ) in Lemma 3 via ˆ ∇ G [ v ] ( ω ; θ ) ≜ 1 m [ v ] ∑ m [ v ] j =1 ˆ µ ( ζ j ) -ˆ µ ( ζ [ v ] ) .

Algorithm 1 Multi-agent behavior inference from distributed and streaming demonstrations

In Algorithm 1 , at online iteration n , each learner v sequentially executes the following two steps: (i) uses current reward and cost function parameters to get the corresponding policy and generate samples (Lines 3 -4 ); (ii) communicates with neighbors and updates its parameters for both outer-level and inner-level problems in a single loop using the samples generated in last step (Lines 5 -7 ). In the update process (Lines 6 -7 ), the first term (convex combination) encourages consensus among different learners and the second term (gradient) drives to the set of stationary points. Notice that even though soft Q-learning and soft actor-critic are designed for unconstrained RL, as shown in Appendix, we can revise them to approximate the constrained soft Bellman policy.

Theorem 1. Suppose Assumptions 1, 2, 3 hold. Let the step sizes α ( n ) = ¯ α n η 1 and β ( n ) = ¯ β n η 2 where η 1 , η 2 ∈ ( 1 2 , 1) and ¯ α, ¯ β ∈ (0 , 1 N L (2 ¯ C L +1) ) , then for any v, v ′ ∈ V in Algorithm 1 : (consensus): || θ [ v ] ( N ) -θ [ v ′ ] ( N ) || ≤ O ( 1 N η 1 + ¯ ϵ N ) , || ω [ v ] ( N ) -ω [ v ′ ] ( N ) || ≤ O ( 1 N η 2 + ¯ ϵ N ) , and sup ( s,a ) ∈S×A {| π ω [ v ] ( N ); θ [ v ] ( N ) ( a | s ) -π ω [ v ′ ] ( N ); θ [ v ′ ] ( N ) ( a | s ) |} ≤ O ( 1 N η 1 + 1 N η 2 +¯ ϵ N )) , where sup ( s,a ) ∈S×A {·} outputs the supremum value over S × A and the expression of ¯ ϵ ∈ (0 , 1) can be

found in Appendix.

(sub-linear cumulative constraint violation): ∑ N n =1 E [ J 2 c E ( π ω [ v ] ( n ); θ [ v ] ( n ) ) ] ≤ O ( N η 2 +1) , where the expectation is taken over randomly demonstrated trajectories at each online iteration.

Theorem 1 shows that the learners will achieve consensus on the reward and cost function parameters, and policy. The average local regret decreases to an upper bound 2( C L ) 2 /l . Notice that this upper bound has similar property with that in [22, 23]: (i) they are both less that the uniform upper bound C 2 L of the global loss function L when l > 2 ; (ii) they are both diminishing if l is dependent on n and l ( n ) = ω (1) [23]. The number of total outer gradient steps taken by each learner in Algorithm 1 is O ( N ) which is smaller than O (2 Nl + l 2 ) in [22] and O ( Nl 2 ) in [23]. The reason of fewer gradient steps in Algorithm 1 is that it is an online gradient descent algorithm, which does not require multiple gradient calls at each online iteration as follow-the-leader-based methods [22, 23] do.

Moreover, if the reward functions are linear as in [20, 21], we have the following stronger result: Corollary 1. If the reward functions are linear, the average difference of cumulative reward between the learned policy and expert policy diminishes: 1 N ∑ N n =1 E [ ( J r E ( π ω [ v ] ( n ); θ [ v ] ( n ) ) -J r E ( π E )) 2 ] ≤ O ( 1 N 1 -η 1 + 1 N ) for any v ∈ V .

5 Simulations

This section shows that Algorithm 1 is effective to both discrete and continuous environments. We use four centralized baselines for comparisons: (i) Behavior inference from centralized and streaming demonstrations (BICS) : This is the centralized counterpart of MA-BIRDS where a central learner obtains all the demonstrations at each online iteration. (ii) Follow the leader (FTL) : This method [22] uses a follow-the-leader-based method to solve online non-convex optimization problems where a (near)-stationary solution of the sum of all the previous loss functions is found at each online iteration. As this method only solves single-level optimization problems, we revise it to a single-loop method to solve bi-level optimization problems. (iii) Double-loop method (DLM) : This method extends D-ICRL [9] to online centralized settings where the inner-level problem is first solved and then the result is used to solve the outer-level problem. At each online iteration, DLM only updates the decision variable of the outer-level problem once. (iv) ME-greedy : This method is an online extension of [18] which assumes the access to the ground truth reward and uses a greedy method to estimate the constraints based on maximum entropy (ME) IRL [11].

5.1 Evasion from patrolled area

We consider the evader-patroller setting (Figure 2a) introduced in [21]. The experts (E1 and E2) are the patrollers patrolling around the area and they aim to switch their positions. They have four actions (i.e., moving up, down, left, and right) and need to avoid collisions with each other and the obstacles (i.e., constraints). The experts are programmed to follow the optimal policy. The learners (L1 and L2) are the evaders who want to learn the behavior model of the experts in order to reach the goal G without being caught. The red crosses in Figure 2 represent the constraints and each state is colored according to the scaled visitation frequency.

(a) Ground truth environment

(b) Learned environment

1.0

E xxxxxxxxxx

E1

XxxXxxxXxXX

区

E1

X

×

0.8

x×x××xx

X

XXXXXX

X

XXXXX

X

X

XXXX

X

X

XXXXX

0.6

0.4

X

+

X

X

0.2

X

X

X

X

X

X

×

E2

E2

0.0

Figure 2b shows the results learned by Algorithm 1 (each learner recovers the same constraints). Two constraints are not recovered as the experts' policy will not change even these constraints are absent. Three constraints are falsely learned because the experts (optimal policy) do not visit those states but the learned policy (constraint soft Bellman policy) have a high chance of visiting the states if the

states are not prohibited as the constrained soft Bellman policy has non-zero probability of choosing any action and this probability can be large if the corresponding action is not heavily penalized.

To reason about the performance of our algorithm, we use four metrics: false positive rate (FPR), false negative rate (FNR), constraint violation rate (CVR), and success rate (SR). The FPR [18], is the proportion of learned constraints that are not the ground truth constraints, FNR is the proportion of the ground truth constraints that are not learned, CVR introduced in [19] is the percentage of the learned policy violating any constraint, and SR is the percentage of the learned policy reaching the destinations and avoiding collisions.

(a) FPR (lower is better)

(b) FNR (lower is better)

(c) CVR (lower is better)

(d) SR (higher is better)

0.14

1.0

1.0

1.0

0.12

0.8

0.8

0.8

0.10

0.08

0.6

0.6

0.6

0.06

0.4

0.4

0.4

0.04

0.02

0.2

0.2

0.2

0.00

0.0

0.0

0.0

0

10

20

30

40

0

10

20

30

40

0

10

20

30

40

0

10

20

30

40

Demonstrations

Demonstrations

Demonstrations

Demonstrations

L1

L2

BICS

FTL

DLM

ME-greedy

Figure 3 shows that Algorithm 1 is on par with and even outperforms the baselines even if it is distributed and does not have the access to the ground truth reward. For the baselines, the centralized learners receive two demonstrations at each online iteration. We can see that the gradient-based methods (i.e., MA-BIRDS, BICS, FTL, and DLM) have much better performance than the greedybased method (i.e., ME-greedy). The reason is that the greedy-based method can only learn one constraint at each online iteration while the gradient-based methods can learn multiple constraints at each online iteration because the gradient-based methods update ω which works on all the possible constraints indicated by the cost feature vector ϕ .

Table 1 shows the final results after the 40 demonstrations are revealed. Notice that even if the gradient-based methods have different results in FPR and FNR, they achieve similar performance in CVR and SR. The reason is that the different constraints they learn will not affect the learned policy as those constraints are either blocked by other constraints or occupy the states that the experts will barely visit (as shown in Figure 2b).

Moreover, to show that MA-BIRDS is suitable for online learning when the streaming data arrives at a fast speed, we propose the following three metrics: scaled time per online iteration (STPOI), 50% SR scaled time ( 50% ST), and 90% SR scaled time ( 90% ST). The STPOI is the scaled time that an algorithm needs to finish the computation of an online iteration, 50% ST is the scaled time that an algorithm needs to reach 50% SR, and 90% ST is the scaled time to

reach 90% SR. We use scaled time instead of actual time because actual time varies a lot on different

hardwares and different problems. The time is scaled in the way that the fastest algorithm has scaled time 1 . 000 .

Table 2 shows that MA-BIRDS and BICS are much faster than the baselines in each online iteration, and thus are more suitable for the online setting with fast streaming data. The MA-BIRDS is slightly faster than BICS in each online iteration because the centralized learner needs to process two new demonstrations at each online iteration while each distributed learner only needs to process one. Moreover, we can see that MA-BIRDS achieves 50% and 90% SR with the shortest time.

5.2 Drone motion planning with obstacles

In this example, we analyze MA-BIRDS on a realworld problem with continuous state and action spaces introduced in [9]. We build a simulator in Gazebo (Figure 4a) where we (humans) control the two drones to their diagonal doors while avoiding collisions. We reveal each of the four learners one demonstration at each online iteration. In this experiment, the learners are the computers recording the demonstrations of the experts (humans) controlling the drones. As the state and action spaces are continuous, the potential constraints can also be continuous. Therefore, we cannot use the metrics FPR and FNR, and we cannot use the baseline ME-greedy. As ME-greedy is designed for discrete state-action space, we replace it with ME-gradient [19] which extends ME-greedy to

××

X

xxxxx

××

continuous settings where a cost function is learned using a gradient-based method. Similar to ME-greedy, ME-gradient is centralized and assumes access to the ground truth reward. Figure 4b shows the learned constraints on which the learners reach consensus and the trajectories of the learned policy. Each of the four trajectories is generated by one of the four learners L 1 -L 4 . Each trajectory consists of the paths of the two drones (i.e., a red path and a blue path).

Table 3, together with Table 2, shows that MA-BIRDS and BICS can reach the same good performance with the baselines and use much shorter time or have fewer requirements. In specific, compared to FTL [22], MA-BIRDS only requires about 20% of the time to reach the same performance and is more than six times faster in each iteration. Compared to DLM [9], MA-BIRDS only requires about 15% of the time to reach the same performance and is also more than six times faster in each iteration. Compared to ME-gradient, MA-BIRDS can reach the same performance within the same amount of time even if it is distributed and has no access to the ground truth reward.

6 Discussion and future work

We propose MA-BIRDS, the first IRL framework that is effective in learning multi-agent behaviors from distributed and streaming demonstrations under continuous and discrete environments. We formulate a distributed online bi-level optimization problem and propose a fast distributed online gradient descent single-loop algorithm with theoretical guarantees that is suitable to online settings.

Experimental results show that MA-BIRDS is effective in both continuous and discrete environments. Despite its benefits, one limitation is the assumption of linear cost functions. We will explore approaches to relax this assumption in the future.

Potential negative social impact . Since MA-BIRDS can infer the experts' behaviors, potential negative social impact may occur if the learners are malicious. Take the evader-patroller setting as an example; the patrollers could be safeguards or park rangers, and the evaders could be poachers. The malicious poachers may use MA-BIRDS to escape. To avoid this situation, the experts should take additional strategies such as regularly demonstrating misleading behaviors so that MA-BIRDS will learn a wrong behavior model of the experts.

References

P. Stone and M. Veloso, 'Multiagent systems: A survey from a machine learning perspective,' Autonomous Robots , vol. 8, no. 3, pp. 345-383, 2000.

J. Verbraeken, M. Wolting, J. Katzy, J. Kloppenburg, T. Verbelen, and J. S. Rellermeyer, 'A survey on distributed machine learning,' ACM Computing Surveys , vol. 53, no. 2, pp. 1-33, 2020.

P. Luo, H. Xiong, K. Lü, and Z. Shi, 'Distributed classification in peer-to-peer networks,' in ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pp. 968976, 2007.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, 'Communication-efficient learning of deep networks from decentralized data,' in Artificial Intelligence and Statistics , pp. 1273-1282, 2017.

M. Bendechache and M.-T. Kechadi, 'Distributed clustering algorithm for spatial data mining,' in IEEE International Conference on Spatial Data Mining and Geographical Knowledge Services , pp. 60-65, 2015.

A. Bhaskara and M. Wijewardena, 'Distributed clustering via lsh based data partitioning,' in International Conference on Machine Learning , pp. 570-579, 2018.

L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. , 'Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures,' in International Conference on Machine Learning , pp. 1407-1416, 2018.

Y. Wang, M. Damani, P. Wang, Y. Cao, and G. Sartoretti, 'Distributed reinforcement learning for robot teams: A review,' Current Robotics Reports , vol. 3, no. 4, pp. 239-257, 2022.

S. Liu and M. Zhu, 'Distributed inverse constrained reinforcement learning for multi-agent systems,' in Advances in Neural Information Processing Systems , vol. 35, pp. 33444-33456, 2022.

P. Abbeel and A. Y. Ng, 'Apprenticeship learning via inverse reinforcement learning,' in International Conference on Machine Learning , pp. 1-8, 2004.

B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, 'Maximum entropy inverse reinforcement learning,' in National Conference on Artificial intelligence , vol. 8, pp. 1433-1438, 2008.

S. Zeng, C. Li, A. Garcia, and M. Hong, 'Maximum-likelihood inverse reinforcement learning with finite-time guarantees,' Advances in Neural Information Processing Systems , vol. 35, pp. 10122-10135, 2022.

S. Zeng, M. Hong, and A. Garcia, 'Structural estimation of markov decision processes in high-dimensional state space with finite-time guarantees,' arXiv preprint arXiv:2210.01282 , 2022.

S. Zeng, C. Li, A. Garcia, and M. Hong, 'Understanding expertise through demonstrations: A maximum likelihood framework for offline inverse reinforcement learning,' arXiv preprint arXiv:2302.07457 , 2023.

T. S. Reddy, V. Gopikrishna, G. Zaruba, and M. Huber, 'Inverse reinforcement learning for decentralized non-cooperative multiagent systems,' in IEEE International Conference on Systems, Man, and Cybernetics , pp. 1930-1935, 2012.

S. Natarajan, G. Kunapuli, K. Judah, P. Tadepalli, K. Kersting, and J. Shavlik, 'Multi-agent inverse reinforcement learning,' in International Conference on Machine Learning and Applications , pp. 395-400, 2010.

L. Yu, J. Song, and S. Ermon, 'Multi-agent adversarial inverse reinforcement learning,' in International Conference on Machine Learning , pp. 7194-7201, 2019.

D. R. Scobee and S. S. Sastry, 'Maximum likelihood constraint inference for inverse reinforcement learning,' in International Conference on Learning Representations , 2019.

S. Malik, U. Anwar, A. Aghasi, and A. Ahmed, 'Inverse constrained reinforcement learning,' in International Conference on Machine Learning , pp. 7390-7399, 2021.

N. Rhinehart and K. M. Kitani, 'First-person activity forecasting with online inverse reinforcement learning,' in IEEE International Conference on Computer Vision , pp. 3696-3705, 2017.

S. Arora, P. Doshi, and B. Banerjee, 'Online inverse reinforcement learning under occlusion,' in International Conference on Autonomous Agents and Multiagent Systems , pp. 1170-1178, 2019.

E. Hazan, K. Singh, and C. Zhang, 'Efficient regret minimization in non-convex games,' in International Conference on Machine Learning , pp. 1433-1441, 2017.

N. Hallak, P. Mertikopoulos, and V. Cevher, 'Regret minimization in stochastic non-convex learning via a proximal-gradient approach,' in International Conference on Machine Learning , pp. 4008-4017, 2021.

N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier, 'Model-based inverse reinforcement learning from visual demonstrations,' in Conference on Robot Learning , pp. 19301942, 2021.

S. Xu and M. Zhu, 'Efficient gradient approximation method for constrained bilevel optimization,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 10, pp. 12509-12517, 2023.

S. Ghadimi and M. Wang, 'Approximation methods for bilevel programming,' arXiv preprint arXiv:1802.02246 , 2018.

N. Couellan and W. Wang, 'On the convergence of stochastic bi-level gradient methods,' Optimization , 2016.

K. Ji, J. Yang, and Y . Liang, 'Bilevel optimization: Convergence analysis and enhanced design,' in International Conference on Machine Learning , pp. 4882-4892, 2021.

M. Hong, H.-T. Wai, Z. Wang, and Z. Yang, 'A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic,' arXiv preprint arXiv:2007.05170 , 2020.

T. Chen, Y. Sun, Q. Xiao, and W. Yin, 'A single-timescale method for stochastic bilevel optimization,' in International Conference on Artificial Intelligence and Statistics , pp. 24662488, 2022.

E. Altman and A. Shwartz, 'Constrained markov games: Nash equilibria,' in Advances in Dynamic Games and Applications , pp. 213-221, 2000.

D. Yuan, A. Proutiere, and G. Shi, 'Distributed online optimization with long-term constraints,' IEEE Transactions on Automatic Control , vol. 67, no. 3, pp. 1089-1104, 2021.

L. Wang, Q. Cai, Z. Yang, and Z. Wang, 'Neural policy gradient methods: Global optimality and rates of convergence,' in International Conference on Learning Representations , 2019.

H. Kumar, A. Koppel, and A. Ribeiro, 'On the sample complexity of actor-critic method for reinforcement learning with function approximation,' arXiv preprint arXiv:1910.08412 , 2019.

K. Zhang, A. Koppel, H. Zhu, and T. Basar, 'Global convergence of policy gradient methods to (almost) locally optimal policies,' SIAM Journal on Control and Optimization , vol. 58, no. 6, pp. 3586-3612, 2020.

K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Basar, 'Fully decentralized multi-agent reinforcement learning with networked agents,' in International Conference on Machine Learning , pp. 5872-5881, 2018.

S. Arora and P. Doshi, 'A survey of inverse reinforcement learning: Challenges, methods and progress,' Artificial Intelligence , vol. 297, p. 103500, 2021.

B. D. Ziebart, J. A. Bagnell, and A. K. Dey, 'Modeling interaction via the principle of maximum causal entropy,' in International Conference on Machine Learning , pp. 1255-1262, 2010.

S. P. Boyd and L. Vandenberghe, Convex Optimization . Cambridge university press, 2004.

B. D. Ziebart, Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy . PhD thesis, Carnegie Mellon University, 2010.

T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, 'Reinforcement learning with deep energy-based policies,' in International Conference on Machine Learning , pp. 1352-1361, 2017.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, 'Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,' in International Conference on Machine Learning , pp. 1861-1870, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of uncovering the incentives and behaviors of a cooperative multiagent system (MAS) that is optimizing for an objective subject to constraints. The authors consider solving this as a distributed optimization problem where several learners observe separate trajectories of the MAS in parallel and infer the parameters of the MASs reward functions and policies. The learners periodically pool this information to arrive at a consensus. This problem is posed as a bi-level optimization problem where the inner problem of recovering a joint policy given a reward function is solved in the dual, and the outer problem is to learn the reward function via maximum likelihood estimation. The authors prove a consensus will be reached and give a convergence rate. They also prove the agreed upon policy enjoys sublinear regret as measured by constraint violation. The authors demonstrate their approach on two domains, a discrete evader-patroller domain and a continuous drone domain. MA-BIRDS is on par with prior centralized approaches (e.g, BICS) in terms of uncovering correct constraints and high performing policies but with slightly shorter runtimes.\n\nSTRENGTHS:\nGiven the complexity of the setting, I thought the authors did an excellent job explaining their setup and solution clearly. They clearly acknowledge prior work and explain their contribution to inferring the behaviors of an MAS with a *distributed*, *online* approach. They support their algorithmic development with theoretical analysis and run experiments comparing against several strong baselines.\n\nWEAKNESSES:\nI feel the paper could better motivate the *distributed* problem. The experiments show centralized approaches are nearly as fast and with similar, if not better performance. Distributed approaches can often take advantage of parallelism to achieve faster convergence at scale, but I did not see that reflected in the theoretical rates, nor did I see experiments testing this.\n\nQUESTIONS:\n- Can you give intuition for why your theoretical convergence rates are independent of $N_L$?\n- One of the benefits of distributed learning is the ability to increase parallel computation to achieve faster learning. Did you vary $N_L$ at all for either experiment and can you include those results if you have them?\n\nLIMITATIONS:\nThe evader-patroller setting suggests the proposed method could be used in a way that has negative societal impact. Typically, we want the patrollers (e.g., park rangers) to capture the evaders (e.g., poachers). Can you discuss this briefly?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new type of the MA-IRL problem is proposed and analysed in this paper, which features online learning, distributed demonstrations and constraints independent of rewards. To tackle challenges resulting from these features, the authors design a novel algorithm called MA-BIRDS that utilises a single-loop mechanism to learn the underlying reward functions, constraints and policies. Due to the decentralised nature, the local regret is used to approximate the actual gradient. Authors upper bound the local regret at an ideal case and provide a theoretical guarantee on the achievement of a consensus over the learned rewards, policy and costs among all agents. Fruitful experiments are executed that validate the effectiveness of the proposed algorithm.\n\nSTRENGTHS:\nThe paper contributes a novel IRL algorithm that is adapted to streaming data and distributed multi-agent systems. The single-loop update of rewards and policies sets this work apart from the others in the literature in the sense that it can considerably alleviate the time pressure and is hence suitable for online scenarios. Theoretical results on convergence and consistency pin the proposed algorithm deeply down to reliability.\n\nWEAKNESSES:\nAlthough I agree that the single-loop update of rewards and policies is a significant contribution to the literature of IRL, I still doubt the online efficiency of the proposed algorithm because the soft-Q learning/soft actor-critic is involved in each round of updates. As far as I know, soft-Q learning or soft actor-critic is not a kind of fast-converging RL algorithm, or at least they cannot always guarantee fast convergence. This would potentially bring some negative effects on online efficiency when the problem scale is large or the online data comes with a complex structure. I would like to increase the score if the authors can make a discussion on this and/or provide some empirical evidence to clarify this.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MA-BIRDS, an Inverse Reinforcement Learning (IRL) framework designed to effectively learn multi-agent behaviors from distributed and streaming demonstrations in continuous and discrete environments. The authors initially frame the problem as a distributed online bi-level optimization problem. They subsequently present a fast distributed online gradient descent single-loop algorithm, accompanied by theoretical guarantees. To showcase the efficacy of MA-BIRDS, the paper includes a set of experiments conducted in both continuous and discrete environments.\n\nSTRENGTHS:\n- The paper demonstrates a commendable effort in addressing the problem at hand and provides a satisfactory solution.\n\nWEAKNESSES:\n- The problem being tackled seems incremental over the existing literature on distributed inverse constrained reinforcement learning.\n\nQUESTIONS:\n1. In reference to line # 43-44: Is it true that there is no existing work on online IRL with non-linear rewards? On a quick search here [1] is a work that works with quadratic rewards in online IRL. \n\n2.  For better readability, I would suggest replacing 'state-of-the-arts' with 'state-of-the-art works' throughout the paper. \n\n3. In reference to line # 66-67: What are the exact challenges in extending single-loop bi-level centralized optimization to de-centralized case? \n\n4. What is `N' in line #76?\n\n5. Is the paper the first one to propose a  'distributed online bi-level optimization problem'?\n\n6. What are downstream applications to the problem setup discussed in para 3 of Introduction? Is the main challenge in ICRL estimating local rewards with streaming data. What are the exact challenges in this? \n\n7. In general in dual optimization implementations, inner or outer optimization problems are not solved fully, how important is it to MA-BIRDS to solve any or all of the optimization problems fully? \n\n[1] Self, R., Abudia, M., & Kamalapurkar, R. (2020, July). Online inverse reinforcement learning for systems with disturbances. In 2020 American control conference (ACC) (pp. 1118-1123). IEEE.\n\nLIMITATIONS:\nThe authors include just one limitation about cost functions being linear. It would be nice to add a more comprehensive literature review.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tries to infer the behaviors of multiple interacting experts by estimating their reward functions and constraints. In their approach, the distributed demonstrated trajectories are sequentially revealed to a group of learners. The problem is set up as a distributed online optimization problem with 2 levels - the outer-level estimates the reward functions  - the inner-level problem learns the constraints and corresponding policies. The solution is called MA-BIRDS - multi-agent behavior inference from distributed and streaming demonstrations algorithm. The results guarantee that 1) the distributed learners achieve consensus on reward functions, constraints, and policies; 2) the average local regret (over N online iterations) decreases, and the cumulative constraint violation increases sub-linearly. A couple of examples illustrate that the approach appears to work well.\n\nSTRENGTHS:\nThe problem is very interesting - the approach seems sound, and the results (e.g. the 1) are solid with extensions of the SOA (e.g. [20-21]).\n\nThe simulation results compare to a good set of baselines\n\nWEAKNESSES:\nThe learned results (constraints) in Figure 2 seems similar, but the visitation frequencies (colors) seems quite different - pls explain. Is the goal being learned as a constraint in fig 2(b) correct?\n\nThe results (fig 3) showed the perf compared to several centralized algorithms, but it would be good to provide evidence that the distributed learning problem is actually hard - ie that a naive distributed learner would fail in this case - would lead to strong results.\n\nFor the drone example, with only 2 goals and a demonstration presumably showing the path to the appropriate goal, it is difficult to see why this problem is hard? Pls provide more evidence that this is a difficult problem that you are succeeding on.\n\nQUESTIONS:\nAre the experts of similar capability? The more interesting case seems to be when they have varied capability and deciding amongst which to use of them?\n\nCan the authors comments on the tightness of the bounds in the contribution statement - especially in the numerical results section?\n\nLIMITATIONS:\nsome discussion", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}