Distributed Inverse Constrained Reinforcement Learning for Multi-agent Systems
Authors: Shicheng Liu
Minghui Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: inverse reinforcement learning
distributed bi-level optimization

Abstract: This paper considers the problem of recovering the policies of multiple interacting experts by estimating their reward functions and constraints where the demonstration data of the experts is distributed to a group of learners. We formulate this problem as a distributed bi-level optimization problem and propose a novel bi-level ``distributed inverse constrained reinforcement learning" (D-ICRL) algorithm that allows the learners to collaboratively estimate the constraints in the outer loop and learn the corresponding policies and reward functions in the inner loop from the distributed demonstrations through intermittent communications. We formally guarantee that the distributed learners asymptotically achieve consensus which belongs to the set of stationary points of the bi-level optimization problem.

Distributed Inverse Constrained Reinforcement Learning for Multi-agent Systems
Shicheng Liu & Minghui Zhu School of Electrical Engineering and Computer Science
Pennsylvania State University
University Park, PA 16802, USA
(sf15539,muze muz16 @psu. edu
Abstract
This paper considers the problem of recovering the policies of multiple interact- ing experts by estimating their reward functions and constraints where the demon- stration data of the experts is distributed group of learners. We formulate this problem as a distributed bi-level optimization problem and propose a novel bi-level "distributed inverse constrained reinforcement learning" (D-ICRL) algo- rithm that allows the learners to collaboratively estimate the constraints in the outer loop and learn the corresponding policies and reward functions in the inner loop from the distributed demonstrations through intermittent communications. We formally guarantee that the distributed learners asymptotically achieve con- sensus which belongs to the set of stationary points of the bi-level optimization problem. Simulations are done to validate the proposed algorithm.
1 Introduction
Multi-agent systems (MASs) have become an effective way to model large-scale networked systems, egg. multi-robot systems and the Internet-of-Things. Multi-agent reinforcement learning (MARL) is a systematic framework to coordinate MASs. It extends RL address the decision-making problem of multiple interacting agents [1]. One challenge in (MA)RL is that manually crafting the reward functions or predesigning the policies can be infeasible for humans when the tasks are too compli- cated. Inverse reinforcement learning (IRL) presents way to tackle this problem where learner aims to recover reward function (and a corresponding policy) that best explains the behavior in the demonstrations of an expert. Current mate-of-the-ane reward function inference problem but still face two other major challenges in MARL and MASs: (i) it is often the case that the expected behavior (experts' behavior) is more precisely explained by a reward function combined with a set of constraints instead of a single reward function; (ii) the demonstration data MASs is usually distributed over multiple learners and unable be shared due to some practical reasons including privacy concerns and communication burdens. This paper aims to bridge the gap and proposes the first algorithm solve the three challenges (i.e., reward function inference, constraint inference, and distributed demonstration data) at once.
IRI. [2, 3, 4, 5] can successfully solve the
Related works. IRL is ambiguous problem [6] as different reward functions could explain the demonstrations. To solve this problem, several fundamental approaches are proposed including feature expectation matching [7], maximum margin planning [8], maximum entropy (ME) [9], and maximum causal entropy (MCE) [10]. Recently, some advanced techniques baring more features proposed. Paper [2] adversarial learning to learn reward functions robust environmental changes, paper [4] applies strongly convex regularizers to the learner's ' policy to avoid the ambiguity problem, and paper [5] proposes scalable Bayesian-basee baaed method working for large control tasks.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In real-world settings, numerous constraints are present and single reward function is not enough to explain complicated behavior. Paper [11] extends ME IRL to environments with constraints, formulates the constraint inference problem as a maximum likelihood estimation (ML.) problem, and leverages a greedy method to solve the problem. Paper [12] extends paper [11] to continuous environments. These two works both assume the access to the ground truth rewards.
While the discussions in the last two paragraphs are limited to single agents, there have been recent works which extend IRL techniques to multiple agents [13, 14, 15, 16, 17J. Notice that previous works on multi-agent IRL do not explicitly distinguish the notions of "experts" and "learners", their "multiple agents" means "multiple experts" and they only consider centralized learning (i.e. a single learner who obtains all demonstrated behavior of the multiple experts). In this paper, we explicitly distinguish the notions of "experts" and "learners", and use "multiple experts" to represent the commonly used "multiple agents" in literature. Moreover, the aforementioned papers do not consider the more general situation where the demonstrations are distributed over multiple learners and unable to be shared. To handle this situation, the learners need to perform distributed learning.
As basic topology in distributed learning, peer-to-peer (P2P) architecture has no central server and each node can communicate with a subset of all the nodes. It reduces the risk of single points of failures [18] in the system. This paper proposes D-ICRL based on the P2P structure where ICRL can learn policies by estimating the reward functions and constraints through demonstrations and distributed learners can perform ICRL without sharing the distributed demonstration data.
In specific, formulate D-ICRL distributed bi-level optimization problem where the learners cooperatively learn the constraints through the outer problem and estimate the corresponding reward functions and policies through the inner problem. Paper [19] provides asymptotic convergence for centralized bi-level optimization problems by assuming that the inner problem can be completely solved and its optimal solution can be obtained. Papers [20. 21, 22] relax this assumption by partially solving the inner problem and provide convergence rate analysis for problems with strongly convex inner objective function. Paper [23] solves a special case of distributed bi-level optimization where the outer and inner problems use identical decision variables. To the best of our knowledge, there is no work on bi-level optimization which (i) assumes that the inner objective function is merely strictly convex and partially solves the inner problem; (ii) solves the general form of bi-level optimization in distributed way. Notice that these two challenges arise in our D-ICRI, problem.
Contribution statement. Our contributions are threefold. First, we extend IRL to a multi-expert- multi-learner (MEML) setting where group of learners collaboratively learn policies by estimating the reward functions and constraints without sharing the distributed demonstrations performed by a set of cooperative experts. We formulate this D-ICRL problem as a distributed bi-level optimization problem. Second, we propose a distributed bi-level learning algorithm to solve this problem where the learners cooperatively estimate the constraints in the outer problem and learn the corresponding reward functions and policies in the inner problem. Third, we provide convergence rate analysis to the optimal solution of the inner problem and asymptotic convergence the set of stationary points of the outer problem. Simulations are conducted to validate the proposed method.
2 Model
In this section, we present the models for cooperative experts and distributed learners.
Experts. There are NE experts whose decision making is based on constrained Markov game (CMG) [24]. A CMG (S, 7, Po, P,rE, CE, b) is defined via state set S & I Na sti,, action set A & I Alt,, discount factor 7 E (0, 1), and initial state distribution Po- The state-action space can be either discrete or continuous. The system state transition function is given by P such that the probability (or probability density in continuous state-action space) of state transition to s' from s by taking action - (a!")..... (Nllll is P(s'ls,a), a). The reward function of expert i I6 is TE S x A + R and the experts are cooperative (i.e., rg A -ENrr!) (il).. The cost function x A + RR+ and the cost function of the whole system cÆ - >iNe [i]] of expert i is of (6) (ali))) [)) represents the probability (or probability density) of expert taking Expert i's policy TE
action alt at state and the joint policy of all the experts is denoted by (=(a]s) - II Th
[6]] (a)))s). Define (T) A F3 D 7'rE(S, A.) as the expected cumulative reward where the initial
2

---

state is drawn from Po and (T) A Yeg (St, A.) as the expected cumulative cost.
The experts' policy gE wants to maximize Jrg (T) subject to Jag (x) I b. Define the constraint set indicated by cg as ch. A ((x,a) ) E SSx A of (x, > 03. Following [12], we study the case where b 0, i.e., hard constraint. Thus, the constraint reduces to Jeg (T) = 0, implying that the probability of reaching any (8,a) E U.C. is zero under policy я. These experts use wE to demonstrate set D - {{' of m trajectories, which is partitioned into N2 subsets. Each trajectory Ç _ sy. ab st a} is a state-action sequence of the experts.
Learners. There is group of Ny learners where each learner e knows (8, A, Po, b. Dlv)) and communicate with other learners. The local demonstration subset Diel has m trajectories. The learners choose an ! -dimensional reward feature vector of :S x + [0, di for expert where d is constant and it is assumed that E = -W// ) Top where w Rd is the (unknown) reward weight vector. The learners also choose an !! ! -dimensional cost feature vector a! :S A + (wee ) oi" where wer is the (unknown) cost weight vector and w/g.: the j-th component of wi! Let ( {(s,a) E A:::A & (8,a)
[d]
[0, d2) where d2 is constant and it is assumed that CF.
[1]
0} be the constraint set indicated by the j-th cost feature db of expert i. As the budget b = 0, the
effect of ( cg.j > O is the same, i.e., c/Ji cannot be visited, thus the learners assume w|dl CE.J E [0, 1].
Communication network. The learners execute a two-time-scale learning algorithm over a com- munication network. The fast-time-scal (inner) algorithm is executed at discrete time k over time-varying directed graph ((k) & (K)) where V {1,... NN is the node (learner) set and E(k) S VV x V the set of directed edges (communication links) time k. The edge (v, v') E E(k) means that learner e receives information from learner v' at time l and (v, v) E E(k) for all k 2 O. NRNN where
The adjacency matrix of the graph at time & is ((k) A [Wlve'I (k.)) (iv''l(k) 0 and only if (v, v') E(k). We use (f))(k) & {v' E (I v') E(k) denote the set of neighbors of learner at time k. The slow-time-scale (outer) algorithm is carried out at discrete time n over a graph ((n) A (V, (n)), where E()). (n) and Vie are defined in an analogous way. The time unit for a much larger than that for k.
Assumption 1. There exists an integer B 2 such that the graph (V, U U... .EKK + B 1)) strongly connected for all k 2
Assumption 2. The adjacency matrix (kk) has the the following properties: (i) (doubly stochastic) T ( 1 and W (k)) = 1 where is the column vector whose entries are all ones. (ii) (non-degenerate) There is an E (0, such that W Wivo] vo) (k) 2 e eorrr all E V and wiw'') (Ik) 2 e (v, u') E(k).
Assumption 3. The set [Tl)(() = for any E V and n 2 0. The adjacency matrix W(n) is doubly stochastic and non-degenerate with constant e.
multi agent system
D-CR
decision maksing (tt)
distribated demensoaaion
Expert
Leamer I
(19)
Dly
Expert i
globul dernonstration
Learner v
(r,c,
Diag
Expert NE
Leamer N
(72, Gc)
Figure 1: Relation between experts and learners
Figure/ provides an illustration of the relation between the experts and the learners. The experts operate in muelti-agent system and demonstrate set D of trajectories and the learners stand outside the system where each learner can observe subset Divl of the global demonstration.
3

---

3
Notions and notations
This section provides notions and notations which will be used in the remaining of the paper.
Reward feature expectation matching (RFEM). Given policy æ. the reward feature expectation vector of expert i is defined as [8] (I)) E3, A Ei= y ol Ar)] and the demonstration D is needed to estimate f4 (Tg): p, A 1. EE= Ei=o 2 o (s], a²),, where (s/,aa) E Ç². The RFEM requires μ (T) = أق for every expert i. However, each learner v can only observe the subset Di and its local estimate g. A
mlo) EeSEDI Ei= 7 (s). a²).
Cost expectation matching (CEM). Similar to RFEM, CEM requires ( Joe (TT). As the true cost function eg is unknown, given an estimate Cac Em (w(")) Top we define the empirical cost estimate of the experts Bw. - 1 m [le] m. EsSEDI Ei Co.
D}= Ei=o Cw,, (6), a.). The local estimate from learner
is Owe
Remark 1. As RFEM is standard notion in IRL literafure, it is "natural" to follow this and use "cost feature expectation matching" (CFEM). However, here we use CEM instead of CFEM because we want to leverage the budget information provided in CMG. We still use RFEM instead of reward expectation matching" " (REM) because REM does not embed any additional useful information as CMG does not have any "budget information" " for cumulative reward.
Notations. We use jr, (T .... ((i) TT] to represent the concatenated empirical reward feature expectation vector. Similarly, we define learner v's concatenated empirical reward feature expectation vector estimate p!") concatenated reward feature expectation vector (T) under T, and concatenated cost feature vector de. Let II be the set of all valid stochastic policies such that every E II satisfies πT (als) 2 0 for any E S and E A and SaeA <[als)da = 1 for any 8 E S. We define the expected cumulative cost of policy æ under cost function estimate Ca as Jac (T) A E ce.. (S,, At) and use Project,, (.) to denote the projection onto the set Ω2.
4 Problem formulation
In an MEML D-ICRL, problem, each learner u aims to use all the available information to collab- oratively recover the policy øe through communications by estimating the reward function rE and the cost function cg- In the learning process, learner e is unwilling to share its private data (i.e., local demonstration and local estimate of reward feature expectation and cost expectation). Notice that estimating cg is in essence estimating the constraint set U, CE. learning problem distributed MIE problem:
We formulate the constraint
No
max w. E..
F(we)
Σ
pi (wec),
(w.) EjEDi Ei= 7¹ In Tw. (a/ ss/) is learner v's local likeli- hood function and Tw, is policy parameterized by we. To find a statistical distribution model for Twa we formulate the following problem based on MCE scheme:
where Q. _ [0, 1j. 19
max NET
(T),,
s.t.
pli (x) = pl! VIEI, Ju. (T) = =b.r
(2)
where H(x) Eie Es,Al->' n (A.[s.) is the (infinite-horizon) causal entropy [25].
Remark 2. Directly matching the ground truth budget (ie., Juc = b) may cause infeasibility of
(2), i.e., the feusible set of (2) is not guaranteed to be nonempty. Therefore, we mutch b which is an estimate of Ja., () 10 ensure the feasibility because now, TE is a feasible solution of (2).
The dual function of (2) G(w,, X;w)) A max H(T) + +
i=1 NE (w)lt)) (T)))T(()) (x) (1.3) (Jw.
(wf ( is to estimate Wrs. Let n A WT,TT ^AT_ it is well-known that G(mwe) is convex in 7 since it is the pointwise maximum of family of affine functions of m. As G(T w.) is global, we introduce the local convex dual function
b...) where the dual variable Wp [(w))))
glvl (m:: w.) e max H(x) + Ne (w. () ( (()
al!.e)) A(Jur. -l.)).

---

Lemma 1. (i) The optimal solution of problem (2) is the constrained soft Bellman policy (we)))) where n1 (we) is optimal solution of min, (1) (ii) The local dual function Glol (n; wo) is
differentiable and its gradient is [(r (Trwe)) 23 (1 TT, (.
1T.
The proof of Lemma is similar to [10. 25, 26]. For the sake of completeness. we still include the proof and the expression of the constrained soft Bellman policy in the Appendix.
Lemma I indicates that (wo))o. the statistical model need in problem (1). Therefore, prob- lem (1) can be reformulated as the following distributed bi-level optimization problem:
max we weefe
F(west 1/" (we))
Flol (Wer
(Wer 1f" (Wo)),
N ol G (17 wo),
s.t. 1/ (we) = arg min
(4)
where F(y) and mlig are known only to learner U.
Remark 3. Problem (4) and arg min, G(r) are equivalent because Eve, ml
EN N2 ml H (Togaee) Σ Ny m(t) E (wl T (Tmpue) (1l:00) + Exe m (Trrwe)
mH (Tppae)) + m Li=1 (w]) (ur (Tapee) - pi + Am (nn
(Ta - bw.) = mG(TTwe))
5
Algorithm and convergence guarantee
In this section, we develop bi-level distributed learning algorithm solve problem (3)-(4) and pro- vide convergence rate to the optimal solution of the inner problem (4) and asymptotic convergence the set of stationary points of the outer problem (3).
Algorithm MEML D-ICRL
Input: {w) [o| (0) {0)) Enple (0)) ((:),, (())
Output: w. (n), (n), (l)),)//((ae (wec" (n)), x
Vo
for = 0,,.... do
2: for v do
3: Receives w!! (m) from e' e (n)
(l" [v] (n)). (m))))) Inner (wl (n))
4:
5: if = then
( A (wo!") (0), (lo (0)))
6:
7: else
8:
Receives Vio'l (n)(()) = Enee, (n)
from o' E (T))())
1) + (olo)(())))))) (n), (n)))
9:
Flll (n) - 1), 7h (wv (n (n-1)))
10:
end if
ا.!l (n) = Project
we (n) + N₂ (l)((m)
11:
|o| (n + 1/2) = [ (n) + 3(n) (n) w (l))())) Receives w!!
12:
[1'] (n + 1/2) from v' E ((e((n)
14: w.l (n 1) ΣNz= Wlo'') (n)ws 10') (n 1/2)
No.
15: end for
16:
end for
At outer iteration n in Algorithm 1, the learners first cooperatively solve the inner problem for each learner through Inner process (lines 3-4) and then use the obtained result 7l (w (n)) (w)))))) to
5

---

Inner process. As (n) V, learner knows the cost weight vector of all the learners. Given we at inner iteration k, each learner receives 17 1 (w!!) k) from neighbors and updates its estimate of the reward weight vector (embedded in n)!")). In specific, at line in Algorithm the first term (convex combination) on the right hand side encourages consensus and the second term (gradient) drives the optimal solution. It is shown Lemma that each learner can reach 17" (we)) at the collaboratively solve the outer problem (3) through the outer process (lines 5-14). In what follows, we will elaborate each process.
Algorithm 2 Inner process(w!"))
Input: wt") Output: (w( AppM (u(")
Nz, ((k)
1: for k == 0. 1,. KK- do
2: T do
3: Receives n1 ((ol(w k) from o' (
4: (ws h + = - w (k))) (wb)),)) k) aa(k)mll)) v (y) k); k):
5: end for
6: end for
7: (w)!") & Ex (w))))
Σt= o(A)
8: " )wl is the corresponding constrained soft Bellman policy.
(oly
rate of ( Vlog K
Outer process. Once obtaining rll!() each learner v faces three challenges in solving the outer problem: (i) the gradient of (w! .7" (wol:)) cannot be obtained because the inner problem not fully solved, i.e., learner does not find 17" (wel but its approximation r7 (we(") ), (ii) Ev Fll1 unknown, and (iii) the local likelihood function Fivl is non-convex. solve these challenges respectively, propose three techniques: local gradient approximation (LGA), global gradient approximation (GGA), and local successive convex approximation (LSCA). By subtly designing these three approximations, Notice that the second challenge also arises in the inner problem but we solve it in a simpler way because the inner problem is convex.
algorithm is guaranteed to converge to the set of stationary points.
(we)), we propose the following LGA of learner To approximate the local gradient V F (we, 17"
F (lu,, 7(4z)) = Ede E 7 (s(ss, ml F pa (E (S,, At)], whose derivation is based on the strict convexity of G(T: we) and (m: The derivation and the proof of strict convexity both included in the Appendix. Moreover, it can be shown (in the Appendix) that the approximation error (defined in the Appendix) upper bounded and reduces to zero as K
While learner e can use LGA get a sound approximation of V F(o),, in order to solve prob- lem (3), it still needs the knowledge of the gradients En'ee p of other learners to get the global gradient. Therefore, we leverage GGA Vivl (lines 5-10) who aims to track the average global gradient Nz
1 VF(W!"), "1" (we However, VF
[1] "1 (wd!))
not available
due to the inaccessibility of 17" (w!!)) Therefore, रiol is designed to track the approximation of the average global gradient.
Nr Dw'.
To tackle the non-convexity issue, learner solves LSCA problem (line 11) and then follows two-step updates of the local cost weight vector !!!l LSCA proposes solve a local convexifica- tion of problem (3). In specific, learner solves the problem arg max en_ [ir w(ll) where
-fllll (w") is a local strongly-convex surrogate of -F(wer 11" (w:)) at w!!l satisfying the gra- dient consistency condition V Fi (wo) Nr iiu, An attractive property of LSCA [27] is

---

that its fixed point exists and is stationary point of functions with gradient N₂ V under certain conditions (in the Appendix).
Once obtaining We" learner executes two-step updates. The first update towards we (line 12) drives learner stationary point of problem (3) and the second update (lines 13-14) encourages the consensus among different learners.
Lemma 2. The result a!!) (n) line 11 in Algorithm I the optimal solution of LSCA problem at we (t) (n) under the GGA N (V)..))
Lemma 3. (Convergence rate of the inner problem) Suppose Assumptions and 2 hold and let where ā is positive constant. Then, for every learner v E V in Algorithm 2.
(kk) R+1
clu)
clu
|1 (we) n7" (w=)l
log K
Vlog K
where cl and clu) are positive constants whose expression can be found in the Appendix.
We define the set of KKT points (also called stationary points) of problem (3)-(4) as [28] [29]: 22 - {wr Ω2 Ef, [(VF(w n² (w²))) (we - w()} 0}, and study the convergence property of the following metric [30]: (w)( 2 max {(V F (wa(n). (we(r))) (w. - we(n))).
Theorem 1. (Asymptotic convergence of the outer problem) Suppose Assumptions
and hold. Let (n)) E (0, E 3(n) = too, and En (3()) < +o0, then in Algorithm 1:
(consensus)
lim max |W! (n) - wo" (n)]l = 0.
(convergence)
lim sup [w (n)) < Vlog K
where M is a positive constant whose existence proved in the Appendix.
Lemma 3 and Theorem I show that the distributed learners can achieve common values of w. and 1)- As the constrained soft Bellman policy is continuous in (we,, 77) (proved in the Appendix), they will also achieve common policy.
Simulation
This section presents two simulation examples. In the first example, the experts are programmed to follow the optimal policy under discrete state and action spaces. In the second example, the experts humans, i.e., may not be optimal, and the states and actions are both continuous.
6.1 Synthetic grid world
We consider the grid world introduced in [11]. Different from paper [11]'s MDP, we imple- ment a CMG on this environment where three experts perform motion planning from su/s// / to sa/s///"" while avoiding collision with obstacles and each other. The environment consists of a 9-by-9 grid states, and the actions of each expert are to stay still, move up, down, left, right, or diagonally by one cell. The dynamics is stochastic each action of an expert has 20% probabil- ity of failure, resulting in staying still. While the grid world looks small, the CMG has more than 387, 000, 000 state-action pairs due to multiple experts. Each state-action pair produces distance- relevant reward feature and the ground truth reward of each expert increases if the distance to its goal decreases. The true CMG (Figure 2a), from which the experts generate demonstration, in- cludes constraints (i.e., obstacles) and the nominal MG does not. We use heat maps to show the relevant visitation frequency of each state and action pair. Each expert will terminate its motion if violating constraint, i.e, colliding with an obstacle.
In our algorithm, there are four distributed learners where each learner can only respectively get 10, 20, 30, 40 demonstrated trajectories of the experts. Notice that the learners and experts are not the same entities, i.c., the learners stand aside observing the experts. The four learners jointly
7

---

recover the experts' policies by learning the constraints and the reward functions over an underlying communication network. The detailed simulation setup included in the Appendix.
Furthermore, we use the following three baselines for comparisons:
ME-greedy: This method is introduced in [11] where greedy method is used to estimate the constraints based on the ME framework.
MCE-greedy. We construct this method by extending ME-greedy the same greedy method based on the MCE framework which is more suitable for stochastic environments.
Centralized inverse constrained inverse reinforcement learning (C-ICRL): This method special case of D-ICRL where there is a centralized learner obtaining all the demonstration data.
Notice that ME-greedy and MCE- MCEE-gredyy assume the access to the ground truth rewards while C- ICRL does not have this assumption. All the three baselines are centralized learning where a single learner get the total 100 demonstrated trajectories. Our algorithm is distributed learning and does not obtain the ground truth rewards, therefore, it solves more difficult problem than the baselines do. However, the simulation shows that our algorithm can be on par with and even outperform the baselines in some aspects. In our bi-level algorithm, the outer loop is to estimate the constraints and the inner loop is recover the corresponding policy by estimating the reward function. To reason about the performance of our algorithm, we use five metrics; cumulative rewards (CR), false positive rate (FPR), false negative rate (FNR), constraint violation rate (CVR), and reward function distance (RFD). The CR. commonly used in IRL and imitation learning literature [31, 32], illustrates the total reward collected by the three experts in an episode, FPR, introduced in [11]. the proportion of learned constraints that are not the ground truth constraints, FNR is the proportion of the ground truth constraints that are not learned, CVR [12] the average percentage of the three experts' violating any constraint in an episode. Since the reward function is linear combination of the reward features, we propose RFD to measure the distance between the learned reward weight vector w and experts' reward weight vector wrg || Wrg I1/| 1/|l
Stabes
3.
30
2.0
inner Iterationc
(a) True CMG
(b) Learned CMG
(c) CR
0.12
3.1
0.00
1.4
3.2
0.58
3.30
375
10
29
30
28
20
20
45
10
26
45
3.0
(d) FPR
(e) FNR
(f) CVR
(g) RFD
D-Leamer
D-Learner
Experts (nominal)
ME-greedy
D-Leamer
D-Learner 4
Experts (ground truth)
MCE-greedy
Figure 2: Algorithm performance synthetic grid world CMG. In subfigures 2a 2b, each state and action cell colored according to the relevant visitation frequency and the red crosses represent the constraints. In subfigures 2c-2g, D-learners 1-4 distributed learners in algorithm. Because ME-greedy and MCE-greedy are not bi-level, we only include their curves against outer iterations.
Figure 2b shows the constraints learned by our algorithm (each learner recovers the same con- straints). Several action constraints are not recovered due to the reason that the experts will barely take these actions even if they are feasible. Figures 2f show that our learning algorithm can suc- cessfully imitate the experts' behavior according to CR and CVR. Figures 2d 2e show that D-ICRL

---

and C-ICRL have better FPR performance compared to the greedy- methods while MCE- greedy has the best FNR performance. From Figure 2e and the small window in Figure 2c, we can see that the distributed learners can quickly achieve consensus even if they start from different initial conditions in both outer and inner loops. Figures 2c and 2g show the performance of inner process in the last outer iteration. Figure 2g shows that D-ICRL. and C-ICRL can approach the ground truth reward functions and stabilize.
Table I: Performance Comparisons. Here, D means distributed and NATR means access to the (ground truth) rewards.
D
NATR
CR
FPR
FNR
CVR
RFD
Experts (ground truth)
N/A
N/A
2002.59 125.71
N/A
N/A
0.000
0.000
D Learner
4
1890.67 135.22
0.00
0.18
0.013
0.748
D-ICRL (our
D D-Larmer 2
7
1873.07
1 162.14
0.00
0.18
0.020
0.750
method)
LeLameer 3
7
1878.51 131.55
0.00
0.18
0.020
0.749
D-Lemeerr 4
7
1884.85 167.16
0.00
0.18
0.010
0.749
C-ICRL
x
/
1884.74 + 158.90
0.00
0.18
0.013
0.731
ME-greedy
1162.24 132.66
0.18
0.24
0.010
N/A
MCE-greedy
1776.84 300.61
0.09
0.057
N/A
Experts (nominal)
N/A
N/A
18.64 + 3.02
N/A
N/A
1.000
0.000
Table shows the comparison result where each learner in our distributed algorithm can be on par with C-ICRL in all the five metrics. Our algorithm outperform ME-greedy introduced in [11] in almost every metric except being on par with in CVR. While MCE-greedy is centralized learning and assumes the access to the ground truth rewards, method can still outperform it in three metries, i... CR, FPR, and CVR.
6.2 Drones motion planning with obstacles
In the second example, we simulate in a physical setting with continuous state-action space. We build simulator in Gazebo where each drone aims reach the door in its diagonal direction while avoiding collisions. We first control the simulated drones to their target doors, record nine pairs of trajectories, and distribute four and five pairs two learners respectively. The demonstration is shown in Figure 3b and each cell in the figure represents the constraint set indicated by a cost feature of each expert, i.e., e/.,. We define the reward feature of each state as its location multiplied with coefficient. Figure 3e shows the learned constraints and trajectories.
(a) Simulator environment
(b) Demonstrated trajectories
(c) Learned trajectories
Figure 3: Drones motion planning with obstacles
We keep the three metrics, FPR, FNR, and CVR last example. Because the ground truth reward not accessible in this physical setting, cannot use RFD and we success rate (SR) instead

---

of CR. The SR is defined as the percentage of the drones' successfully reaching the doors without any collision. We use the reward function learned from D-ICRL the input reward function of MR-greedy. Because this environment is deterministic, MCE-greedy reduces to ME-ggredy.
Table Performance Comparisons.
D
NATR
FPR
FNR
CVR
SR
D-Learner
V
v
0.028
0.00
0.03
0.97
D-ICRL
D-Learner
0.028
0.00
0.03
0.97
C-ICRL
X
0.018
0.00
0.02
0.98
M(C) greedy
x
x
0.037
0.13
0.16
0.84
7 Discussion and future work
We propose D-ICRL, the first mulli-exper--mult-llaaeeer MCE ICRL framework that is effective to CMG with both discrete and continuous environments. We employ MLE to estimate the constraints, use an MCF--based optimization problem to learn the corresponding reward function and policy, and derive our algorithm based novel distributed bi-level optimization theoretical framework. Experimental results show that D-ICRL can imitate the experts' behavior as well as recover the environmental constraints. Despite its benefits, the limitations are that (i) the learners assume that the reward/cost function is linear combination of some features; (ii) the communication between learners is frequent. We will address the issues in the future.
8 Acknowledgements
This work is partially supported by National Science Foundation through grants CNS 1830390, ECCS 1846706, and ECCS 2140175. We would like to thank the reviewers for their insightful and constructive suggestions.
10

---

References
[1] K. Zhang, Z. Yang, and Başar, "Multi-agent reinforcement learning: A selective overview of theories and algorithms," Handbook of Reinforcement Learning and Control, pp. 321-384, 2021.
[2] J. Fu, K. Luo, and S. Levine, "Learning robust rewards with adverserial inverse reinforcement learning." in International Conference on Learning Representations, 2018.
[3] L. Yu, J. Song, and S. Ermon, "Multi-agent adversarial inverse reinforvement learning," in International Conference on Machine Learning, pp. 7194-7201, 2019.
[4] W. Jeon, C.-Y. Su, P. Barde, T. Doan, D. Nowrouzeezhrrai, and J. Pineau, "Regularized inverse reinforcement learning," in International Conference on Learning Representations, 2020.
[5] M. Imani and S. F. Ghoreishi, "Scalable inverse reinforcement learning through multifidelity Bayesian optimization," IEEE Transactions on Neural Networks and Learning Systems, pp. 1- 8, 2021.
[6] A. Y. Ng and S. J. Russell, "Algorithms for inverse reinforcement learning," in International Conference on Machine Learning, pp. 663-670, 2000.
[7] P. Abbeel and A. Y. Ng, "Apprenticeship learning via inverse reinforcement learning," in In- ternational Conference on Machine Learning, pp. 1-8, 2004.
[8] N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich, "Maximum margin planning," Interna- fional Conference Machine Learning, pp. 729-736, 2006.
[9] B. D. Ziebart, L. Maas, J. A. Bagnell, and A. K. Dey, "Maximum entropy inverse rein- forcement learning," in National Conference Artificial intelligence, vol. 8, pp. 1433-1438, 2008.
[10] B. D. Ziebart, J. A. Bagnell, and A. K. Dey. "Modeling interaction via the principle of maxi- mum causal entropy," in International Conference on Machine Learning, pp.
2010.
[11] D. R. Scobee and Sastry, "Maximum likelihood constraint inference for inverse reinforce- ment learning," in International Conference on Learning Representations, 2019.
[12] S. Malik, U. Anwar, A. Aghasi, and A. Ahmed, "Inverse constrained reinforcement learning." International Conference on Machine Learning, pp. 7390-7399, 2021.
[13] S. Natarajan, G. Kunapuli, K. Judah, P. Tadepalli, K. Kersting, and J. Shavlik, "Multi-agent inverse reinforeement learning," in International Conference on Machine Learning and Appli- cations, pp. 395-400, 2010.
[14] T. S. Reddy, V. Gopikrishna, G. Zaruba, and M. Huber, "Inverse reinforcement learning for decentralized non-cooperative multiagent systems," " in IEEE International Conference on Sys- tems, Man, and Cybernetics, pp. 1930-1935, 2012.
[15] Inga, E. Bischoff, F. Köpf, M. Flad, and S. Hohmann, "Inverse cooperative and non- cooperative dynamic games based on maximum entropy inverse reinforcement learning," arXiv preprint arXiv; 1911,070000 2019.
[16] W.-C. Ma, D.-A. Huang, N. Lee, and Kitani, "Forecasting interactive dynamies of pedes- trians with fictitious play." in IEEE Conference on Computer Vision and Pattern Recognition, pp. 774-782, 2017.
[17] K. Bogert and P. Doshi, "Multi-robot inverse reinforcement learning under occlusion with interactions." in International Conference on Autonomous Agents and Multi-agent Systems, 173-180, 2014.
[18] J. Verbraeken, M. Wolting, J. Katzy, J. Kloppenburg, T. Verbelen, and J. S. Rellermeyer, "A survey on distributed machine learning." ACM Computing Surveys, vol. 53, no. 2, pp. 1-33, 2020.

---

[19] N. Couellan and W. Wang, "On the convergence of stochastic bi-level gradient methods," Op- timization, 2016.
[20] Ghadimi and M. Wang, "Approximation methods for bilevel programming," arXiv preprint arXiv: 17Xi::1802.02245 2018.
[21] M. Hong, H.-T. Wai, Z. Wang, and Z. Yang. "A two-timescale framework for bilevel optimiza- tion: Complexity analysis and application to actor-critie," arXiv preprint arXiv:2007.05170, 2020.
[22] K. Ji, J. Yang, and Y. Liang, "Bilevel optimization: Convergence analysis and enhanced de- sign," " in International Conference on Machine Learning, pp. 4882-4892, 2021.
[23] Yousefian, "Bilevel distributed optimization in directed networks," in American Control Conference, pp. 2230-2235, 2021.
[24] E. Altman and Shwartz, "Constrained markov games: Nash equilibria," in Advances in Dynamic Games and Applications, pp. 213-221, 2000.
[25] Z. Zhou, M. Bloem, and N. Bambos, "Infinite time horizon maximum causal entropy inverse reinforcement learning," IEEE Transactions on Automatic Control, vol. 63, no. 9, pp. 2787- 2802, 2017.
[26] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, "Reinforcement learning with deep energy- based policies," International Conference on Machine Learning, pp. 1352-1361, 2017.
[27] F. Facchinei, G. Scutari, and S. Sagratella, "Parallel selective algorithms for nonconvex big data optimization," IEEE Transactions Signal Processing, vol. 63, 7, pp. 1874-1889, 2015.
[28] P. Bianchi and J. Jakubowicz, "Convergence of multi-agent projected stochastic gradient algorithm for non-convex optimization." IEEE Transactions on Automatic Control, vol. 58,
no. 2, pp. 391-405, 2012.
[29] P. Di Lorenzo and G. Scutari, "Next: In-network nonconvex optimization," IEEE Transactions on Signal and Information Processing over Networks, vol. 2, no. 2, pp. 120-136, 2016.
[30] H.-T. Wai, J. Lafond, A. Scaglione, and Moulines, "Decentralized frank-wolfe algorithm for convex and nonconvex problems," IEEE Transactions on Automatic Control, vol. 62, no. 11, pp. 5522-5537, 2017.
[31] X. Wang and D. Klabjan, "Competitive multi-agent inverse reinforcement learning with sub- optimal demonstrations," in International Conference Machine Learning, pp. 5143-5151, 2018.
[32] V. Tangkaratt, N. Charoenphakdee, and M. Sugiyama, "Robust imitation learning from noisy demonstrations," in International Conference on Artificial Intelligence and Statistics, pp. 298- 306, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] We have the formal contribution statement in Section
1.
(b) Did you describe the limitations of your work? [Yes] We discuss the limitations in
Section 7.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? YYss]
2. If you are including theoretical results...
12

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes] We have the formal assumption statements in Section 2
(b) Did you include complete proofs of all theoretical results? [Yes] We include all the proofs in the Appendix (Section 9)
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material as URL)? [Yes] We include the simulation setups in Section 6 and also in the Appendix (Section 10). We also include the code in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We include some details in Section 6 and the rest details are included the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In the first experiment (subsection 6.1), include the mean and standard deviation the cumulative reward.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The compute type is included in the Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] The first experiment uses the environment in reference and we cite it. The second experiment is in Gazebo based on hector quaadroor...
(b) Did you mention the license of the assets? [Yes] We include the licenses in the sup- plementary materials.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] In the second experiment, we create a new simulator in Gazebo based on hec- tor quaddrooror..
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifi able information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We do not conduct research human subjects.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] We do not conduct research on human subjects.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] We do not conduct research on human subjects.
13


Reviews:
[{"review_id": "iK-dpC5kVyJ", "reviewer": "Reviewer_7HCk", "summary": "This paper proposes an inverse RL problem that requires inferring both the rewards and constraints in a multi-agent system. Unlike previous multi-agent IRL work, this paper introduces a setting where sharing expert demonstrations between agents are not allowed. To solve the problem, this paper proposes a distributed bi-level learning algorithm that iteratively solves 1) an inner problem for learning rewards and imitation policy and 2) an outer problem for inferring constraints. This paper analyses the convergence rate of their algorithm in terms of finding the optimal solution to the inner problem and converging to the set of stationary points of the outer problem.", "questions": "1. In line 90, the paper mentions \"we study the case where b = 0, i.e., hard constraint.\", but later the paper estimates a budget b_wc in Equation (2). I believe the estimated b_wc is not equivalent to b, so the paper is solving a soft constraint problem. Please explain more about this budget.\n\n2. In line 151, the paper defines the learner $\\nu$’s local likelihood function as a discounted cumulative log-likelihood. Why do we need the discounting term here? This is the likelihood for a given trajectory j instead of state-action pairs. \n\n3. This paper defines a bi-level optimization problem in formulas 3) and 4), but the following part has not introduced any insights or designed any algorithms for bi-level optimization.  It seems this paper just solves the inner and outer problems separately while the mainstream bi-level optimization algorithms often follow a single-level reformulation for better theoretical and practical properties. check [1]. Without these properties, what's the benefit of defining such a bi-level optimization problem?\n\n4. The paper mentions that \"the demonstration data are unable to be shared because of privacy concerns and communication burdens\", but in algorithms 1 and 2, the agents are sharing information (the gradients and parameters) with each other. This information-sharing behavior happens in both the inner and the outer problems (is very frequent during learning) and thus it creates even larger communication burdens, which makes the motivation less significant. Have the authors thought about minimizing the sharing frequency?\n\n5. I am wondering if it is necessary to learn a separate constraint function for each of the learner agents. What's the issue of learning a global constraint function? Since the information needs to be shared across the agents anyway. \n\n6. I am wondering if the drone's motion planning environment is indeed a multi-agent environment. It seems the expert trajectories run across each other a lot. Will the drones crash into each other in this way?\n\n7. In line 166. These citations have not mentioned constraints. \n\n6. In algorithm 1, what does n+1/2 means? \n\n[1] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, Zhouchen Lin: Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond. CoRR abs/2101.11517 (2021)", "limitations": "### limitations\n\nSection 7 is for the conclusion and future work. The discussion about limitations is unsatisfying. I suggest including my concerns about the reward functions and learning settings (see the weaknesses about paper significance mentioned above) in the limitations. In fact, many works have been aware of these weaknesses and addressed them in many ways. I encourage the authors to explore more about it.\n\n### negative societal impact \nThis paper has not discussed the negative societal impacts.\n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n**originality.** This paper extends the constraint IRL problem to the multi-agent domain, and more importantly, they assume the ground truth rewards are not available. To the best of my knowledge, previous works commonly assume one of these two dynamics is known (either assume the constraints are known or rewards are known) in order to infer the other one. This paper solves a very challenging problem, but it also gives me a feeling that maybe the problem itself is ill-posed since the IRL system can assign negative rewards in order to prevent some illegal movements. On the other hand, some constraint RL methods (e.g., Reward Constrained Policy Optimization [1]) augment the constraint signals to rewards by using the Lagrange multiplier. I mention this example since I believe the **learned** rewards can embed constraint signals unless you have other assumptions, for example, 1) the **system** rewards are fixed and we need constraints to explain the expert demonstrations (This is what Constraint IRL has). 2) the policy in maximum entropy IRL cannot assign zero probability to some actions no matter how bad the reward is, and thus we need some extra cost function in order to **ensure** hard constraints.  \n\n[1] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations ICLR. OpenReview.net, 2019.\n\n**quality.** The quality of this paper is satisfying. The problem is generally well-defined. Under some reasonable assumptions (linear cost and reward functions), the bi-level objective ensures some important properties (e.g., local convexity in sub-problem). The proposed algorithm looks fine. This paper shows some theoretical results by analyzing the convergence rate of their methods.  The empirical simulation makes sense.\n\n\n### Weaknesses\n\n**clarity.** This paper defines a complicated problem, but it is not in a reader-friendly format. This paper splits the definition of experts and learners agents and put them into a multi-agent system. Following these definitions is difficult especially when some of the definitions are missing and some symbols are used before defining (see typos). I strongly recommend adding some comments to the algorithm 1 and 2, or at least tell the reader where to find their explanations (on page 6 I think). The overlapping subscripts in algorithm 1 (e.g. line 4 of the algorithm) is difficult to follow. I spend lots of effort understanding them, but I am still not sure whether I am right. See questions as follows. \n\ntypos:\n1. lc anf lr in line 97 and line 99 are not defined.\n2. I was confused by the \\nu in line 93 and line 94, but then I find their meaning in the later sections when I read the part about communication networks.\n\n**significance.** I am concerned about the significance of this work from the following perspectives. \n\n- This algorithm assumes an open-box box environment with a known transition dynamic (Line 95, P and P_0 are the input of learners). The experiment uses two full-observable girdworld environments. More advanced benchmarks for constraint RL (e.g., safe-gym) have not been explored. Scaling the performance to solve the problem with high-dimensional and continuous state space will be difficult. \n\n- This paper follows classic apprenticeship learning algorithms that restrict rewards to convex sets given by linear combinations of feature vectors. It is known that apprenticeship learning algorithms generally do not recover expert-like policies if the set is too restrictive, which is often the case for the linear subspaces used by feature expectation matching unless the feature functions are very carefully designed [2]. I think this is the reason why this paper adds an extra cost function to fill the gap. \n\n[2] Jonathan Ho, and Stefano Ermon. Generative Adversarial Imitation Learning. In Neurips.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YJ6hyFk2alc", "reviewer": "Reviewer_AoHD", "summary": "This study proposes an effective learning approach for “distributed inverse constrained reinforcement learning” (D-ICRL). The current study solves this targeted problem by using a bi-level distributed optimization approach in estimating the constraints of the Markov game, reward, and the optimum policy of the cooperating expert agents. ", "questions": "1. Each distributed learner tries to learn the policies for all agents and estimate the cost function while utilizing the given demonstration data? It would be nice to graphically represent the target parameters that each learner tries to estimate and the scope of demonstrating data that is available for each learning agent. \n\n2. Please explain in detail how the problem (1) is converted into (2) and (3). The proposed bi-level formulation is proposed by the authors and thus is considered to be novel or this process is a natural and straightforward derivation induced while transforming the centralized constrained optimization into the distributed dual formulation. If the bi-level formulation has unique technical novelty, the author should explain the derivation procedure and intuition behind it, otherwise, the current paper should not highlight the bi-level formulation part. \n\n\n\n====Post Rebuttal========\nThanks for the detailed reply. I adjusted the score because the answer answered all of my questions. The reason that I cannot provide a more positive evaluation is that I don't fully understand the mathematical details. But, I believe this paper can contribute to the community.", "limitations": "The paper presentation is not clear. Instead of directly formulating the distributed version of constrained inverse multi-agent reinforcement learning, it would be nice to first precisely formulate the centralized version of ICRL.\n\nThe experiment section is too short and just provides numerical results without explaining the meaning and significance of the results.\n\nThe theoretical analysis looks rigorous, however, the meaning of such long results is not well explained.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "<Strength>\nThe proposed paper aims to tackle a challenging problem that has never been dealt with before. \n\n\n<Weakness>\n\n•\tMotivation for having multiple learners is not clear. \n\n•\tProblem formulation for Distributed Inverse Constrained Reinforcement Learning for Multi-agent Systems is not very clear. As a result, the derivation procedure of distributed algorithm is also unclear. \n\n•\tExperiment setup is too simple and hard to evaluate in supporting the effectiveness of the proposed model\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ss3oFM5YOYJ", "reviewer": "Reviewer_cHBi", "summary": "This paper considers a distributed inverse constrained RL problem formulated as a distributed bi-level optimization problem. It proposes a bi-level distributed algorithm with theoretical guarantees. ", "questions": "1. Is it true that every learner knows global action and state spaces?  If it is the case, is there any application examples for such a setting? \n\n2. In distributed RL algorithms, if the underlying matrices are doubly stochastic and assuming global action and state spaces, the distributed algorithm can be simplifies to a single agent counterpart by looking at the average of all agents' states. Is this also the case in this distributed inverse RL algorithm? \n\n3. It is stated that Lines 5 - 10 in the algorithm are to track the average global gradient. Is it an inner loop averaging process? does the average need to be accurately achieved? ", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper considers a very interesting problem in RL. The algorithm is novel. \n\nThe notation of the paper quite heavy and thus makes it not easy to follow. \n\nMy major concern is the technical novelty of the paper. The paper assumes doubly stochastic matrices in both inner and outer loops, and global observability of state and action spaces, which seem significantly simplifies the distributed algorithm design and analysis. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SfJw9fZQUTP", "reviewer": "Reviewer_SMXn", "summary": "The paper considers inverse RL (IRL), in a distributed setup with multiple experts demonstrating for multiple learners and includes constraints (areas of the state space that are not allowed).  The general model is setup, building on other IRL work, now generalizing to include multiple collaborative learners.  A multi-agent distributed learning algorithm is developed, based on a bi-level optimization approach.  The learning agents share features (they do not share their separate observations of the multiple experts).  The learners simultaneously estimate policies based on reward, and also constraints. The inner-outer loops run at different time scales (the outer loop is slower), and consensus iterations are used for collaboration among the learners.  An ‘order-of’ convergence proof is given under some reasonable assumptions.  Two experimental cases are studied in detail (a grid-world and a drone flying example), and in the drone case the experts are humans providing trajectories in gazebo. Several metrics are evaluated, and the new algorithm is compared with various benchmarks, such as a greedy method.  ", "questions": "How is the communications connectivity modeled or defined (how is the graph determined)?  (I think a local neighbor distance, but not sure I saw this explicitly.) This is time varying?  (Not clear in the experiments.) Any sense of how the communications topology affects the performance, convergence, etc.?\n\nNot quite clear on C-IRCL: this has a centralized single processor (so no consensus), correct?  Whereas if you run D-IRCL with a fully connected network, then you would still use consensus round(s)?  Can you compare these?  And importantly, what is an optimal centralized solution (the most important benchmark)?\n\nNot quite clear what is meant by \"action constraint\" (discussion of Fig  1).\n\nPlease clarify how Fig 1g is generated. Do you fix the number of consensus iterations and run the entire learning algorithm?  (And I think the relation of inner-outer time scales in the experiments wasn't clear, or I missed it.)\n\nGrid-world: It is interesting that you picked a different number of expert trajectories for each learner (10,20,30,40).  All of these are independently generated?  What happens, for example, if all agents have the same number?  Or if an agent has very few (or even zero), then it will rely on consensus to learn and might slow convergence because of poor initial estimation, correct?\n\nGrid-world: I don’t quite see a specific model for the expert policies, or their eventual knowledge of the environment.  From the Appendix, actually the experts are the same and acting independently unless they are close (and then some collision avoidance occurs)? The expert policy is stochastic, which accounts for CVR not being zero?  \n\nAlso for the experts: Terminating a trajectory on collision (in grid world) is in effect an infinitely negative reward, or actually highly informative for finding the constraint regions?  Wouldn’t it be simpler to have some agents learn the constraints as an independent first step (it seems exactly like related earlier single agent RL work that is focused on “safety” metrics, e.g., Abeel)?  It seems collisions are to be thought of as absolute disaster, yet the experts do this with some finite probability.  (The reviewer does recognize that a goal here is to work from human experts, and it may be different in that context.)\n\n6.2 drone example: “distribute the demonstration to two learners”.  How? Both learners have all of them?\n\nSection 7: Future work -- The linear combination question is interesting.  Ultimately, is there more to learn with a nonlinear model?  I suppose this is strongly linked to the particular application setup?\n", "limitations": "No potential negative social impact is apparent.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper breaks new ground by adding multiple collaborative learners to the IRL problem.  Overall, the paper has a nice balance of ideas, models, analysis, and experiments.  The work is nicely presented and very readable, with the models and algorithm well and clearly defined.  The notation is consistent with other works on IRL, which aids the reader.   The convergence proof is interesting and reveals aspects of asymptotic convergence rate. The experiments are well done and comparisons are interesting. \n\nAn interesting aspect is how the learners infer the constraints, even if an expert has never visited that state. \n\nThere do not appear to be any significant issues with this work.  Scalability, communications among the agents,  and the distributed method of optimization and convergence rate are all aspects that will likely be interesting to explore going forward.  \n\nThe questions below may be useful to add clarity and interpretation, especially for the experiments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept