Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers inverse RL (IRL), in a distributed setup with multiple experts demonstrating for multiple learners and includes constraints (areas of the state space that are not allowed).  The general model is setup, building on other IRL work, now generalizing to include multiple collaborative learners.  A multi-agent distributed learning algorithm is developed, based on a bi-level optimization approach.  The learning agents share features (they do not share their separate observations of the multiple experts).  The learners simultaneously estimate policies based on reward, and also constraints. The inner-outer loops run at different time scales (the outer loop is slower), and consensus iterations are used for collaboration among the learners.  An ‘order-of’ convergence proof is given under some reasonable assumptions.  Two experimental cases are studied in detail (a grid-world and a drone flying example), and in the drone case the experts are humans providing trajectories in gazebo. Several metrics are evaluated, and the new algorithm is compared with various benchmarks, such as a greedy method.  
strengths_and_weaknesses: The paper breaks new ground by adding multiple collaborative learners to the IRL problem.  Overall, the paper has a nice balance of ideas, models, analysis, and experiments.  The work is nicely presented and very readable, with the models and algorithm well and clearly defined.  The notation is consistent with other works on IRL, which aids the reader.   The convergence proof is interesting and reveals aspects of asymptotic convergence rate. The experiments are well done and comparisons are interesting. 

An interesting aspect is how the learners infer the constraints, even if an expert has never visited that state. 

There do not appear to be any significant issues with this work.  Scalability, communications among the agents,  and the distributed method of optimization and convergence rate are all aspects that will likely be interesting to explore going forward.  

The questions below may be useful to add clarity and interpretation, especially for the experiments.

questions: How is the communications connectivity modeled or defined (how is the graph determined)?  (I think a local neighbor distance, but not sure I saw this explicitly.) This is time varying?  (Not clear in the experiments.) Any sense of how the communications topology affects the performance, convergence, etc.?

Not quite clear on C-IRCL: this has a centralized single processor (so no consensus), correct?  Whereas if you run D-IRCL with a fully connected network, then you would still use consensus round(s)?  Can you compare these?  And importantly, what is an optimal centralized solution (the most important benchmark)?

Not quite clear what is meant by "action constraint" (discussion of Fig  1).

Please clarify how Fig 1g is generated. Do you fix the number of consensus iterations and run the entire learning algorithm?  (And I think the relation of inner-outer time scales in the experiments wasn't clear, or I missed it.)

Grid-world: It is interesting that you picked a different number of expert trajectories for each learner (10,20,30,40).  All of these are independently generated?  What happens, for example, if all agents have the same number?  Or if an agent has very few (or even zero), then it will rely on consensus to learn and might slow convergence because of poor initial estimation, correct?

Grid-world: I don’t quite see a specific model for the expert policies, or their eventual knowledge of the environment.  From the Appendix, actually the experts are the same and acting independently unless they are close (and then some collision avoidance occurs)? The expert policy is stochastic, which accounts for CVR not being zero?  

Also for the experts: Terminating a trajectory on collision (in grid world) is in effect an infinitely negative reward, or actually highly informative for finding the constraint regions?  Wouldn’t it be simpler to have some agents learn the constraints as an independent first step (it seems exactly like related earlier single agent RL work that is focused on “safety” metrics, e.g., Abeel)?  It seems collisions are to be thought of as absolute disaster, yet the experts do this with some finite probability.  (The reviewer does recognize that a goal here is to work from human experts, and it may be different in that context.)

6.2 drone example: “distribute the demonstration to two learners”.  How? Both learners have all of them?

Section 7: Future work -- The linear combination question is interesting.  Ultimately, is there more to learn with a nonlinear model?  I suppose this is strongly linked to the particular application setup?

limitations: No potential negative social impact is apparent.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers a distributed inverse constrained RL problem formulated as a distributed bi-level optimization problem. It proposes a bi-level distributed algorithm with theoretical guarantees. 
strengths_and_weaknesses: The paper considers a very interesting problem in RL. The algorithm is novel. 

The notation of the paper quite heavy and thus makes it not easy to follow. 

My major concern is the technical novelty of the paper. The paper assumes doubly stochastic matrices in both inner and outer loops, and global observability of state and action spaces, which seem significantly simplifies the distributed algorithm design and analysis. 
questions: 1. Is it true that every learner knows global action and state spaces?  If it is the case, is there any application examples for such a setting? 

2. In distributed RL algorithms, if the underlying matrices are doubly stochastic and assuming global action and state spaces, the distributed algorithm can be simplifies to a single agent counterpart by looking at the average of all agents' states. Is this also the case in this distributed inverse RL algorithm? 

3. It is stated that Lines 5 - 10 in the algorithm are to track the average global gradient. Is it an inner loop averaging process? does the average need to be accurately achieved? 
limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This study proposes an effective learning approach for “distributed inverse constrained reinforcement learning” (D-ICRL). The current study solves this targeted problem by using a bi-level distributed optimization approach in estimating the constraints of the Markov game, reward, and the optimum policy of the cooperating expert agents. 
strengths_and_weaknesses: <Strength>
The proposed paper aims to tackle a challenging problem that has never been dealt with before. 


<Weakness>

•	Motivation for having multiple learners is not clear. 

•	Problem formulation for Distributed Inverse Constrained Reinforcement Learning for Multi-agent Systems is not very clear. As a result, the derivation procedure of distributed algorithm is also unclear. 

•	Experiment setup is too simple and hard to evaluate in supporting the effectiveness of the proposed model

questions: 1. Each distributed learner tries to learn the policies for all agents and estimate the cost function while utilizing the given demonstration data? It would be nice to graphically represent the target parameters that each learner tries to estimate and the scope of demonstrating data that is available for each learning agent. 

2. Please explain in detail how the problem (1) is converted into (2) and (3). The proposed bi-level formulation is proposed by the authors and thus is considered to be novel or this process is a natural and straightforward derivation induced while transforming the centralized constrained optimization into the distributed dual formulation. If the bi-level formulation has unique technical novelty, the author should explain the derivation procedure and intuition behind it, otherwise, the current paper should not highlight the bi-level formulation part. 



====Post Rebuttal========
Thanks for the detailed reply. I adjusted the score because the answer answered all of my questions. The reason that I cannot provide a more positive evaluation is that I don't fully understand the mathematical details. But, I believe this paper can contribute to the community.
limitations: The paper presentation is not clear. Instead of directly formulating the distributed version of constrained inverse multi-agent reinforcement learning, it would be nice to first precisely formulate the centralized version of ICRL.

The experiment section is too short and just provides numerical results without explaining the meaning and significance of the results.

The theoretical analysis looks rigorous, however, the meaning of such long results is not well explained.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an inverse RL problem that requires inferring both the rewards and constraints in a multi-agent system. Unlike previous multi-agent IRL work, this paper introduces a setting where sharing expert demonstrations between agents are not allowed. To solve the problem, this paper proposes a distributed bi-level learning algorithm that iteratively solves 1) an inner problem for learning rewards and imitation policy and 2) an outer problem for inferring constraints. This paper analyses the convergence rate of their algorithm in terms of finding the optimal solution to the inner problem and converging to the set of stationary points of the outer problem.
strengths_and_weaknesses: ### Strengths

**originality.** This paper extends the constraint IRL problem to the multi-agent domain, and more importantly, they assume the ground truth rewards are not available. To the best of my knowledge, previous works commonly assume one of these two dynamics is known (either assume the constraints are known or rewards are known) in order to infer the other one. This paper solves a very challenging problem, but it also gives me a feeling that maybe the problem itself is ill-posed since the IRL system can assign negative rewards in order to prevent some illegal movements. On the other hand, some constraint RL methods (e.g., Reward Constrained Policy Optimization [1]) augment the constraint signals to rewards by using the Lagrange multiplier. I mention this example since I believe the **learned** rewards can embed constraint signals unless you have other assumptions, for example, 1) the **system** rewards are fixed and we need constraints to explain the expert demonstrations (This is what Constraint IRL has). 2) the policy in maximum entropy IRL cannot assign zero probability to some actions no matter how bad the reward is, and thus we need some extra cost function in order to **ensure** hard constraints.  

[1] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations ICLR. OpenReview.net, 2019.

**quality.** The quality of this paper is satisfying. The problem is generally well-defined. Under some reasonable assumptions (linear cost and reward functions), the bi-level objective ensures some important properties (e.g., local convexity in sub-problem). The proposed algorithm looks fine. This paper shows some theoretical results by analyzing the convergence rate of their methods.  The empirical simulation makes sense.


### Weaknesses

**clarity.** This paper defines a complicated problem, but it is not in a reader-friendly format. This paper splits the definition of experts and learners agents and put them into a multi-agent system. Following these definitions is difficult especially when some of the definitions are missing and some symbols are used before defining (see typos). I strongly recommend adding some comments to the algorithm 1 and 2, or at least tell the reader where to find their explanations (on page 6 I think). The overlapping subscripts in algorithm 1 (e.g. line 4 of the algorithm) is difficult to follow. I spend lots of effort understanding them, but I am still not sure whether I am right. See questions as follows. 

typos:
1. lc anf lr in line 97 and line 99 are not defined.
2. I was confused by the \nu in line 93 and line 94, but then I find their meaning in the later sections when I read the part about communication networks.

**significance.** I am concerned about the significance of this work from the following perspectives. 

- This algorithm assumes an open-box box environment with a known transition dynamic (Line 95, P and P_0 are the input of learners). The experiment uses two full-observable girdworld environments. More advanced benchmarks for constraint RL (e.g., safe-gym) have not been explored. Scaling the performance to solve the problem with high-dimensional and continuous state space will be difficult. 

- This paper follows classic apprenticeship learning algorithms that restrict rewards to convex sets given by linear combinations of feature vectors. It is known that apprenticeship learning algorithms generally do not recover expert-like policies if the set is too restrictive, which is often the case for the linear subspaces used by feature expectation matching unless the feature functions are very carefully designed [2]. I think this is the reason why this paper adds an extra cost function to fill the gap. 

[2] Jonathan Ho, and Stefano Ermon. Generative Adversarial Imitation Learning. In Neurips.
questions: 1. In line 90, the paper mentions "we study the case where b = 0, i.e., hard constraint.", but later the paper estimates a budget b_wc in Equation (2). I believe the estimated b_wc is not equivalent to b, so the paper is solving a soft constraint problem. Please explain more about this budget.

2. In line 151, the paper defines the learner $\nu$’s local likelihood function as a discounted cumulative log-likelihood. Why do we need the discounting term here? This is the likelihood for a given trajectory j instead of state-action pairs. 

3. This paper defines a bi-level optimization problem in formulas 3) and 4), but the following part has not introduced any insights or designed any algorithms for bi-level optimization.  It seems this paper just solves the inner and outer problems separately while the mainstream bi-level optimization algorithms often follow a single-level reformulation for better theoretical and practical properties. check [1]. Without these properties, what's the benefit of defining such a bi-level optimization problem?

4. The paper mentions that "the demonstration data are unable to be shared because of privacy concerns and communication burdens", but in algorithms 1 and 2, the agents are sharing information (the gradients and parameters) with each other. This information-sharing behavior happens in both the inner and the outer problems (is very frequent during learning) and thus it creates even larger communication burdens, which makes the motivation less significant. Have the authors thought about minimizing the sharing frequency?

5. I am wondering if it is necessary to learn a separate constraint function for each of the learner agents. What's the issue of learning a global constraint function? Since the information needs to be shared across the agents anyway. 

6. I am wondering if the drone's motion planning environment is indeed a multi-agent environment. It seems the expert trajectories run across each other a lot. Will the drones crash into each other in this way?

7. In line 166. These citations have not mentioned constraints. 

6. In algorithm 1, what does n+1/2 means? 

[1] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, Zhouchen Lin: Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond. CoRR abs/2101.11517 (2021)
limitations: ### limitations

Section 7 is for the conclusion and future work. The discussion about limitations is unsatisfying. I suggest including my concerns about the reward functions and learning settings (see the weaknesses about paper significance mentioned above) in the limitations. In fact, many works have been aware of these weaknesses and addressed them in many ways. I encourage the authors to explore more about it.

### negative societal impact 
This paper has not discussed the negative societal impacts.
review_title: null

Meta-review: metareview: The paper produces one of the first analysis of distributed inverse reinforcement learning, which is formulated as a bilevel optimization problem. The paper initiates a new line of research, contains a good mix of theoretical and empirical results, and has received relatively high scores from reviewers. 
confidence: Less certain
Decision: Accept