Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the safe RL in the multi-agent setting. Specifically, the author leverage the theories from constrained policy optimization and multi-agent trust region learning to propose two algorithms: MACPO and MAPPO-Lagrangian. From the theoretical side, the author shows that in the idea setting, the proposed algorithm is guarantee to improve the objective function at iteration and the constraint satisfications can always be guaranteed. The author also demonstrate the effectiveness of their proposed algorithms in the new environment named "Safe MAMuJoCo".
main_review: Strength:

This paper proposes two interesting algorithms to solve the safe RL problem in the multi-agent setting based on CPO, PPO and primal-dual approches. The author provides theoretical guarantee for their proposed algorithm (Thm 1). The effectiveness of their proposed algorithms is supported by extensive experiments.

Weakness: 

1. More discussions are need to highlight the novelty and challenging compare with CPO (Achiam 2017).

2. The theoretical result (Thm1) can be improved if the author can provide quantitative characterization of policy improvement and constraint satisfication similar to Proposition 1 and Proposition 2 in CPO (Achiam 2017), respectively.

2. The experiment part misses some details, which I list below:

    (1) It is not clear how to apply the algorithm MACPO in the multi-constrains setting in experiment since solving eq. (4) in the multi- constraints setting is very challenging.

    (2) It is also not clear how MACPO updates the policy at the beginning of the algorithm, when the initialization point is infeasible. From the theoretical results, it seems that the constraint satisfications can only be guaranteed after the algorithm enter into the feasible region.

2. The update rule of MACPO and MAPPO-LAGRANGIAN seems to be not efficient when the algorithm is not feasible. Because in the infeasible region we hope the algorithm can enter into the feasible region as soon as possible. Therefore, instead of optimizing a mixed objective function of both reward and cost, we may want to only minizing the cost.
summary_of_the_review: I can consider raise my score if the author can address my question in the "main review".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the multi-agent reinforcement learning (MARL) problem with safety constraints. The authors proposed two methods, Multi-Agent Constrained Policy Optimisation (MACPO) and MAPPO-Lagrangian, by leveraging the theories from both constrained policy optimization and multi-agent trust-region learning. Their method is shown valid both theoretically and empirically. My main concerns lie in their novelty compared with existing literature and the problem set.
main_review: **Strengths**

1. This paper addressed an interesting and important question in MARL, i.e., how to learn the optimal policy with certain constraints.

2. Their method was shown to be valid and promising both theoretically and empirically.

**Weaknesses**

1. The main idea of this work is to combine and extend [1] and [2], while there is no sufficient discussion on the methodological and theoretical contribution of this work beyond these two cited works. More necessary elaboration and justification should be made to solid the novelty of this paper.

[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In
International Conference on Machine Learning, pp. 22–31. PMLR, 2017.

[2] Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11251, 2021a.

2. This paper particularly cares about a safe constraint, while the relationship between cost and reward is not very clear. Based on MAPPO-Lagrangian where the authors use a hard constraint on satisfying the safe condition, it seems that the cost itself is more important than reward. However, in the real application, there may exist a trade-off between cost and reward. I felt some human preference or domain knowledge as required for the constraint optimization literature.
 

summary_of_the_review: I think this is a borderline paper that addressed an important question with reasonably good performance while lacking necessary elaboration and justification on their novelty, as commented in my 'Main Review'. I am willing to upgrade if my concerns can be addressed during the rebuttal period.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two algorithms for multi-agent reinforcement learning with constraints using policy optimization based approaches. The evaluations of the proposed approaches are provided. 
main_review: Strengths:
The algorithms are described well, and evaluations are provided. The problem is important.

Weaknesses:
1. Missing literature: For single agent, authors are suggested to see CRPO Xu et al. (2021), PDSC Chen et al. (2021), Triple-Q Wei et al. (2021), CSPDA Bai et al. (2021). 
2. For multi-agent, see ECML paper https://2021.ecmlpkdd.org/wp-content/uploads/2021/07/sub_181.pdf which also gives a model-free approach. Comparisons between the approaches are needed.
3. There are no guarantees for the proposed approach. The combination of approaches does not seem to have enough novelty. 
4. Although the constraints mentioned in Section 3 has the form of  "average constraints". I think what actually used in the experiments can be seen as a "peak constraint".  So I'm curious about  the performance of the algorithms  in plain "average constraints"(e.g. if the Cost function can return cost less than 0 when c_j=0). Moreover, since the constraint in experiments is "peak constraint", why not compare the two proposed algorithms to the simple baseline that add penalty toward the global reward once the constraint is violated.  

summary_of_the_review: It would be good to compare with the ECML paper approach mentioned above for scalability and performance.
review_title: null

Meta-review: The paper addresses safe multi-agent reinforcement learning and makes two key contributions. First is a safety concerned multi agent benchmark, which is an extension of MAMuJoCo. Second, is the formulation and two solution to safety MARL problem. The authors pose safe MARL, and MARL problem with safety constraints, as a constrained Markov game.

The safety constrained MARL is an important, difficult, and understudied problem. The problem is more difficult that the single agent safe RL because of the non-stationarity in the MARL setting, which renders any theoretical guaranties conditioned on the assumptions of the behaviors of other agents. The authors are right to point out the lack of the benchmarks in the space. 

That said, reflecting on the reviewers' feedback and my own reading of the paper, this paper is attempting to do too much (benchmark, problem formulation, and two methods), in too little space, and is falling short. For example, the benchmark is an important contribution, but it is barely mentioned in the main text of the paper. If this was fully safety benchmark paper, there is an opportunity to go beyond MAMuJoCo, which feels like a forced multi-agent problem, and construct a safety benchmark with energy constraints, cooperative and competitive tasks etc... If this was fully methods paper, there would be an opportunity for more in-depth analysis of the results that the reviewers' pointed out. In it's current form, the paper feels like proposing a benchmark not grounded in a real world problem, and then a method to solve the problem.

I would suggest the authors to either:
- submit the paper to a journal where a space constraint would not be in a way, or
- split it into two papers, a more comprehensive benchmark, and methods paper evaluated on more difficult problems. 

Minor:
- Please update the literature. Some of the papers have been published, and they are cited as Arxiv papers.
Decision: Reject