CAMA: A New Framework for Safe Multi-Agent Reinforcement Learning  Using Constraint Augmentation
Authors: Ziyan Wang
Yali Du
Aivar Sootla
Haitham Bou Ammar
Jun Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Safe
Multi-agent Reinforcement Learning
Augmentation

Abstract: With the widespread application of multi-agent reinforcement learning (MARL) in real-life settings, the ability to meet safety constraints has become an urgent problem to solve. For example, it is necessary to avoid collisions to reach a common goal in controlling multiple drones. We address this problem by introducing the Constraint Augmented Multi-Agent framework --- CAMA. CAMA can serve as a plug-and-play module to the popular MARL algorithms, including centralized training, decentralized execution and independent learning frameworks. In our approach, we represent the safety constraint as the sum of discounted safety costs bounded by the predefined value, which we call the safety budget. Experiments demonstrate that CAMA can converge quickly to a high degree of constraint satisfaction and surpasses other state-of-the-art safety counterpart algorithms in both cooperative and competitive settings. 

Under review as a conference paper at ICLR 2023
CAMA: A NEW FRAMEWORK FOR SAFE MULTI- AGENT REINFORCEMENT LEARNING USING CON- STRAINT AUGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
With the widespread application of multi agent reinforcement learning (MARL) in real-life settings, the ability to meet safety constraints has become an urgent problem to solve. For example, is necessary to avoid collisions to reach com- mon goal in controlling multiple drones. We address this problem by introducing the Constraint Augmented Multi-Agent framework - CAMA. CAMA can serve a polug-and-pee -lay module to the popular MARL algorithms, including centralized training, decentralized execution and independent learning frameworks. In our ap- proach, represent the safety constraint as the sum of discounted safety costs bounded by the predefined value, which we call the safety budget. Experiments demonstrate that CAMA can converge quickly high degree of constraint isfaction and surpasses other ttate-of-the-ee safety counterpart algorithms in both cooperative and competitive settings.
INTRODUCTION
Multi-agent problems are ubiquitous in real world, such as robotics Al-Abbasi et al. 2019 Mguni
al., 202T), transportation systems (Zhou al. 2020 Chu al.) 2019), network optimization
(Wang et al. 2020; Wai et al. 2018). and multi multi-player player video games (Du et al. 2019 Samvelyan
al.) 2019 Han et al. 2019 Peng al. 2017]. modern approach to solving these decision- making problems is multi-agent reinforcement learning (MARL.,, which tackles these problems us- ing only interactions with the environment. There are many different frameworks within MARI. such as fully centralized Berner et al. (2019); /Skkhaatarr & Fergus (2016), independent learning (IL) de Witt et al. (220); Zhang et al. (2018) and a hybrid framework which is the centralized train- ing and decentralized execution (CTDE) (Foerster al. 2018 Lowe et al. 2017 Yang al. 2018). However, within the deployment of MARL, safety is still a crucial problem, which has not been fully solved yet.
In recent years, several works have incorporated safety constraints into RL training, such as opti- mizing policy under constraints (Di Castro et al.] 2012 Tessler et al. 2018 Achiam et al.) 2017] Chow et af.] (2018), adding safety layers (Dalal et al. 2018), or constructing verifiable safe explo- ration (Anderson al. (2020), etc. In the context of safe MARL, recent papers extend constrained policy optimization (Achiam et al.) (017) to multi-agent domain (Gu et al. 2021]) as model-free safe MARL algorithms. But there are still challenges with low reward performance compare the non-safe MARL algorithms. There are also some works performed constrained policy optimization by transforming it into min-max game (Lu et al.) [iu et al., 202T). However, which limited by the specific framework, it cannot generalize to other framework such as solving the competi- tive game. Therefore, a more general safe MARI framework with high reward performance still lacking at this stage.
To fill this gap, in this paper we propose a general module that can be incorporated into differ- ent MARL algorithms. The proposed Constraint Augmented Multi-Agent framework, coined as CAMA. is a plug-and-play method to improve cutting-edge non-safe MARL algorithms satisfying the adding constraints. Furthermore, CAMA aims to address both cooperation and competitive set- ting under CTDE and IL frameworks. In our algorithm, we represent the safety constraint as the sum of discounted safety costs bounded by a pre-defined scalar, which call the safety budget.

---

Under review as a conference paper at ICLR 2023
The main idea of this approach is the introduction of the hazard value, which tracks the accumulated costs and represents the remaining safety budget. When the hazard value falls below zero, CAMA assigns a low negative reward to the agents, incentivizing them to learn a safe policy. The implemen- tation of CAMA can be seen as a direct modification of the environment and indirectly influencing the algorithm by constraints augmentation so that there is no need for any new algorithm- assumptions.
In summary, our contributions are three-fold. Firstly, CAMA flexible framework with a plug- and-play feature, which can be combined with many existing MARL algorithms. Secondly, CAMA can work in both CTDE and IL settings, including cooperative and competitive multi-agent games. Lastly, we evaluate CAMA on a series of multi-agent control tasks in SMAMujoco (Gu et al. 202T) and Gym Compete (Bansal et al.) 2017). Empirical results demonstrate the effectiveness of our solutions both in terms of constraint satisfaction and reward maximisation compared to their state- of-the-art counterparts.
2 RELATED WORK.
In the following, we review the related works on general MARL, MARL with constraint and state augmentation method.
Existing MARL algorithms are often developed under the paradigm of CTDE and IL (Oroo- jlooyJadid & Hajinezhad, [ CTDE is commonly used learning framework, which updates decentralized policies by using the centralized critic architecture (Du et al. 2019: Gu et al.) 2021). In CTDE, the joint critic network is based on the all agents' states and actions, and thus generally handles the joint team reward scenarios (Kuba et al., 2022, Yu et al., 202T). Conversely, if the setting is competitive game only focus on the individual reward, then IL is another MARL paradigm to be based on. The earliest discussions of IL in a multi-agent- be traced back to (Tan 1993). It subsequently evolved into the IL algorithm using neural networks as function approximators (Foerster et al. 2018 Rashid et al.] 2020). Some recent attempts, like de Witt et al. (2020) extend single agent proximal policy optimization algorithms (Schulman al., 2017) into the multi-agent IL setting.
environment can
Although MARL has received significant attention in recent years, there still many unresolved safety related challenges (Gu et al., 2022), such as the multiple constraints setting, low algorithm efficiency problem etc. There are generally several ways to solve additional safe constraints. Recent attempts such as MACPO, MAPPO-L(Gu et al.) 2021) proposed to fill such gap as the first safe model-free MARL algorithms, which are extensions of CPO (Achiam al., 2017) and HATRPO (Kuba et al.] 2022). respectively. However, neither MACPO nor MAPPO-L L. is guaranteed to be applied in competitive games, which resulting in limited scalability. Another research direction based on the parameters-sharing hypothesis. For example, the CMIX (Liu et al. [022] can combine multi-objective programming and the Q-mix framework to solve the constraint MARL problem. However, in CMIX, different Q-function approximators agent, which leads to scalability and efficiency challenges. Another cerammetrr-shhrinng based ap- proach, Safe Dec-PG (Lu et al.. 2021) aims to stratify the constraint by passing the parameters through a predefined communication network. Instead, our framework requires no communica- tion during policy execution. Another route avoid unsafe action is use shielding and barrier functions, such as ElSayed-Aly et al. (2021); Cai et al. (2021). However, those approaches require pre-training or strong prior knowledge create the shields for filtering actions. Moreover, they cannot generalize to new scenarios where safety shield is not known. In contrast, CAMA is more flexible when dealing with new constrained environments. We test different types of tasks with varying agents without designing or pre-train a particular shielding function for each task.
required for each constraint and each
The state augmentation method extend the specific state to an environment, in order to enhance policy performance or satisfy certain constraints (Calvo-Fullany et al.) (02T). Recent works like QQu et al.) 202I) augmented the CVaR to measure over the learned distributions of individuals' Q values, Chen al. (2020) and Foerster et al. (0017) augmented delay awareness and experience replay, respectively. The idea of enhancing safety-related variables has been considered in the past, e.g., in classical control methods (Daryin & Kurzhanski 2005) and in single-agens aggnt safe RL (Sootla ctt al. 2022a b Chow et al. (017).. We apply it in a multi-agent framework, but multiple constraints and multiple policy settings hinder the direct extension. Some works, such (Chen et al. 2020
2

---

Under review as a conference paper at ICLR 2023
Foerster et al. 2017). also apply augmentation methods to MARL problems, but the first to augment the constrait to solve the safe MARL problem.
PRELIMINARIES
Constrained Multi-agent MDP (CMMDP) A CMMDP (Boutilier, 1996) extends CMDP (Ma et al. 1986) by adding a factored action space, which can be defined by tuple M (N, A,S.AAP, Nr Ye: p., R, C, B). Here, N .l.....,]]............... is the set of agents, & denotes the state space, and A = =III=1 transition function. Te (0, 1) are the discount factor for reward and cost. R(s, a) 8 - R
the joint action space, P (s' s,a): SxAxx - [o. 1] is the probabilistic
the reward function of state 8 and joint action a, = [ isismi set safety costs, where
iEn
Cý is the jth element of the total m² constraints, Cj : S A' R. B (b) isjsmi the set of corresponding safety budgets, the maximum value of agents can be violated, p² is the initial state dis- tribution. Denote agent i's policy as n, S A, - [0, 11,
VEN iEN
{ CMMDP aims to find a op-
timal joint policy π" that maximizes the joint reward JR(T) (T) - Esc.as... 1E of (8t, az)] while satisfying the safety constraints Jz( i, E [ 7.CH (st, a;)]
I b', where C = =E, Cj
50 p², af Ti (-1 8t) 8
St: ax) for all i N.
Two commonly used architectures are Centralized Training with Decentralized Execution (CTDE) and Independent Learning (IL). CTDE update independent policies by maximizing the expected team reward JR using the policy gradient. IL decomposes an n-agent MARL problem into n decentralized single-agent problems, and ignores other agents by treating them as part of the environment. The policy update is conditioned on the local observations or global state and actions π, : 8; X a, >> IR, then received the local reward and cost. CTDE widely used in cooperative tasks with shared team reward, and IL can work in both cooperative and competitive settings.
Constraints State Augmentation. State augmentation eliminates safety constraints by incorporat- ing them into the state space (Chow et al. 2017; (ootla et al. (002a)). Defining 2t as scaled version of the remaining safety budget where 2 can be calculate as #1+1 = (zt - c(s,, a,)) /Tes 2 = b. Using this new variable, they reshape the reward by
R(st. Rt, az)
R(St, at)
2,
21 <<
where k<0 is an unsafe reward. Using these simple modifications, Sootla et al. (202aa) showed that an algorithm solving this problem actually solves safe RL with probability one constraints under certain conditions.
4 METHODOLOGY
In this section, we first define the basic definition of CAMA-MDP and list the basic assumptions and propositions required. After that, introduce the specific derivation process of CAMA under the CTDE and IL paradigms in subsections and 4.3 Finally, we made an algorithm to summarize the workflow of CAMA.
4.1 THE DEFINITION OF CAMA
We formalize our approach by considering the following definition,
Definition 1
Given a CMMDP, define Constraint Augmented Multi-Agent Markov Decision Process (CAMA-MDP) as tuple: M = (, Tr. Te: R, C)C where š = x h, is the constraint augmented state space, h; { denotes the of hazard value, : 5x R. Let R = Σn R, denotes reward function, where R. 5 A + [0, +oo) is the 5th element of R defined in the tuple.
To guarantee the optimal policy of CAMA-MDI exists and is also the same as CMMDP's T", we can adapt the theoretical results: Assumption. Al 1-- from Sootla al. (2022a) in a straightforward
3

---

Under review as a conference paper at ICLR 2023
manner for CAMA IL frameworks. But in the CTDE setting, still need the following regularity assumption, in particular, to guarantee that the policy set is closed and bounded:
Assumption There exists 0 E R, such that 0 < 0 < 1, and for every agent E N, the policy spuce II' is 0-soft, which means for every x² E II's E S, and a' E A', we have X¹ (at (8) 2 O.
This assumption has been discussed in Lemma 3 (Kuba et al. 2022). For more details, we provided the theoretical analysis in Appendix
4.2 CAMA IN CTDE
Let us firstly consider a CMMDP problem with multi-constraints and joint reward. Since the Je(T,, i, always greater than which can be equivalent enforcing the finite number of the following constrains:
Σ Cn (sk, ax) S byj V 2 00 1m=0
(2)
Therefore, when an agent violates the limit at specific moment 4. it will be in hazard zone for a series of subsequent time points 2 l. which allows us to incorporate constraints into the instantancous time as the safe cost, taking safety into account when solving the task. In the multi- agent case, defining the hazard value is slightly more complicated due to the number of constraints. We introduce a hazard value for each constraint and each agent as follows.
h
b
C (sE) I76--
A=
For simplicity, we will use the notation h for the vector (hi.e
hin..) and he for the vector (h) h²). . This shows that the remaining safety budget and can easily be tracked order to assess the constraint satisfaction. Similarly, to the single agent case the recursive update is
h h1),411 (hi,, Cý (st, a.)) /To-
(4)
Then, during the policy training, we can reshape the reward by using the vector of hazard values h4:
R(3 a.) = R((s min h < O.
ming h 2
(5)
In the CTDE setting, according to equation 5 the algorithms will firstly find the minimum hazard value for each agent in the multiple constraints hazard vector miny hi Then, suppose there ists one hazard value less than O. In that case, this agent's trajectory already violates the adding constraints, all agents will get an unsafe reward & to avoid the same unsafe situation in the next time-step. Finally, we can formalise the problem to
Σ (. ai)
00
j & JT((T) Ezg Co....
s.t. 3, = 81 x h3 h2 80 ~ (8o), n3,0 = 11. ~ Te(. 1i),
(6)
m²
844 P 51 "t.....,,
a.)
h 1,41
h
- (ss /
Note that the hazard values can be considered part of the transition dynamics, so that can stay the Markov feature. Further, the overall problem is the standard CTDE formulation, which enables the aog--an--pay feature.
4.3
CAMA IN IL.
The CAMA in IL has the same process to calculate the hazard value vector, following equation and updating the rule by equation 4 However, each agent can only observe its hazard value vector. Therefore, reshaped the individual reward functions as follows,
R (s), af)
miny h 2 0
-.- R, (3, of)
miny h/. < 0.
4

---

Under review as a conference paper at ICLR 2023
Where for CAMA in the IL setting, considering each agent has its local critic and actor, and the reshaped reward function the part that each agent interacts with the environment. Note that the reward is replaced by the negative number l when this agent's minimum local hazard value is less than O. The henefit is that it follows the original intention of the II. setting, which is to treat the surrounding agents environment that neighbors' violations do not affect their rewards.
Algorithm 1: Constraint Augmented Multi-Agent Reinforcement Learning (CAMA)
1:
for Nepisodes do
2: Initialize the scenario and the agent
3: Reset state So, Initialize a random process Po for action exploration
4: for 0, 1,....... do
5: for each agent do
6:
Select the action a, under the current policy n; and its exploration.
7:
Store those in the buffer. end for
8:
9:
Executes the actions and calculate the new state s'
10: Sample the hazard value vector h; and augmented into the state s, // Eq
11: Get the new reshaped Reward (5r, as) // Choose Eq 5 or depending on the game
setting.
12:
Updating policies æ, to maximise JR (TT through the MARL algorithm //Eq end for 14: end for
Therefore, we can present CAMA's framework and internal logic through the algorithms For both CTDE and II. settings, CAMA will follow the general update process steps 1-9 until all agents' following actions and their ground truth rewards are obtained. Then, in steps 10 to 12, the hazard values and reshaped rewards are updated according to the two settings discussed above. In the end, update the policy by using new reshaped reward and augmented hazard value to state to wait next time-steps training. This flexible framework enables CAMA to improve the safety without changing the combined algorithm.
5 EXPERIMENTAL RESULTS
We compare CAMA with the Safety MAMujoco baseline (Gu et al. 2021) and the GymCompete baseline (Bansal et al. 2017). We also provide ablation studies unsafe reward and safety budget components.
General Implementation. To the CAMA framework, only need to calculate the hazard value, reshape the rewards, and augment them into their trajectory. Thus, there is no need to modify the non-safe MARL algorithm itself. During the training, the reset and step function will augment
the hazard value and reshape the new cost function according to equation [5] We combine the CAMA
with HAPPO (Kuba et al. 2022). MAPPO et. 202T) and IPPO (de Witt al. 2020) create HAPPO, and CAMA-MAPPO for CTDE setting and CAMA-
for IL setting. For the baseline comparisons, we compare our method with the MACPO, MAPPO-Lagrange (Gu et al., 2021), and those originally non-safe algorithms itself. We used the hyperparameters that reported the best performance of the baseline algorithms. More details of the hyperparameters setting can be found in Appendix B
5.1 CTDE EXPERIMENT
In this subsection, we mainly introduce the performance of CAMA in the cooperative MARI, game. In cooperative tasks, different agents will share common goal and maximize team reward.
Joint Reward Environments setup. We demonstrate the benefit and limitations of our method on SMAMuJoCo (Gu et al. 2021). which is safety-aware extension of Mujoco (Todorov et al. 2012) designed for safe MARL research. The main three scenarios we used are Ants, Half-Cheetahs and ManyAnt (schematically depicted in Figure [ The number behind the robot name is the number of agents and the parts each agent controls, like Ant 2x4, which means there are two agents and control two neighbour legs together (each lag has two parts), and Ant 2x4d indicate the diagonal two legs
5

---

Under review as a conference paper at ICLR 2023
(b) ManyAnt 2x3 (c) Half-Choctal (d) Ant & ManyAnt Game (e) Half Cheetah Game
(a) Ant 2x4
Figure I: Cooperate Scenarios: Panels The view of robots. Multi-agents control the body by different colour parts. Panels d: The Ant and ManyAnt scenario with overthrow-able walls, Panels The Half Cheetah scenario with moving heading obstacle and overthrow-able walls.
control by two agents. All the task aims to let agents jointly learn the manipulation of the robot while avoiding crashing dark red walls into an unsafe area.
For the environment costs design, we follow the previous work by (Gu et al., 2021). more details listed in Appendix B every agent will receive cost from the environment, and we calculate the average and maximum for performance comparisons. We all run in 5 different seeds for all following experiments and evaluate the result using the saved intermediate policies for every agent. Please note that, in each episode, for each agent, the system will present two types of rewards. One is true reward rt based on the original task reward, we sum all agents' true reward as the team reward (titled Average Episode Cost in the figure). The other one is reshaped reward based on hazard value 1 the policy receives this as a feedback from the environment for training.
In addition, main nyper-parameten reward k. In order to have fair comparison, we set the safety budget equal 50 and the unsafe reward equal to k -R(s ae) for all tasks, for more setting discussion and its ablation test follow in sec 5.3
be tuned in CAMA: the safety budget b and unsafe
HAPPO
MAPPO
MACPO
MAPPO Lagrange
CAMA-HAPPO (ours)
CAMA MAPPO lours)
Average Episode Reward
Average Episode Cost
Max Episode Cast
3500
800
Safety budget 50
250
2500
500
2000
150
5 1500
300
1000
300
100
0.0
0.0
0.0
4000
HailCheethh 3x2
6:0
400
2000
300
400-
200
100
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
2300
700
8:00
600
ManyAnt 2x3
2000
6000
1500
1000
200
-500
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0,8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment stops
Figure 2: CAMA in SMAMuJoCo with joint reward. Performance comparisons between CAMA- MAPPO, CAMA-HAPPO with their rivals in the CTDE settings. Our approach converges average and max costs to a priori safety budgets. Easily enable non-safe MARL algorithms that meet con- straints.

---

Under review as a conference paper at ICLR 2023
Main Result in CTDE. Figure demonstrate the performance comparisons in the CTDE setting, including tasks of Ant 2x4d. Half-Cheetah 3x2, Many Ant 2x3 in terms of Average Episode Reward, Average Episode Cost and Max Episode Cost. The dotted line are the safety budget set to 50 for all tasks. In terms of cost, compared to the original non-safe algorithms MAPPO and HAPPO, the CAMA version ensures that the average cost stays close zero and the Max cost converges to the safety budget. Regarding reward, the CAMA-HAPPO consistently outperforms MACPO and MAPPO-L but performs its original non-safe algorithm. The CAMA-MAPPO can hold similar performance compared to MACPO but lower than MAPPO and MAPPO-Lagrame The figures show that our approach significantly improved the safety of the algorithm with only slight reduction in the efficiency of the reward.
5.2 IL EXPERIMENT
To demonstrate the plug-and-play nature of CAMA, and its generality, we also test its performance in both cooperative and competitive IL. games.
Sum Independent Reward Environments setup. (Cooperative) The setting of the en- vironment is similar to SMAMuloCo CTDE. Each agent has an independent policy (rather than each part of the robot), but updating the policy refers to its local state and reward. To better show the whole system's performance, we will calculate the sum of the independent re- wards and costs by each agent. Note that those measure values are only used for analysis.
(a) Sumo-Human
(b) Sumo-Ant
Figure 3: Gym Compete Sumo Scenarios: Agents compete in circular arena, each with the goal of knocking another agent to the ground or pushing them out of the ring.
Zero-sum Game Environments setup. (Com- petitive)We involve the CAMA in Gym Com- pete (Bansal al., 2017) to demonstrate per- formance in a competitive setting. The tested scene is shown in Figure which called Sumo. We mainly tested two different robots, namely Ant and Humans, and each color represents an agent (Note that, unlike SMAMuJoCo, the robot is no longer divided into multiple parts). The Sumo- Human environment will be more difficult than the Sumo-Ant, since the initial state dimensions of the two robots are different (DHuman 24; D 15), please refer to Appendix for details.
In sumo competition, each player must stay on the field within 5m. And try to push your opponent out of bounds as much as possible to win. For each agent, the closer the edge, the more dangerous it is. Therefore, CAMA can convert "not falling" into a condition that needs to be met. The direct idea to use the distance from the agent to the center as reference. We set the safe radius to 2.5m and the safety budget to 1.0. The environment will give the agent a 0. cost when out of range. Similar to use five random seeds for testing and calculate the average winning
the previous implementation, rate and episode cost for analysing
Main Result in II. Figure demonntrates the results for the cooperative IL. setting, which includes Ant 2x4d, and Half-Cheetah 3x2 tasks. In terms of cost, compared to the original IPPO, the CAMA- IPPO ensures that the average cost stays close to zero, and the Max cost converges to the safety budget in both tasks. In terms of reward, CAMA-IPPO not only have a similar performance to IPPO in half- cheetah, but it also outperformed in the ant task. The reason probably stems from the reward setting of the environment, as the reward includes a portion of the cost penalty, leading to relatively higher reward for the safe algorithm. All in all, the figures show that our approach results in significant improvement in constraint satisfaction and limited enhancements in rewards.
Figure 5 shows the CAMA player verse the original IPPO player in the competitive II. game. In sumo-ant, CAMA can significantly improve the win rate while converging to the lower cost. We also found that CAMA tends to be a counter-attack player, which tries to attack the opponent while ensuring itself in a safe area. We speculate that CAMA will choose to avoid the attack due to the heavier penalties for violating the safety constraints in the later stage. On the contrary, the original IPPO attacked desperately and easily rushed to the edge. In the more challenging scenario human, we found that although CAMA will still be a low win rate at the beginning, will eventually tend to converge state of equilibrium (the winning rate of both sides tends to be

---

Under review as a conference paper at ICLR 2023
IPP
CAMA-IPPO [ours]
Average Episode Reward
Average Episode Cost
Max Episade Cost
2000
800
Safety budget
1750
700
1500
600
1250
1000
750
300
500
100
200
250
50
100
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
2500
800
2000
HaltChessan
1500
1000
400
200
0.0
0.4
0.6
Figure 4: CAMA in SMAMuJoCo with independent reward. Performance comparisons between CAMA-IPPO and IPPO the IL settings. With the benefit of satisfying the constraints, has been possible to exceed the performance of IPPO in the both scenario.
50%). According to the cost figure, in higher dimensional and more complex situations, CAMA is hard to learn attack strategies and instead aims to avoid attack and stalemate with the opponent.
This strategy is reminiscent Nonetheless, there is an essential difference between the two: CAMA players are defences chosen under the premise of satisfying constraints and still have the possibility the same time, adversarial strategies change the victim through the natural behaviour of the attacker state. Interestingly, the two perform similarly, and may be a potential research direction to guide the agent to learn aggressive strategies by adjusting constraints.
the attack strategy proposed by Gleave al. (2019) in performance.
anti-policy learning. At
Win Rate
Average Episede Cost
IPPO
4.0
CAMA-IPPO
0.8
2.5
2.0
0.4
1.5
1.0
0.0
000
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.00
0.25
0.50
0.75
1.00
1.55
1.50
1.75
2.00
5-
0.6
3-
0.4
2-
0.2
0.0
0-
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Figure CAMA in GymCompete with independent reward. Performance comparisons for CAMA-IPPO verse IPPO in competitive game. In both sumo-ant and sumo-human, CAMA based IPPO have higher win rate and lower average cost.
8

---

Under review as a conference paper at ICLR 2023
5.3 ABLATION STUDIES
The Dynamic Unsafe Reward in CTDE. In order to facilitate the calculation, the unsafe reward is usually defined as a very large negative number, which is feasible in the IL setting. However, in the CTDE environment, substantial unsafe rewards will lead to the problem of low learning efficiency and con-convergence, and we discuss the reasons in detail in Appendix D Intuitively, in the setting of CTDE, all other agents are severely penalized for a violation by one ogent,which affects the policy exploration of the safe agent. Therefore, we changed the unsafe reward from fixed to dynamic. We let the unsafety reward become proportional to the getting reward at the time step, as h2 (s, ar). After testing, we found that when one or more agents violate the constraints in multi- agent to some extent, this dynamic unsafe reward can reduce the impact on the rewards of other agents. Therefore, we do not need to tune this as pepee-paraeeter anymore.
Average Episode Reward
Average Episode Cost
Max Episode Cost
4000
250
500
3500
15
M
Safety Budget 50
400
3000
75
2500
150
300
2000
300
1500
500
100
200
1000
50
0.0
0.2
0.4
0..6
0.0
1.0
0.0
0.2
0.4
0..6
0.0
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 6: Ablation Result in Safety Budget, unsafe reward by = RR(s ac)
Safety Budget. As the initial value of the h, we can generalize the different budgets before or after the training process. We progressively selected seven different budgets in the Ant2x4 joint reward game, ranging from very low initial constraints "5" to almost no constraints at the beginning "500". The results are shown in Figure 6 and the final convergence details can be found in Appendix D table D It can be found that under the same setting of the unsafe reward- RR(*,, ae), all algorithms have high efficiency in average team reward, and the maximum cost can almost satisfy the constraints in the end. We set the safety budget as 50 in SMAMuJoco experiments and 1.0 in all Gym Compete Sumo experiments, and other details of the settings can be viewed in appendix
Ablation on CAMA components. Figure 7 demonstrates the performance after removing two critical components in CAMA. Removing constraint enhancement leads to lower learn- ing efficiency but still keeps the safety con- straints satisfied. It appears that the constraints and its budget value still be involved during re-
NO.RS
300
ward calculation, thus being forced converge to the conservative policy. Conversely, remov- ing reward reshaping has better rewards but no longer satisfies the constraints. While in con- trast, the cost is still lower than the original gorithm, which is related to the augmentation of the hazard value. Overall, this experiment demonstrates that these two components play crucial role in CAMA.
16
82
04
0.5
82
32
33
02
62
Figure 7: The ablation on CAMA components in joint reward Ant2x4d scenario. "No CA" stands for without constraint augmentation, "No RS" stands for without reward reshaping.
CONCLUSION
This work proposed CAMA, which is plug-and-play framework of Safe MARL using constraint augmentation. The proposed framework improve safety of existing multi-agent learning algorithms without pre-training and complex implementations. Furthermore, our framework is compatible with different MARL settings and interface seamlessly with both cooperative and competitive settings. Empirical results on cooperative and competitive tasks demonstrate the superiority of the CAMA- based al algorithms in improving safety and maintaining the same level of reward performance.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The details of the experiment settings are provided in Section 5.1 and 5.2 The hyparameter for the proposed algorithm and baselines are in Appendix [ The pseuco code is summarized in Algo-
rithm [ The ablation test detail are provided Section 5.3 and Appendix D We provide detailed
proof of theoretical analysis in Appendix A A more detailed description and implementation setting can be found in Appendix for SMAMujoco and Appendix C for Gym Compete.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML 2017), pp. 22-31. PMLR, 2017.
Abubakr O. Al-Abbasi, Arnob Ghosh, and Vaneet Aggarwal. DeepPool: Distributed Model-Free Al-
gorithm for Ride-Sharing Using Deep Reinforcement Learning. IEEE Transactions on Intelligent Transportation Systems, pp. pages 4714-4727, 2019.
Greg Anderson, Abhinav Verma, Isil Dillig, and Swarat Chaudhuri. Neurosymbolic reinforcement learning with formally verified exploration. Advances in neural information processing systems (NeurIPS 2020), volume 33:6172-61 2020.
Aristotle Arapostathis, Vivek Borkar, Emmanuel Fernández Femández-Gauuchrrinnn Gaucherand, Mrinal K Ghosh, and Steven Marcus. Discrete-time controlled markov processes with average cost criterion: survey. SIAM Journal on Control and Optimization, 31 31(2):882-344, 1993.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. arXiv preprint arXiv:1710.03748, 2017.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912 2019.
Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In TARK, volume 96, pp. 195-210. Citeseer, 1996.
Zhiyuan Cai, Huanhui Cao, Wenjie Lu, Lin Zhang, and Hao Xiong. Safe multi-agent rein- forcement learning through decentralized multiple control barrier functions. arXiv preprint arXiv:2103.1 103.1553,, 2021.
Miguel Calvo-Fullann Santiago Paternain, Luiz F. O. Chamon, and Alejandro Ribeiro. State aug- mented constrained reinforcement learning: Overcoming the limitations of learning with rewards. arXiv preprint urXiv:2102.1194// 2021.
Baiming Chen, Mengdi Xu, Zuxin Liu, Liang Li, and Ding Zhao. Delay-aware multi- agent reinforcement learning for cooperative and competitive environments. arXiv preprint arXiv:2005.0544/ 2020.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained re- inforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18 (1):6070-6120, 2017.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov- based approach to safe reinforcement learning. Advances in neural information processing sys- tems (NeurIPS 2018), volume 31, 2018.
Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 21(3): 1086-1095, 2019.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.0877 2018.
AN Daryin and AB Kurzhanski. Nonlinear control synthesis under double constraints. IFAC Pro- ceedings Volumes, 38( 277):247-252, 2005.
10

---

Under review as a conference paper at ICLR 2023
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020.
Dotan Di Castro, Aviv Tamar, and Shie Mannor. Policy gradients with variance related risk criteria.
arXiv preprint arXiv: 1506.6220.64444 2012.
Yali Du, Lei Han, Meng Fang. Ji Liu, Tianhong Dai, and Dacheng Tao. LIIR:
Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS 2019), volume 32. Curran Associates, Inc., 2019.
Ingy ElSayed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topcu, and Lu Feng. Safe multi-agent reinforcement learning via shielding. arXiv preprint arXiv:2101.11 2021.
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In International conference machine learning (ICML 2017), pp. 1146-1 1155-115, PMLR, 2017.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual Multi-gent Policy Gradients. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- 10615, 2019.
sarial policies: Attacking deep reinforcement learning. arXiv preprint
Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang. Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint arXiv:2110.02793, 2021.
Shangding Gu, Long Yang. Yali Du Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. review safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
Lei Han, Peng Sun, Yali Du, Jicchao Xiong, Qing Wang. Xinghai Sun, Han Liu, and Tong Zhang. Grid-wisc control for multi-agent reinforcement learning in video game AI. In Kamalika Chaud- huri and Ruslan Salakhutdinov (eds.), International Conference on Machine Learning (ICML 2019), volume 97, pp.
PMLR, 2019.
Onésimo Hemmnndez-Lerma and Jean B Lasserre. Discrete-time Markov control processes: basic optimality criteria, volume 30. Springer Science & Business Media, 2012.
Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning. In International Conference on Learning Representations (ICLR 2022), 2022.
Chenyi Liu, Nan Geng, Vaneet Aggarwal, Tian Lan, Yuan Yang, and Mingwei Xu. CMIX: Deep multi-agent reinforcement learning with peak and average constraints. In Machine Learning and Knowledge Discovery Databases. Research Track, pp. 157-173. Springer International Pub- lishing, 2021.
Ryan Lowe, Yi I. Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- environments. Advances in neural informa-
agent actor-critic for mixed fion processing systems (NeurIPS 2017), volume 30, 2017.
Songtao Lu, Kaiqing Zhang, Tianyi Chen, Tamer Başar, and Lior Horesh. Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning. In Proceedings the AAAI Conference on Artificial Intelligence, volume 35, pp. 8767-8775, 2021.
D-J Ma, Armand M Makowski, and Adam Shwartz. Estimation and optimal control for constrained markov chains. In 1986 25th IEEE Conference on Decision and Control, pp. 994-999999999999 IEEE, 1986.
11

---

Under review as a conference paper at ICLR 2023
David H. Mguni, Yutong Wu, Yali Du, Yaodong Yang, Ziyi Wang, Minne Li, Ying Wen, Joel Jen- nings, and Jun Wang. Learning in nonzero-sum stochastic games with potentials. In International Conference on Machine Learning (ICML 2021), volume 139, pp. 7688-7699, PMLR, 2021.
Afshin OroojlooyJadid and Davood Hajinezhad. A Review of Cooperative Multi-Agent Deep Rein- forcement Learning. arXiv preprint arXiv: 1908. 03963, 2021.
Peng Peng, Ying Wen, Yaodong Yang. Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang- Multiagent biidrectionally-Cordiiaaa Nets: Emergence of Human-level Coordination in Learn- ing to Play StarCraft Combat Games. arXiv preprint Xrviv: I 10069, 2017.
Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich, Rmix: Learning risk-sensitive policies for cooperative reinforcement learning agents. Advances in Neural Information Processing Systems, 01.23049-233665 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob N Fo- erster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent rein- forcement learning. Journal of Machine Learning Research, 21, 2020.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft aueti- challenge. arXiv preprint wrXiv::902.04044 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David Mguni, Jun Wang, and Haitham Bou-Ammar. Saute rl: Almost surely safe reinforcement learning using state aug- mentation. In International Conference on Machine Learning (ICMI 2022). PMLR, 2022a.
Aivar Sootla, Alexander I Towen-Rivers, Jun Wang, and Haitham Bou Ammar Enhancing safe exploration using safety state augmentation. arXiv preprint 22Xi::2206602675, 2022b.
Sainbayar Sukhbaatar and Rob Fergus. Learning multiagent communication with backpropagation.
Advances in neural information processing systems (NeurIPS 2016), volume 29, 2016. Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference machine learning (ICML 1993), pp. 330-337. PMLR, 1993.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: 1805. 1074, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model- control. In 2012 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS 20/2), pp- 0026-5033. October 2012.
Ho Wai, Zhuoran Yang. Zhaoran Wang. and Mingyi Hong. Multi-agent reinforcement learn- ing via double averaging primal-dual optimization. Advances in Neural Information Processing Systems (NeurIPS 20/8), 31, 2018.
Fangxin Wang, Feng Wang, Jiangchuan Liu, Ryan Shea, and Lifeng Sun. Intelligent video caching at network edge: A multi-agent deep reinforcement learning approach. In IEEE Conference on Computer Communications, pp. 2499-2508. IEEE, 2020.
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), volume 80, pp. 5571-5580. PMLR, 2018.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2/03.0/11 2021.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi- agent reinforcement learning with networked agents. In International Conference on Machine Learning (ICMI. 2018), pp. 5872-5881. PMLR, 2018.
12

---

Under review as a conference paper at ICLR 2023
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, Iman Fadakar, and Zheng Chen. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv preprint a:X1v:2010.09776. 2020.
13

---

Under review as a conference paper at ICLR 2023
A THEORETICAL ANALYSIS
In this paper, we extend the theoretical analysis for the single agent case Sootla et al. (2022a) to the multi-agent case. The authors of Sootla al. (2022a) define the problem in the following form,
max Esp.ao.... (T) a; (-11))
(8a)
t: JC(T) b,, W.P.
(8b)
where b is the safety budget and W.P. stands for with probability one. In this setting the goal is to find the policy (8a), which maximizes the task reward while making the cost less than a pre-defined
safety budget with probability one (8b). To facilitate training, they actually solved the following problem,
max E aB........ JR. a, x(s;)
R
Te $1, ar)
Their key result was show that with n -G the cost J converges monotonically JR. the optimal cost after solving Problem g The authors of Sootla et al. (2022a) invoked classical results for optimal control to prove this assertion (Hernández-I.c following three assumptions:
Lasserre 2012). They made the
Assumption The functions R(3,a) a) are bounded, measurable, nonnegative, and upper semi- continuous on S A
Assumption A is compact;
Assumption 4 The transition P is weakly continuous on S x A, i.e, for any continuous and bounded function on S the map (5, + Sz u(z, ()d(, dy(3, a) is continuous.
Note that in the results in Sections 4.2 and 4.3 in (Hernández-Leema used milder assumptions than Assumptions A1 and A2. However, Sootla et have found that As- sumptions Al and A2 are often met in RL applications, and hence there was need complicate the presentation. Assumption A3 is also mild in the RL setting as discussed in (Arapostathis et al.) 1993] in Section 2.4. Indeed, this assumption hols when the transition is a Gaussian distribu- tion with continuous mean and variance. Now, consider the problem with the objective J in Equation 9 satisfyyng the assumptions above. Then
Lasserre 2012 the authors
Theorem For any finite unsafe reward k, the Bellman equation is satisfied, i.e., there exists function V (3) such that
(8)
max (R) +3 aEA
where a)) Furthermore, the optimal policy solving JB has the representation a x (-18).
Theorem The optimal value functions V for J converge monotonically to V - -he optimal value function for J
Theorem Suppose there exists an optimal policy x- (-13e) solving Equation Iinuationoorrr the objective J with finite cost, then n² (-(3e) is an optimal policy for Equation
Now we discuss this results from MARL perspective. In the IL setting, the independent learners consider other agents as the environment. Hence intuitively if these independent learners satisfy the Bellman equation in the unconstrained case, then Theorem would show that safe IL would also satisfy the Bellman equation in the constrained case. Therefore. for the safe version and guarantee their convergence under those assumptions, at the same time each agent's ss optimal policy would be Markovian.
critic-based methods
14

---

Under review as a conference paper at ICLR 2023
In the CTDE setting, Assumption Al can be directly extended because the reward function will evaluate all agents and return the team reward by summing, in case of upper semi-continuous, JR - Jk According to (Kuba et al. 2022). in some CTDE cases we also need the following regularity assumptions, which in particular, guarantees that the set of policies is closed and bounded:
Assumption There exists R, such that < & 1, and for every agent E N,, the policy space IIo is 0-soft, which means for every x² E ITI, E S, and at E A', we have n¹ (at ( 2 0.
Indeed, according Lemma' 3 (Kuba al. 2022), let (=t) t=o be convergent sequence of policies of agent Then according to the Assumption A4, for any E N, E S, and a² A', oi (a Z 8, hence, n' (a" 8) = lim qे (a' 8) 2 lim 2 8. Furthermore, since sum of probability for actions sampled from policy is one, so that | (a' () S boundedness of the set of polices.
max S 1, which proved the
Finally, let us discuss Assumption A3 in CTDE. Although, the CTDE setting increases the dimen- sion of the hazard value according to the cost equation, each agent will eventually return the smallest and continuous hazard value for augmentation with the current state, so the assumption is still true. Therefore, we can extend Theorem to both CTDE and II. by satisfying the assumptions A1-A4.
B SMAMUJoCo ENVIRONMENTS:
Safe Multi-Agent MuJoCo (SMAMuJoCo) Gu et al. 2021 is an extension of MuJoCo (Todorov al. 2012). In particular, while retaining the background environment, proxy, physics simulator and reward functions, adding collision obstacles in the environment, such as fixed walls and moving bafiles, can be touched and knocked over when agents valid the constraint. During the training process, when the agent produces unsafe behaviours like crash, the environment will generate costs according to the settings of different scenarios. This section will introduce the cost settings for each task used in this paper.
MANYANT & ANT
AnnyAnt environment has linear corridor with width of 9m. Ants (8b) environment has three sections of polyline corridors with an angle of 30 degrees, and the width between the two walls is 10m. In both environments, which returns cost the distance between the agent and the wall is less than 1.8m or the agents falls over. Therefore,
0, (Safe), If 0.2 < #+1 S 1.0 and [
agent
x wall 1|, 2 1.8,
=
1, (Unsafe), Otherwise
agent is the agents torso's x-coordinate. For agent is the agents torso's 2-coordinate, and z where 2++1 each time step, the system need to calculate the distance between agents torso with x wall which is the z-coordinate of the wall.
cerri
(a) ManyAnt Scenario
(b) Scenario
HALP-CHEETAI
15

---

Under review as a conference paper at ICLR 2023
In this mission, the agent moves in special corri- dor, the fixed walls on both sides no longer incur costs and do not get knocked down, but which will restricts agents' movement. Newly added is moving baffle in the corridor, which incurs cost of if the distance between the agent and the baflle is less than m, then the cost et can be demonstrated as,
0, (Safe), If B141
agent yhull 2 9
Figure 9: Ant 2x4
C: 1, (Unsafe), otherwise.
where V
agent is the y-coordinate of the agents torso, and blffie is the y-coordinate of the baffle.
C COMPETITIVE ENVIRONMENTS:
5m
2.5m
Safe
Unsafe
Figure 10: Sumo area
Gym Compete (Bansal al. 2017) Sumo allows agents compete in 1-vs-1 system following standard sumo rules. There are two types of agents, Ant and Humans, which have different anatom- ical structures. During the game, The players' positions observes its own and its opponent's positions, its own joint angles, the corresponding speed, and the force applied to its own body (i.e., the equivalent of touch). The action space is continuous. Figure describes desseribes the safe area in the arena, that is, circle with radius of 2.5m. The cost given by the environment is set as follows.
randomly initialized, and each agent
0, (Safe),
2 t+1
x
center
2.5
C = 0.1, (Unsafe), otherwise.
2
where agent circle.
z-coordinate of agents torso, and center is the x-coordinate of the center of
16

---

Under review as a conference paper at ICLR 2023
D ABLATION EXPERIMENTS
b
Average
Max C
Average R
Max C
0
1709.78 + 123.66
8
R(st, at)
3299.01 + 518.19
14..0 6.41]
2233.00 429.62
(6.2+0.499
-0
2959.334 357.12
+3.0
15
2508828 + 390.03
[14.7
-5
1964.27 = -311.90
[45.0 1..0+9.5550 9.559
50
33999001 + 518.19
14.0+...4
-10
[1447.00 = -364.32
[41.0 12.77
75
[2923.27 = 180.27
11.84
-20
[1170.36 -371.57
(69.33 =
150
33376288 = 603.48
166 = 24.07
-50
[522.46 = 195.17
= 19.05
= 17..0 7.799
300
(3627.82 576.99
273 - 65.65
-100
[364.72 = 51.68
500
[3737.58 97.50]
416 89.43
1000
[31.34 - 18.55]
[48.0 = 12.86)
Table Ablation test in Safety Budget 6
Table 2: Ablation in Unsafe Reward k
Further Ablation experiments for unsafe rewards. In previous experiments, we know that for different scenarios Markov game settings, we should use different unsafe rewards. Here, we test the performance of both fixed and dynamic values of k under safety budget of 50, and present the results of one million training evaluations in Table was found that when the fixed & reduced, the policy learning is inefficient even unable to converge. This result significantly differs from the single-agent setting. We consider this a unique situation worth discussing in the multi-agent environment. Under the CTDE setting, when one of the agents violates the constraint, the team reward becomes the unsafe reward times the number of agents. Then means that the reshaped rewards will be much smaller than the real rewards given to the agents by the environment. As result, the centralized critic will ignore the general policy exploration of agents who do not violate the constraints, resulting in low training efficiency. At the to other numerical issues in training. Therefore, for this situation, it is necessary to balance the relationship between environmental reward and unsafe reward to improve learning efficiency while satisfying the constraints. In order to do that, an intuitive idea is to set dynamic that grows larger over the time, and eventually reaching negative infinity. Thus, we k R(s,, a.). The advantage is that the reward is small at the beginning, so the corresponding penalty is also small, which encouraging agents explore the policy. With this dynamic setup, we find that CAMA can better adapt to difficult tasks and tends to converge (Table 2 first row). Therefore, in CTDE and IL, we set (s,, dt).
time, too small h: also lead
Through the detailed convergence comparison in Table 2 we finally determined to use -0.2 as an unsafe reward the Ant environment to ensure the stability of learning.
HYPER-PARAMETERS SETTING FOR EXPERIMENTS
The following four tables describe all the hyperparameters used in practice, respectively. For CAMA-base MARL algorithms, besides the Safety Budget and Unsafe Reward, other parameters are the same as the occupation, in order to have the fair comparison in experiments. Notably, these used hyperparameters all cited from their best performing experiments in the original paper.
hyperparameters
value
hyperparameters
value
hyperparameters
value
critic Ir gamma((Te>x)
5e-3
optimizer
Adam
num mini-batch
40
0.99
optim eps
le-5
batch size
16000
gain
0.01
hidden layer
training threads
std y coef
0.5
actor network
mlp
rollout threads
16
std coef
1
eval episodes
32
episode length
1000
activation
ReLU
hidden layer dim
64
max grad norm
10
Table 3: Common hyperparameters used for CAMA-HAPPO. CAMA-MAPPO, CAMA-IPPO, MAPPO-Lagrangiann MAPPO, HAPPO, IPPO, and MACPO in the Safe Multi-Agent MuJoCo Anny Ant, Half-Cheetah, and Gym Compete Sumo-Ant, Sumo-Human
17

---

Under review as a conference paper at ICLR 2023
Algorithms
CAMA-HAPPO
HAPPO
CAMA-MAPPO
MAPPO
MACPO
actor Ir ppo epoch kl-threshold ppo-clip fraction fraction coef
9e-5
9e-5
9e-5
9e-5
5
5
/
0.0065
0,2
0.2
0.2
0.2
0.5
/
0.27
I Lagrangian coef Lagrangian
Table Different hyperparameters used for CAMA-HAPPO НАРPО, CAMA-MAPPO, MAPPO, and MACPO in the SMAMuloCo domains.
Algorithms
CAMA-IPPO
IPPO
MAPPO-Lagrangian
actor Ir ppo epoch kl kl-threshold ppo-clip fraction fraction coef Lagrangian coef Lagrangian
9e-5
9e-5
9e-5
5
5
5
1
7
0.2
0.2
0.2
/
/
0.78
le-3
Table 5: Different hyperparameters used for CAMA-IPPO IPPO, MAPPO-Lagrangian in the both SMAMuoCCo and Gym Compete domains.
task
Safety Budget
Unsafe Reward
Ant(2x4d)
50
-R(se, ac)
HalfCheetah(3x2)
40
-R(st, ac)
ManxAA((223)
30
-R(s ac)
Table 6: The new two hyper-parameters Safety Budget and Unsafe Reward used for HAMA-- CAMA-MAPPO, CAMA-IPPO in the SMAMuJoCo domains.
task
Safety Budget
Unsafe Reward
Sumo-Ant
1.0
RR(s,, a+)
Sumo-Human
1.0
-R(s
Table The new two soper-parameter, Safety Budget and Unsafe Reward used for CAMA-IPPO in the Gym Compete domains.
18


Reviews:
[{"review_id": "wNTo8CfF1y", "reviewer": "Reviewer_CFTd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces CAMA, a safety approach for MARL agents based on constraint augmentation of the search space, and especially the reward function under the notion of a safety budget and hazard values.\nCAMA is integrated into two MARL paradigms: CTDE and IL.\nExperiments are performed to compare CAMA to baselines methods. ", "summary_of_the_review": "The CAMA idea seems good and the results confirm that the safety of the agents is improved.\nI have some concerns regarding the degradation of performance in some settings and the actual novelty of the work.", "strength_and_weaknesses": "The approach of CAMA is compatible with many MARL algorithms and does not require too big changes to be introduced.\nExperiments show that the safety budget is better honoured than in other approaches, and that in some cases a better performance in terms of reward is achieved.\nAt the same time, the reward improvement is not consistent and in some environments CAMA performs worse (although honouring safety).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d2NoTGkUbU", "reviewer": "Reviewer_FCBX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors introduce the Constraint Augmented Multi-Agent framework — CAMA, which can serve\nas a plug-and-play module to the popular MARL algorithms. They propose an approach that represents the safety constraint as discounted safety costs, known as the safety budget and demonstrate in experiments that CAMA converges  quickly to policies with constraint satisfaction and have better return performance when compared with other state-of-the-art safety RL algorithms.", "summary_of_the_review": "The work is quite neat in terms of addressing the safe RL problem in the MARL setting. The CAMA framework is also quite flexible and can be adapted to different MARL algorithms. Experiments potentially show the superiority of this algorithm in terms of enforcing safety for multi-agent systems as well as effective return maximzation.\n\nHowever, novelty wise I think the underlying technique is a direct extension of safety budget in standard safe RL, which is quite incremental.", "strength_and_weaknesses": "Strengths:\nIn this work, the authors proposed the CAMA framework for safety-aware multi-agent RL problems.\nCAMA is quite modular and can be used in conjunction with different standard MARL algorithms (to make them safety-aware). \nEmpirically the authors showed CAMA beats most multi-agent safe RL SOTA in terms of safety and return maximzation.\n\nWeakness:\nThe idea of safety budget has been well studied by many previous paper, but mainly for single constraint safe RL problem. It seems besides the CAMA framework and experimentations, the main novelty is on using this safety budget for MARL setting, which is quite incremental.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ts-IosXqp4", "reviewer": "Reviewer_VkFF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents CAMA, a framework for multi-agents that incorporates safety constraints into multi-agent reinforcement learning algorithms. The framework can be added to different multi-agent reinforcement learning algorithms as a plug-and-play method. The safety constraints are represented as the sum of discounted safety costs. Since there is a safety budget, it is easy to compute the hazard value, which is the remaining safety budget. The paper includes empirical results that show the effectiveness of the method.", "summary_of_the_review": "The paper presents a module that can be added to multi-agent reinforcement learning algorithms to enforce safety constraints. The method is supported by extensive experimental results obtained in simulation.", "strength_and_weaknesses": "Strengths: \n- Having a modular component that can be added to multi-agent reinforcement learning algorithms. \n- Having a method that works both with centralized training with decentralized execution and with independent learning.\n- The experimental results are extensive. They have been obtained in multiple scenarios and comparing different algorithms.\nWeaknesses: \n- The contribution is valid but the novelty is limited.\n- All the proofs are in an Appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Gu0hRPIBi5", "reviewer": "Reviewer_nfFv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors introduced the framework of Constraint Augmented Multi-Agent (CAMA) for solving constrained multi-agent reinforcement learning problems. Earlier Sootla et al. [2022] augmented the safety constrains into the cost function by defining a safety budget and panelizing the agent when crossing that budget. CAMA brings this trick into the multi-agent setting for both the centralized training and decentralized execution (CTDE), and individual learning (IL). The latter is pretty similar to the earlier work (single agent), while the CTDE required more expansion. Authors compared their algorithms against 4 approaches in 5 domains showing promising results. \n", "summary_of_the_review": "The paper has great direction, yet I am not sure if the contributions of the paper is enough for a publication. Moreover the writing of the paper needs non-trivial amount of work.", "strength_and_weaknesses": "Strengths\n+ Bringing the constraint state augmentation into the multi-agent setting is appealing\n+ The approach can be combined with any method, as the result of CAMA formulation is a MARL problem.\n\nWeaknesses\n- The writing is not great. Several improvements are needed (see details below)\n- Certain definitions seems incorrect. For example why \\tilde R_i is assumed to be non-negative in Definition 1 given that k is often negative? (see details below)\n- The contribution seems incremental given the work of [Sootla et al. 2022].\n- Experimental results don't show a major difference with MAPPO-Lagrange approach. In the literature review section authors mentioned: \"Although these methods satisfy the constraints to a certain extent, a part of the reward performance is still sacrificed\". This is also the case for their approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject