Title: Multi-Agent First Order Constrained Optimization in Policy Space
Venue: NeurIPS 2023
Accepted: yes
Keywords: Safe Multi-agent Reinforcement Learning

constrained policy optimisation

first-order optimisation
Abstract: In the realm of multi-agent reinforcement learning (MARL), achieving high performance is crucial for a successful multi-agent system.
Meanwhile, the ability to avoid unsafe actions is becoming an urgent and imperative problem to solve for real-life applications. 
Whereas, it is still challenging to develop a safety-aware method for multi-agent systems in MARL. In this work, we introduce a novel approach called Multi-Agent First Order Constrained Optimization in Policy Space (MAFOCOPS), which effectively addresses the dual objectives of attaining satisfactory performance and enforcing safety constraints. Using data generated from the current policy, MAFOCOPS first finds the optimal update policy by solving a constrained optimization problem in the nonparameterized policy space. Then, the update policy is projected back into the parametric policy space to achieve a feasible policy. Notably, our method is first-order in nature, ensuring the ease of implementation, and exhibits an approximate upper bound on the worst-case constraint violation. Empirical results show that our approach achieves remarkable performance while satisfying safe constraints on several safe MARL benchmarks.

Multi-Agent First Order Constrained Optimization in Policy Space

Abstract

In the realm of multi-agent reinforcement learning (MARL), achieving high performance is crucial for a successful multi-agent system. Meanwhile, the ability to avoid unsafe actions is becoming an urgent and imperative problem to solve for real-life applications. Whereas, it is still challenging to develop a safety-aware method for multi-agent systems in MARL. In this work, we introduce a novel approach called Multi-Agent First Order Constrained Optimization in Policy Space (MAFOCOPS), which effectively addresses the dual objectives of attaining satisfactory performance and enforcing safety constraints. Using data generated from the current policy, MAFOCOPS first finds the optimal update policy by solving a constrained optimization problem in the nonparameterized policy space. Then, the update policy is projected back into the parametric policy space to achieve a feasible policy. Notably, our method is first-order in nature, ensuring the ease of implementation, and exhibits an approximate upper bound on the worst-case constraint violation. Empirical results show that our approach achieves remarkable performance while satisfying safe constraints on several safe MARL benchmarks.

1 Introduction

Cooperative multi-agent systems play a significant role in various domains, where a group of agents coordinate with each other to accomplish tasks and collaboratively optimize cumulative rewards for the team[1, 2]. Such a setting is frequently employed in many real-life scenarios such as robotics [3], autonomous vehicles [4], traffic light control [5] and the smart grid [6].Thanks to the recent remarkable advance of reinforcement learning techniques in various complex tasks [7-9], multi-agent reinforcement learning (MARL) has attracted substantial attention and quite a few algorithms have been proposed, including value-based methods [10-14] and policy gradient methods [15-18]. Despite the notable achievement in academia, most MARL algorithms prioritize policy optimization solely for reward maximization, while disregarding potential negative or harmful consequences resulting from the agents' behaviors. Consequently, these methods can not be directly deployed in practice. In reality, many applications often require the agents to refrain from taking certain actions or visiting particular states [19, 20]. For instance, an unmanned car must adhere to traffic regulations by not crossing a red light, even while pursuing its destination, in order to prioritize safety.

To address the above issues, researchers have been devoted to developing algorithms that learn policies which adhere to safety constraints and great progress has been made in single-agent RL setting[2023]. However, it is still a daunting challenge to develop safe policies for multi-agent systems. Because of the existence of multiple agents, the environment may suffer from non-stationarity due to simultaneously learning agents, posing a non-negligible challenge to the training process. Moreover, ensuring safety in MARL is highly intricate. To provide a better depiction of the inherent challenges,

3

Policy Space for Agent 1

Policy Space for Agent 2

0.500

0.857

-0.85

Safe area for agent 1

1.429

-0.857

-0.500

0

1.140

-0.500

-0.286

consider safety

0

Safe area for agent 2

0.857

2500

-0.286

28

direction

X

Update

0.000

opoconsider.safety

+d.500

0.286

-0.286

Updatedirection

0.857

-0.500

0.2

0.286

9

140

-0.857

-1.140

1.429

JointSafeArea

-1.14.

-0.857

0.500

6

0.500

0.857

we show an illustrative example in Figure 1. This figure showcases the policy space for each agent, wherein the reward distribution is presented. The green rectangles delineate the individual safe areas for each agent and the blue ones depict the joint safe area. The green arrows symbolize the update direction for each agent, whereas the dashed ones signify the intended update direction if each agent solely considers its own objectives. When agents are randomly initialized, they naturally tend to update the policy along the dashed line arrows to maximize the reward. However, due to the presence of safety constraints, they must adjust their updates along the solid line arrows to ensure their policies fall within the safe area. This is non-trivial as it may conflict with their goal of reward maximization. Furthermore, after individually addressing their safety constraints, the agents must also consider the safety constraints of others to collectively converge to the joint safe area, adding more complexity to the optimization process for the entire system. The interplay of these factors underscores the inherent difficulties in solving the problem of safe MARL.

Recently, as a notable advancement in the field of safe MARL, Multi-Agent Constrained Policy Optimization (MACPO) [24] has been proposed as a safe and effective solution. MACPO attains the properties of both monotonic improvement guarantee and safety constraints satisfaction guarantee at every iteration during training. However, this algorithm involves solving an optimisation problem using Taylor approximations and inverting a high-dimensional Fisher information matrix. As a result, the computation is very complex and the policy update often becomes infeasible. To alleviate this issue, it requires additional recovery steps, which, however, sometimes causes updates to be backtracked and samples to be wasted.

Taking inspiration from the sequential policy update scheme introduced in HATRPO [18], we have devised a simpler approach to incorporate safety constraints in solving safe MARL problems compared with MACPO. The resulting algorithm, Multi-Agent First Order Constrained Optimization in Policy Space (MAFOCOPS), aims to address the following question of how to achieve the best constraint-satisfying policy update given the current policy for each agent. Our method follows a twostep process to give a solution to this problem. First, based on the theoretical foundations presented in MACPO, we demonstrate that the best policy update has a near-closed form solution when attempting to solve the optimal policy in the nonparametric policy space. Subsequently, we project the policy back into the parametric policy space as direct evaluation of the optimal policy is usually not feasible. This can be realized by sampling from the current policy and evaluating a loss function between the parameterized policy and the optimal policy obtained in the nonparametric policy space. Notably, our algorithm only employs first-order approximations, making it straightforward to implement, and has an approximate upper bound for worst-case constraint violation. To validate the effectiveness of our approach, we conduct experiments on two safe MARL benchmarks proposed by [24], namely Safe MAMuJoCo and Safe MAIG. The experimental results demonstrate the superior performance of MAFOCOPS compared to MACPO, despite being a simpler algorithm.

2 Related Work

Safety has become a crucial and longstanding concern in the field of reinforcement learning [25]. In this section, we discuss recent advancements in the domain of safe RL for multi-agent environments.

The realm of safe multi-agent reinforcement learning is a nascent area of research that has gained increasing importance [26]. Several attempts have been made to tackle safe MARL, but most existing approaches have limitations. For example, CMIX [27] leverages the value function decomposition method and modifies the reward function to account for constraint violations, yet this algorithm fails to provide safety guarantees during training. Another approach is Safe Dec-PG [28] which proposes a decentralized policy gradient descent-ascent method by means of a consensus network and employs a primal-dual framework to balance reward maximization and cost minimization. Nevertheless, the consensus requirement in this work equivalently imposes an extra constraint of parameter sharing among neighbouring agents, potentially leading to sub-optimal policy [18]. It is noteworthy that most current safe MARL methods are tailored to robotics tasks, utilizing techniques such as barrier certificates [29, 30] or model predictive shielding [31] to address safety issues. These methods, however, are specifically designed for robotics applications. Besides, they often require supervised learning based approaches or specific assumptions concerning the state space and environment dynamics.

As a recent remarkable solution, MACPO [24] incorporates a sequential policy update scheme. This algorithm develops the multi-agent trust region learning based on CPO [20] and provides theoretical guarantees of both monotonic improvement in reward and compliance with cost constraints in a multi-agent setting. To achieve practical solutions to the safe MARL problems, the policy for each agent needs to be parameterized with a neural network, which effectively represents the policy in policy space. However, achieving parameterized policies in MACPO involves solving optimization problems using first and second-order Taylor approximation, which includes taking the inverse of Fisher information matrix, and the computation is implemented by the conjugate gradient method [32]. These operations can introduce nonnegligible approximation errors, thereby compelling MACPO to undertake additional steps during each update in the training process in order to recover from constraint violations. In contrast, our algorithm takes a different approach by solving the optimization problem within the nonparametric space and then projecting the results back into the parameter space. By leveraging a simple first-order method to eliminate the approximation error, our algorithm ultimately outperforms MACPO.

3 Problem Formulation

A safe MARL problem can be formulated as a Constrained Markov Decision process, which is described by a tuple denoted as < N , S , A , p, ρ 0 , γ, R, C , c > . Here, N = { 1 , 2 , · · · , n } denotes the set of agents involved in the system, S is the state space, A = Π n i =1 A i represents the product of the action spaces of agents, i.e., joint action space, p : S × A ×S → R is the probabilistic transition function, ρ 0 is the initial state distribution and γ ∈ [0 , 1) is the discounted factor. The team reward function R : S × A → R maps state and joint actions to a scalar reward while C = { C i j } i ∈N 1 ≤ j ≤ m i is the set of sets of cost functions denoted in the form C i j : S × A i → R (every agent i has m i cost functions). The set of corresponding cost-constraining values is given by c = { c i j } i ∈N 1 ≤ j ≤ m i . At time step t , the multi-agent system is situated in state s t and every agent i selects an action a i t based on its policy π i ( a i | s ) , forming a joint action a t = ( a 1 t , · · · , a n t ) and joint policy π ( a | s ) = Π n i =1 π i ( a i | s ) . This leads the environment to transit to a new state s t +1 ∼ p ( ·| s t , a t ) according to the probabilistic transition function and the system receives the reward R ( s t , a t ) while each agent i incurs individual costs C i j , ∀ j = 1 , · · · , m i . This study focuses on a fully-cooperative multi-agent setting where all agents share the same reward function, aimed at maximizing the expected total reward:

Moreover, we impose that each agent i satisfies its safety constraint, which is defined as

We define the state-action value and the state-value function in terms of reward as

̸

It's worth noting even though the action a i t taken by agent i does not directly impact the costs { C k j ( s t , a k t ) } m k j =1 of other agents k = i from the above formulation, the total costs can still be influenced by this action implicitly due to its influence on the subsequent state s t +1 . This formulation captures the realistic multi-agent interactions in real world. For instance, when a car runs a red light, although other cars may not be immediately endangered by this action, the resulting disruption in traffic flow may lead to potential hazards later on. To illustrate the j th cost function of agent i , we express the corresponding state-action cost value function and the state cost value function as below:

Notably, although similar in form to traditional Q π and V π , the cost value function Q j, π and V j, π involve additional indices i and j , where the subscript i refers to an agent and j denotes the j th cost.

Motivated by the sequential policy update scheme, we pay close attention to determining the contribution of different subsets of agents to overall performance in this study. We denote an arbitrary subset { i 1 , · · · , i h } of agents as i 1: h while -i 1: h refers to its complement. Given the agent subset i 1: h , we define the multi-agent state-action value function:

Furthermore, for disjoint sets i 1: h and j 1: k , the multi-agent advantage function is defined as follows:

An interesting and critical observation concerning the aforementioned multi-agent advantage function is that the advantage A i 1: h π can be written as the sum of sequentially-unfold multi-agent advantages of individual agents, that is,

Lemma 1 (Multi-agent advantage decomposition [18]) For any state s ∈ S , subsets of agents i 1: h ∈ N and joint action a i 1: h , the following identity holds

4 Method

Expanding on the above foundational concepts and the derivatives of multi-agent trust region learning with constraints, MACPO provided an important insight. It highlighted that when the policy changes for all agents are sufficiently small, each agent can learn a better policy ¯ π i by only considering its own surrogate return and costs, which is consistent with the sequential policy update scheme. In our work, building upon the formulas in MACPO, we can deduce that, for agent i h and the index of its cost functions j , given the joint policy π θ k at the k th iteration and updated policies of the previous agent sets i 1: h -1 , namely π i 1: h -1 θ k +1 , the new policy is obtained by solving the following optimization problem:

where ¯ D KL ( π i h θ , π i h θ k ) ≜ E s ∼ ρ π θ k [ D KL ( π i h θ k ( ·| s ) , π i h θ ( ·| s ))] . A simple proof is presented in Appendix A. When solving the optimization problem (1-3), we use a two-step approach:

For agent i h , when provided with the joint policy ¯ π θ k and updated policy π i 1: h -1 θ k +1 , find an optimal update policy π i h ∗ in the nonparameterized policy space, denoted by Π .

Project the policy found in previous step back into parameterized policy space Π θ by solving for the closest policy π θ ∈ Π θ to obtain π i h θ k +1 . Here we consider the set of parameterized policies Π θ = { π θ : θ ∈ Θ } ∈ Π which allows for evaluation and sampling.

4.1 Finding the Optimal Update Policy

In the first step, we aim to find the optimal nonparameterized policy and propose the following solutions (see Appendix B for proof):

Theorem 1. For agent i h , we define b i h j = c i h j -J i h j ( π θ k ) . If π θ k is a feasible policy, the optimal policy takes the form

where η π θ k ( s, a i h ) = E a i 1: h -1 ∼ π i 1: h -1 θ k +1 [ A i h π θ k ( s, a i 1: h -1 , a i h )] , Z λ j ,ν j ( s ) is the partition function that ensures Equation 4 to be a valid probability distribution, and λ j as well as ν j are solutions to the following optimization problem:

The structure of the optimal policy exhibits an intuitive nature, as it assigns a substantial probability mass to regions of the state-action space with high returns. This allocation is counterbalanced by a penalty term multiplied by the cost advantage. Another desirable property of the optimal update policy is that for feasible policy π θ k , it has an upper bound for worst-case guarantee for cost constraint satisfaction according to Lemma.2 in MACPO [24].

4.2 Approximating the Optimal Update Policy

When addressing the optimization problem 1-3 in the first step, the optimal update policy for agent i h may not necessarily reside within parameterized policy space Π θ . Consequently, evaluating or sampling from this policy becomes unfeasible. To this end, in the second step, we need to project the optimal update policy back into the parameterized policy space by minimizing the loss function:

In this context, π i h θ ∈ Π θ represents some projected policy that serves as an approximation of the optimal update policy. To minimize this loss function, first-order methods can be employed. In doing so, we can leverage the following result as a useful tool in our optimization efforts:

Corollary 1. The gradient of L ( θ ) takes the form

where

The proof is shown in Appendix C.

It's to be noted that 27 can be estimated by sampling from the trajectories generated by policy π θ k , which allows us to train our policy using stochastic gradients, a key aspect of our methodology. This corollary serves as a guiding framework for our algorithm. At every iteration, we begin with a policy π θ k , utilizing it to generate trajectories and collect relevant data. Subsequently, we employ this data in conjunction with 5 to estimate λ j and ν j . We then draw a minibatch from the collected data to estimate ∇ θ L ( θ ) as outlined in Corollary 1. After taking a gradient step using Equation 27, we draw another minibatch and repeat the process.

4.3 Practical Implementation

For agent i h , solving the dual problems presented in 5 is computationally challenging when dealing with large state/action spaces as calculating the partition function Z λ j ,ν j ( s ) often involves evaluating a high-dimensional integral or sum. Moreover, λ j and ν j are dependent on the iteration k and need to be adjusted at every iteration to ensure the effectiveness of optimization.

Based on the structure of the optimal update policy, it is observed that as λ j → 0 , π i h ∗ tends to be greedy while the policy becomes more exploratory when λ j increases. We note that λ j exhibits similarities to the temperature term utilized in maximum entropy reinforcement learning [33]. Fixed values of λ have been demonstrated to yield reasonable outcomes in training [34, 35, 22]. In practical implementations, we have observed favorable results by using fixed λ j through hyperparameter sweeps. However continuous adaptation of ν j during training is necessary to ensure cost constraint satisfaction. In this regard, we appeal to an intuitive heuristic based on primal-dual gradient methods [36] to determine the appropriate value of ν j . Recall that by strong duality, the optimal λ ∗ j and ν ∗ j minimizes the dual function 5 which we will denote by L ( π i h ∗ , λ j , ν j ) . Then we can adopt gradient descent w.r.t ν j to minimize L ( π i h ∗ , λ j , ν j ) as follows:

Corollary 2. The derivative of L ( π i h ∗ , λ j , ν j ) w.r.t ν j is

The proof is shown in Appendix D.

The last term in the gradient expression poses a challenge since π i h ∗ may not locate in parameterized policy space, leading direct evaluation of the term to be infeasible. Nevertheless, due to the proximity between π i h θ and π i h ∗ enforced by the KL divergence constraint, it's reasonable to assume that E s ∼ ρ π θ k ,a i h ∼ π i h ∗ [ A i h j, π θ k ( s, a i h )] ≈ E s ∼ ρ π θ k ,a i h ∼ π i h θ k [ A i h j, π θ k ( s, a i h )] = 0 . In practice, we have observed that setting this term to zero yields favorable outcomes, which gives the update term as follows:

where α is the step size to control the magnitude of the update. The projection operator proj ν j ensures that ν j remains within the interval [0 , ν max ] , with ν max chosen to prevent ν j from being excessively large. In practical applications, the estimation of J i h j ( π θ k ) can be accomplished using Monte Carlo methods by leveraging trajectories collected from π θ k . This approach aligns with the update rule employed in MACPO [24]. We recall that in 4, ν j acts as a cost penalty term where increasing ν j makes the likelihood of state-action pairs with higher costs being sampled by π i h ∗ diminish. Consequently, the update rule presented in 10 exhibits an intuitive characteristic: it raises ν j if J i h j ( π θ k ) > c i h j , indicating a violation of the cost constraint for π θ k , and reduces ν j otherwise. Using the proposed update rule, ν j can be updated before updating the policy parameter π i h .

To be noted, our algorithm is a first-order method, which implies that the approximations made are accurate only around the initial condition ( i.e., π i h θ = π i h θ k ). To better enforce this condition, we have introduced a per-state acceptance indicator function I ( s j ) = 1 D KL ( π i h θ ,π i h θ k ) ≤ δ to 27. This function helps in rejecting sampled states whose D KL ( π i h θ , π i h θ k )[ s ] is too large, thereby improving the accuracy of the gradient update. The resulting sample gradient update equation is as follows:

where N is the number of samples collected using policy π θ k . The estimates of the advantages functions for the returns and costs, denoted as ˆ η π θ k and ˆ A i h j, π θ k , are obtained from critic networks referring to MACPO. We estimate these advantages using Generalized Advantage Estimator (GAE) [37] and apply stochastic gradient descent using 11. During the training process, our algorithm employs the early stopping criterion to ensure that the updated policy satisfies the trust region constraint. Specifically, this criterion is defined as 1 N ∑ N j =1 D KL ( π i h θ || π i h θ k )[ s j ] > δ . To update the value network, we minimize the Mean Square Error (MSE) between the output and target value. The procedure of our algorithm is presented in the Appendix E.

JUY,

(a)

(b)

(c)

(d)

5 Experiments

We evaluate the effectiveness of our algorithm on two benchmarks of safe MARL: Safe MAMuJoCo and Safe Multi-Agent Isaac Gym (MAIG). The former is a safety-aware modification of MAMuJoCo [38], where there exist obstacles in the environment. Meanwhile, Safe MAIG is developed on top of Issac Gym [39], a GPU-based platform for robotics tasks. Being an extension of DexterourHands [40], Safe MAIG requires agents to control the robot hands while optimizing both the reward and safety performance. We present some example tasks in Figure 5 and more details about the environments are introduced in the appendix.

While our primary aim is to propose a simpler alternative to MACPO, we also evaluate the performance of MAPPO-Lagrangian (MAPPO-L), which is put forward alongside MACPO [24]. This method adopts Lagrangian multipliers to solve optimization problems in MACPO. Without the requirements of repetitive computation of the Hessian matrix whose size grows quadratically with the dimension of the parameter vector, the Lagrangian method is also first-order and simple to implement. However, unlike MACPO and MAFOCOPS, whether this algorithm satisfies any worst-case constraint guarantees remains unknown. In addition, we compare the results of our algorithm with two standard MARL baseline algorithms, namely MAPPO [41]and HAPPO [18]. Both sets of experiments are carried out using the MACPO codebase and our experiments are conducted on GeForce RTX 3090 GPUS. More implementation details can be found in supplementary materials.

5.1 Performance on Safe MAMuJoCo

In this section, we select several experiment scenarios of Safe MAMuJoCo and execute each algorithm for 10 million samples per task. The cost thresholds are determined by taking 50% of the cost achieved by standard MARL algorithms after 1 million sample runs. To be noted, while most training parameters remain the same as those in the codebase, such as the learning rate and settings of optimizers, we adjusted some certain hyperparameters associated with the cost thresholds to make the algorithms best suit the experiment scenarios. It is noteworthy that the performance of MAPPO-L highly relies on the Lagrangian coefficient, rendering it more sensitive to the hyperparameters. In light of this observation, we adopt distinct hyperparameters for MAPPO-L in different categories of tasks. In contrast, the other two safe MARL algorithms are implemented with uniform hyperparameter settings across all experimental scenarios in this benchmark, indicating their potential efficacy.

Due to page limit, we present partial results of our experiments in Figures 3 and more results can be seen in Appendix G. The experimental results show that all the safety-aware algorithms are able to satisfy the safety constraints while our proposed MAFOCOPS manages to achieve the best overall performance across all tasks. Even when our method achieves similar performance to the other two algorithms in HalfCheetah scenarios, it still exhibits faster learning, demonstrating the advantages of our approach. For MAPPO-Lagrangian, we note that this baseline algorithm always achieves a similar performance as MAPPO, except in HalfCheetah scenarios where the cost threshold is significantly smaller compared to cost achieved by HAPPO and MAPPO. This may be due to that MAPPO-Lagrangian being built upon Lagrangian multiplier combined with standard MARL algorithms, leading to a performance more similar to safety-unaware MARL algorithms. In addition, from our discussion about MAPPO-L algorithm, we can know that it maintains a rather soft safety

ManyAgent Ant 2x3

ManyAgent Ant 3x2

ManyAgent Ant 6x1

Average Episode Cost

500.0

300.0

600.0

400.0

200.0

happo

mafocops

mappolag

mappo

macpo

Average Episode Cost

500.0

300.0

600.0

400.0

200.0

Average Episode Cost

400.0

500.0

000C

600.0

200.0

Average Episode Cost

250.0

100.0

200.0

150.0

100.0

100.0

100.0

50.0

0.0

0.0

0.0

0.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

ManyAgentAnt 2x3

ManyAgent Ant 3x2

ManyAgent Ant 6x1

3000.0

3000.0

3000.0

4000.0

Average Episode Reward

2500.0

Average Episode Reward

2500.0

Average Episode Reward

2500.0

Average Episode Reward

3500.0

200.0

2000.0

2000.0

300.0

1500.0

1500.0

1500.0

2500.0

1000.0

100.0

1000.0

200.0

500.0

500.0

500.0

150.0

0.0

0.0

0.0

00004

500.0

-500.0

500.0

500.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

Ant4x2

HalfCheetah2x3

HalfCheetah3x2

Average Episode Cost

250.0

mafocops

mappo

mappolag

Average Episode Cost

200.0

250.0

Average Episode Cost

500.0

600.0

Average Episode Cost

500.0

600.0

150.0

200.0

happo

macpo

150.0

400.0

00.0

40.0

30.0

100.0

000

200.0

20.0

50.0

50.0

100.0

100.0

0.0

0.0

0.0

0.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

HalfCheetah2x3

HalfCheetah3x2

Average Episode Reward

250000

200

300.0

3500.0

4000.0

NN

Average Episode Reward

3500.0

3000

2500.0

000

4000.0

Average Episode Reward

200.0

3000.0

4000.0

200.0

4000.0

300.0

150.0

150.0

000

100.0

0000

1000.0

500.0

500.0

0.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

00

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

awareness, but the other two algorithms reaches safety via hard constraints. Therefore, MACPO and MAFOCOPS possess more promising properties, thus being the focus of our study.

Interestingly, our intuition suggests that a higher number of agents usually leads to increased complexity of the environment and hence worse performance. However, this phenomenon is not always observed. We assume that this is because, in the same type of map, the multi-agent system has to control the same robot, and the number of agents determines how the robot is partitioned. Fewer agents means that each agent has to control more parts of the robot, making the difficulty of the environment non-linearly dependent on the number of agents. Whereas, in scenarios of 6x1 ManyAgent Ant, we still note that as the number of agents increases, there exists a degradation in the performance of MACPO, which can attributed to the complexity of the computation worsening with the growth in the number of agents. As for MAFOCOPS, our first-order method shows stronger resistance to task complexity, showing the advantages of our algorithm when coping with multi-agent problems. What's more, we provide videos of the trained policies of both our algorithm and MACPO in the supplementary materials to intuitively demonstrate the benefits that our approach brings.

5.2 Performance on Safe MAIG

Apart from experiments in Safe MAMuJoCo, we also conduct experiments in the Safe MAIG benchmark. In this part of experiment, the cost thresholds are set as 25% of the cost obtained by standard MARL algorithms after running for one-tenth of the entire training process. The total training settings are the same as those in Safe MAMuJoCo, while some specific hyperparameter values are adjusted considering the great difference between these two experiment environments.

Experiment results in this benchmark are presented in Figure 4. Under this complicated environment,

ShadowHandOver

ShadowHandOver

ShadowHandReOrientation

ShadowHandReOrientation

Average Episode Cost

70.0

60.0

happo

mappo

macpo

Average Episode Reward

0

30.0

Average Episode Cost

600.0

500.0

Average Episode Reward

1000.0

800.0

50.0

mappolag

mafocops

20.0

400.0

600.0

40.0

30.0

15.0

300.0

400.0

20.0

10.0

200.0

200.0

10.0

5.0

100.0

0.0

0.0

20.0

Environment steps (million)

40.0

60.0

0'08

100.0

0.0

20.0

Environment steps (million)

40.0

60.0

80.0

100.0

0.0

10.0

Environment steps (million)

20.0

30.0

40.0

50.0

0.0

0.0

10.0

Environment steps (million)

20.0

30.0

40.0

50.0

it can be observed that the soft constraint algorithm, MAPPO-L, exhibits a delay in guaranteeing safety. Notably, in ShadowHandReOrientation task, it experiences a sudden drop in cost, which significantly impacts the reward. Conversely, although similar phenomenon occurs with MAFOCOPS, our method manages to optimize the reward in this condition and attains an acceptable performance. Although the reward of MACPO remains relatively stable, its final performance is not as favorable as our method. The different performance between MAPPO-L and the other two algorithms indicates that the property of upper bound violation is important in safe reinforcement learning, which also demonstrates the significance of our work.

5.3 Efficiency Analysis between MAFOCOPS and MACPO

As a first-order method, our MAFOCOPS not only eliminates the approximation errors when solving the optimization problems, leading to better performance, but also significantly reduces the computation cost. In this part, we evaluate the memory cost and frame per second (FPS), which is often adopted to measure the training efficiency of reinforcement learning algorithms, between MAFOCPS and MACPO. For simplicity, we adopt memory monitor tools to track memory utilization after 200000 samples and record the average FPS metric.

Due to the page limit, the results are shown in Appendix H of supplementary materials. From the results, it is evident that our algorithm obtains substantial improvement in computational efficiency and shows the ability to effectively save memory resources, especially when the number of agents grows. Consequently, we can infer that when confronted with tasks with multiple agents, second-order algorithms may be ill-suited due to their substantial computational costs. In contrast, our proposed method can successfully address such scenarios without succumbing to the computational burden.

5.4 Sensitivity Analysis

Based on the description provided in Section 4, it is apparent that the Lagrange multipliers, λ j and ν max , are crucial to the performance of our approach. In addition, analyzing the sensitivity of our algorithm to changes in the safety bound is also meaningful. Hence, we seek to investigate the sensitivity of our algorithm to these hyperparameters as well as the safety bound. In this way, we conduct some ablation studies using some scenarios of Safe MAMuJoCo and more details are presented in Appendix I.

Considering the intricacy of multi-agent environments, it is difficult to delineate the correlation between the performance of our method and the hyperparameters λ j and ν max . Nonetheless, choosing hyperparameters in proximity to the optimal values that we employed typically leads to favorable outcomes. Moreover, our approach's effectiveness is relatively insensitive to variations in these hyperparameter values. As an illustration, even setting ν max = ∞ does not significantly affect the reward achieved by our method, only resulting in an average 6.7% degradation. This finding indicates the robustness of our method, and further underscores its superiority over other approaches. What's more, from the sensitivity of our method to safety bound, we learn that although the reward performance of MAFOCOPS decreases with the stricter safety constraints, the algorithm's overall effectiveness remains unchanged across different safety levels.

6 Conclusion

In this paper, we introduce a novel method for training multi-agent systems while incorporating safety constraints. Building upon the problem formulation and sequential update scheme established in the

MACPO framework [24], our algorithm offers rigorous theoretical guarantees and relies exclusively on first-order optimization techniques, thereby ensuring simplicity in its implementation. Different from MACPO, which relies on second-order optimization to achieve parameterized policies, our work proposes a fundamentally different way to tackle the optimization problem, complemented by the comprehensive provision of derivatives. The empirical experiment results demonstrate the outstanding performance and computation efficiency of our algorithm, compared to the more intricate second-order method. To sum up, our paper offers a novel and distinct contribution to the realm of safe multi-agent reinforcement learning, as evidenced by the distinctions in methodology, proofs, and experimental results in comparison with previous work.

Because the benchmarks we adopt only return one cost for agents, the performance of our algorithm across multiple costs is not evaluated. In the future, we plan to test our approach in more environments and physical settings. Moreover, there exist a number of promising prospects for future research such as incorporating off-policy data to further enhance the training efficiency. Designing methods aimed at offline settings, which precludes interactions with environments, represents another valuable direction for study.

Acknowledgments

This work is supported by National Key R&D Program of China under Contract 2022ZD0119802 as well as National Natural Science Foundation of China under Contract 61836011. What's more, it is funded by Collective Intelligence & Collaboration Laboratory (Open Fund Project No. QXZ23014101) and by Young Elite Scientists Sponsorship Program by CAST 2022QNRC001. Besides, this work is also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.

References

L. Busoniu, R. Babuska, and B. De Schutter, 'A comprehensive survey of multiagent reinforcement learning.' IEEE Transactions on Systems, Man, and Cybernetics , vol. 38, no. 2, pp. 156-172, 2008.

K. Tuyls and G. Weiss, 'Multiagent learning: Basics, challenges, and prospects.' AI Magazine , vol. 33, no. 3, pp. 41-41, 2012.

A. O. Al-Abbasi, A. Ghosh, and V. Aggarwal, 'Deeppool: Distributed model-free algorithm for ride-sharing using deep reinforcement learning,' IEEE Transactions on Intelligent Transportation Systems , vol. 20, no. 12, pp. 4714-4727, 2019.

Y. Cao, W. Yu, W. Ren, and G. Chen, 'An overview of recent progress in the study of distributed multi-agent coordination,' IEEE Transactions on Industrial Informatics , vol. 9, no. 1, pp. 427-438, 2012.

T. Wu, P. Zhou, K. Liu, Y. Yuan, X. Wang, H. Huang, and D. O. Wu, 'Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks,' IEEE Transactions on Vehicular Technology , vol. 69, no. 8, pp. 8243-8256, 2020.

K. Zhang, Z. Yang, and T. Ba¸ sar, 'Decentralized multi-agent reinforcement learning with networked agents: recent advances,' Frontiers of Information Technology & Electronic Engineering , vol. 22, no. 6, pp. 802-814, 2021.

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al. , 'Mastering the game of go with deep neural networks and tree search,' Nature , vol. 529, no. 7587, pp. 484-489, 2016.

Y. Yang, L. Yu, Y. Bai, J. Wang, W. Zhang, Y. Wen, and Y. Yu, 'A study of ai population dynamics with million-agent reinforcement learning,' arXiv preprint arXiv:1709.04511 , 2017.

C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever,

Tang, F. Wolski, and S. Zhang, 'Dota 2 with large scale deep reinforcement learning.' arXiv preprint arXiv:1912.06680 , 2019.

P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls et al. , 'Value-decomposition networks for cooperative multiagent learning based on team reward.' in International Conference on Autonomous Agents and MultiAgent Systems , 2018, pp. 2085-2087.

T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, 'Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning.' in International Conference on Machine Learning , 2018, pp. 4295-4304.

K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi, 'Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning.' in International Conference on Machine Learning , 2019, pp. 5887-5896.

Y. Yang, J. Hao, B. Liao, K. Shao, G. Chen, W. Liu, and H. Tang, 'Qatten: A general framework for cooperative multiagent reinforcement learning.' arXiv preprint arXiv:2002.03939 , 2020.

J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang, 'Qplex: Duplex dueling multi-agent q-learning,' in International Conference on Learning Representations , 2021.

R. Lowe, Y. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, 'Multi-agent actor-critic for mixed cooperative-competitive environments,' in Advances in Neural Information Processing Systems , vol. 30, 2017, pp. 6379-6390.

J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, 'Counterfactual multi-agent policy gradients,' in AAAI Conference on Artificial Intelligence , vol. 32, no. 1, 2018.

Y. Wang, B. Han, T. Wang, H. Dong, and C. Zhang, 'Dop: Off-policy multi-agent decomposed policy gradients,' in International Conference on Learning Representations , 2021.

J. G. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang, 'Trust region policy optimisation in multi-agent reinforcement learning,' in International Conference on Learning Representations , 2022.

T. M. Moldovan and P. Abbeel, 'Safe exploration in markov decision processes,' in International Conference on Machine Learning , 2012, pp. 1451-1458.

J. Achiam, D. Held, A. Tamar, and P. Abbeel, 'Constrained policy optimization,' in International Conference on Machine Learning , 2017, pp. 22-31.

A. Ray, J. Achiam, and D. Amodei, 'Benchmarking safe exploration in deep reinforcement learning,' arXiv preprint arXiv:1910.01708 , vol. 7, no. 1, p. 2, 2019.

Y. Zhang, Q. Vuong, and K. Ross, 'First order constrained optimization in policy space,' Advances in Neural Information Processing Systems , vol. 33, pp. 15 338-15 349, 2020.

T. Xu, Y. Liang, and G. Lan, 'Crpo: A new approach for safe reinforcement learning with convergence guarantee,' in International Conference on Machine Learning , 2021, pp. 11 48011 491.

S. Gu, J. G. Kuba, Y. Chen, Y. Du, L. Yang, A. Knoll, and Y. Yang, 'Safe multi-agent reinforcement learning for multi-robot control,' Artificial Intelligence , p. 103905, 2023.

D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, 'Concrete problems in ai safety,' arXiv preprint arXiv:1606.06565 , 2016.

S. Shalev-Shwartz, S. Shammah, and A. Shashua, 'Safe, multi-agent, reinforcement learning for autonomous driving,' arXiv preprint arXiv:1610.03295 , 2016.

C. Liu, N. Geng, V. Aggarwal, T. Lan, Y. Yang, and M. Xu, 'Cmix: Deep multi-agent reinforcement learning with peak and average constraints,' in Machine Learning and Knowledge Discovery in Databases. , 2021, pp. 157-173.

S. Lu, K. Zhang, T. Chen, T. Ba¸ sar, and L. Horesh, 'Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning,' in AAAI Conference on Artificial Intelligence , vol. 35, no. 10, 2021, pp. 8767-8775.

U. Borrmann, L. Wang, A. D. Ames, and M. Egerstedt, 'Control barrier certificates for safe swarm behavior,' IFAC-PapersOnLine , vol. 48, no. 27, pp. 68-73, 2015.

Z. Qin, K. Zhang, Y. Chen, J. Chen, and C. Fan, 'Learning safe multi-agent control with decentralized neural barrier certificates,' arXiv preprint arXiv:2101.05436 , 2021.

W. Zhang, O. Bastani, and V. Kumar, 'Mamps: Safe multi-agent reinforcement learning via model predictive shielding,' arXiv preprint arXiv:1910.12639 , 2019.

J. L. Nazareth, 'Conjugate gradient method,' Wiley Interdisciplinary Reviews: Computational Statistics , vol. 1, no. 3, pp. 348-353, 2009.

B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey et al. , 'Maximum entropy inverse reinforcement learning.' in Aaai , vol. 8. Chicago, IL, USA, 2008, pp. 1433-1438.

J. Schulman, X. Chen, and P. Abbeel, 'Equivalence between policy gradients and soft q-learning,' arXiv preprint arXiv:1704.06440 , 2017.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, 'Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,' in International Conference on Machine Learning , 2018, pp. 1861-1870.

D. P. Bertsekas, Constrained optimization and Lagrange multiplier methods . Academic press, 2014.

J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, 'High-dimensional continuous control using generalized advantage estimation,' arXiv preprint arXiv:1506.02438 , 2015.

C. S. de Witt, B. Peng, P.-A. Kamienny, P. Torr, W. Böhmer, and S. Whiteson, 'Deep multiagent reinforcement learning for decentralized continuous cooperative control,' arXiv preprint arXiv:2003.06709 , vol. 19, 2020.

V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa et al. , 'Isaac gym: High performance gpu-based physics simulation for robot learning,' arXiv preprint arXiv:2108.10470 , 2021.

Y. Chen, T. Wu, S. Wang, X. Feng, J. Jiang, Z. Lu, S. McAleer, H. Dong, S.-C. Zhu, and Y. Yang, 'Towards human-level bimanual dexterous manipulation with reinforcement learning,' Advances in Neural Information Processing Systems , vol. 35, pp. 5150-5163, 2022.

C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y . Wu, 'The surprising effectiveness of ppo in cooperative multi-agent games,' Advances in Neural Information Processing Systems , vol. 35, pp. 24 611-24 624, 2022.

J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, 'Trust region policy optimization,' in International conference on machine learning , 2015, pp. 1889-1897.

I. Csiszár and J. Körner, Information theory: coding theorems for discrete memoryless systems . Cambridge University Press, 2011.

S. Boyd, S. P. Boyd, and L. Vandenberghe, Convex optimization . Cambridge university press, 2004.

A Proof of the Optimization problem

According to the problem formulation, we give a definition of the 'surrogate' cost, which aligns with what is employed in MACPO [24]:

Definition. Let π be a joint policy, and ¯ π i be some other policy of agent i .Then for any of its costs of index j ∈ 1 , · · · , m i , we define

In this way, consider π and ¯ π be joint policies, i ∈ N be an agent and j ∈ 1 , · · · , m i be an index of one of its costs. From the proof of Theorem 1 in TRPO [42], (in particular, equations (41) ∼ (45)), applying it to joint policies π and ¯ π , we can conclude that

where α = D max TV ( π , ¯ π ) = max s D TV ( π ( ·| s ) , ¯ π ( ·| s )) . According to the definition of total variance divergence, defined by D TV ( p || q ) = 1 2 ∑ i | p i -q i | , we can know that D TV ( p || q ) = D TV ( q || p ) . Using Pinsker's inequality D TV ( p || q ) 2 ≤ D KL ( p || q ) 2 [43], we can change the order of policy in the divergence computation and obtain:

It's to be noted that E s ∼ ρ π , a ∼ ¯ π [ A i j, π ( s, a i )] = E s ∼ ρ π ,a i ∼ ¯ π i [ A i j, π ( s, a i )] as the actions of other agents than i do not change the value of the variable inside of the expectation. Furthermore, D max KL ( ¯ π , π ) = max s D KL ( ¯ π ( ·| s ) , π ( ·| s )) = max s ( ∑ n l =1 D KL (¯ π l ( ·| s ) , π l ( ·| s ))) ≤ ∑ n l =1 max s D KL (¯ π l ( ·| s ) , π l ( ·| s ))) = ∑ n l =1 D max KL (¯ π l , π l ) . Setting ν i j = 2 γmax s,a i | A i j. π ( s,a i ) | (1 -γ ) 2 , we can finally obtain:

The aforementioned equation is similar to Lemma 2 in MACPO, with the only difference being the order of policies in the Kullback-Leibler (KL) divergence term. However, this variation does not impact the subsequent derivations. To this end, we can establish the ultimate optimization problem presented in our work as follows:

where ¯ D KL ( π i h θ , π i h θ k ) ≜ E s ∼ ρ π θ k [ D KL ( π i h θ k ( ·| s ) , π i h θ ( ·| s ))] .

B Proof of Theorem 1

We first demonstrate the optimization problem to be solved when finding optimization problem within nonparameterized policy space:

Proof. We initiate our analysis by demonstrating the convexity of Problem (18-20) is convex w.r.t. π i h . Here we know that π θ k and θ i 1: h -1 k +1 are given and J i h j ( π θ k ) remains constant w.r.t. π i h . In this

way, E s ∼ ρ π θ k ,a i 1: h -1 ∼ π i 1: h -1 θ k +1 ,a i h ∼ πi h [ A i h π θ k ( s, a i 1: h -1 , a i h )] and E s ∼ ρ π θ k ,a i h ∼ π i h [ A i h j, π θ k ( s, a i h )] are similar so that we only need to consider the latter one. We can divide this formula like this: E s ∼ ρ π θ k ,a i h ∼ π i h [ A i h j, π θ k ( s, a i h )] = ∑ s ρ π θ k ( s ) ∑ a i h π i h ( a i h | s ) A i h j, π θ k ( s, a i h ) , where ρ π θ k ( s ) represents state visitation frequencies. We easily know that ρ π θ k ( s ) is not affected by π i h . Similarly, for each action of agent i h , the A i h j, π θ k ( s, a i h ) is also not relative to π i h . To this end, when π θ k and θ i 1: h -1 k +1 are given, E s ∼ ρ π θ k ,a i h ∼ π i h [ A i h j, π θ k ( s, a i h )] is convex to π i h . Applying these analysis, the convexity of Equation 18 and 19 can be confirmed. Concerning constraint 20, it can be rewritten as ∑ s ρ π θ k ( s ) D KL ( π i h , π i h θ k )[ s ] ≤ δ . Notably, KL divergence is convex w.r.t. its first argument, hence constraint 20 can be represented as a linear combination of convex functions, confirming its convexity as well. As π i h θ k fulfills constraint 19 and serves as an interior point within the set defined by constraint 20, therefore Slater's constraint qualification holds and strong duality holds.

Based on above discussion, we can solve for the optimal value for the problem (18 - 20) p ∗ by solving the corresponding dual problem. We define b i h j = c i h j -J i h j ( π θ k ) , then

Therefore,

where we invoked strong duality in the second equality. According to the theory of convex optimization [44], if π i h ∗ , λ ∗ j , ν ∗ j are optimal for 22, π i h ∗ is also optimal for Problem 18-20.

Consider the inner maximization problem in 22, we can decompose this problem into separate problems, one for each s .

As E a i 1: h -1 ∼ π i 1: h -1 θ k +1 [ A i h π θ k ( s, a i 1: h -1 , a i h )] is irrelevant to π i h , we rename this term as η i h π θ k ( s, a i h ) for simplicity. This is clearly a convex optimization problem which can be solved using a simple Lagrangian argument. We can then get

where ζ is the Lagrange multiplier associated with the constraint ∑ π i h ( a | s ) = 1 . Differentiating G ( π ) w.r.t for some a :

Set 25 to 0 and similar to FOCOPS, we can know

where Z λ j ,ν j ( s ) is the partition function that ensures π i h ∗ to be a probability function, i.e., ∑ a π i h ∗ ( a | s ) = 1 . Putting this π ∗ back into equation 22, we can get

What's more, we give a simple description to show that for feasible policy π θ k , the optimal policy update π i h ∗ has an upper bound for worst-case guarantee for cost constraint satisfaction. For agent i h , according to Equation 14, after getting the optimal joint update policy for all agents, J i j ( π ∗ ) ≤ J i j ( π θ k ) + L i j, π θ k ( π i h ∗ ) + ν i h j ∑ n l =1 D max KL ( π l ∗ , π l θ k ) can be obtained. According to the definition of L i j, π (¯ π i ) and the constraint 16 in the optimization problem, we can know that J i j ( π θ k ) + L i j, π θ k ( π i h ∗ ) ≤ c i h j , thus leading to J i j ( π ∗ ) ≤ c i h j + ν i h j ∑ n l =1 D max KL ( π l ∗ , π l θ k ) . In addition, we can know that the kl divergence between update policy and π θ k for each agent l has an upper bound, which we call δ l . To this end, we achieve J i j ( π ∗ ) ≤ c i h j + 2 γmax s,a i | A i j. π ( s,a i ) | (1 -γ ) 2 ∑ n l =1 δ l , which is the upper bound for worst-case guarantee for cost constraint satisfaction. According to the result, we can know that with more agents, the upper bound for worst-case guarantee is higher, which means that optimization for more agents is more challenging, consistent with our intuition.

C Proof of Corollary 1

Corollary 1. The gradient of L ( θ ) takes the form

where

Proof. Using the definition of KL divergence, we note that

where H ( π i h θ )[ s ] is the entropy and H ( π i h θ , π i h ∗ )[ s ] is the cross-entropy. We expand the cross-entropy term which gives us:

Then put this term back into Equation29:

In this way, take the gradient on both sides and we can get:

D Proof of Corollary 2

Corollary 2. The derivative of L ( π i h ∗ , λ j , ν j ) w.r.t ν j is

Proof. From the definition of L ( π i h ∗ , λ j , ν j ) and above discussion, we can know that

The first two terms is an affine function for ν j we focus on the expectation in the last term.

For simplicity, we record exp ( 1 λ j ( η π θ k ( s, a i h ) -ν j A i h j, π θ k ( s, a i h ))) as e ( x ) , so π i h ∗ ( a | s ) = π i h θ k ( a | s ) Z λ j ,ν j ( s ) ∗ e ( x ) . In this way,

Therefore, the derivative of the expectation in the last term of L ( π i h ∗ , λ j , ν j ) can be written as:

In addition, according to the definition of Z λ j ,ν j , we can get:

What's more,

Putting this result to above equation, we can get

To sum up, the derivative of ν j to function L ( π i h ∗ , λ j , ν j ) can be written:

where b i h j = c i h j -J i h j ( π θ k ) . In this way, we can update ν j by ν j ← proj ν j [ ν j -α ( c i h j -J i h j ( π θ k ))]

E Procedure of MAFOCOPS

In this section, we describe the procedure of our algorithm, outlined in Algorithm 1. To be noted, hyperparameters for each agent are identical throughout the algorithm.

Algorithm 1 MAFOCOPS

Require: number of agents n , number of updates K , minibatch size B , temperature { λ j } 1 ≤ j ≤ m i , initial cost constraint parameter { ν j } 1 ≤ j ≤ m i , cost constraint parameter bound v max , learning rate for cost constraint parameter α ν , trust region bound δ , cost bound b j

1: Initialize, policy networks { π i θ 0 , i ∈ N} , global value network { ϕ 0 } and cost value networks { ϕ i j, 0 } i ∈N 1 ≤ j ≤ m i , replay buffer B

2: for k = 0 , 1 , . . . do

3: Generate trajectories τ ∼ π θ k π θ k π θ k , save the data into the buffer and sample a batch of data;

4: Estimate the C-returns ˆ J C by averaging over the cost return for all episodes.

6: Draw a permutation i 1: n of agents at random.

5: Compute the advantage functions ˆ A π θ k ( s,a ) and ˆ A i j, π θ k ( s, a i ) using GAE;

7: Set M i 1 ( s, a ) = ˆ A π θ k ( s,a )

8: for agent i h = i 1 , i 2 , · · · , i n do

10: for K epochs do

11:

for each minibatch data of size

B

do

Update value networks (and cost value networks analogously) by minimizing the MSE loss ϕ = argmin ∑ T ( V ( s ) -ˆ R ) 2 , where ˆ R is the target return.

12: k +1 ϕ t =0 ϕ k t t

13: Update policy network by the derived equation of ∇ θ L ( θ ) (Eq. 11), where ˆ η π θ k ( s, a i h is estimated by M i 1: h ( s, a ) .

14: end for

17: end if

18: end for

20: end for

21: end for

F Experiment Environment Introduction

In this section, we introduce the environments that we adopt in the experiments.

)

HAT

(a)

(b)

(c)

F.1 Safe MAMuJoCo

This environment is an extension of MAMuJoCo [38], maintaining the background environment, agents, physics simulator, and reward function. However, in the Safe MAMuJoCo setting, additional obstacles such as walls or pitfalls are introduced, and the environment emits cost with the increasing risk of an agent stumbling upon an obstacle. Here, we mainly introduce the scenarios that we employ in our work and present them in Figure 5.

ManyAgent Ant task & Ant task The corridor in the environment is bounded by two walls, with a width of 9 m for ManyAgent Ant and 10 m for Ant. The environment emits the cost of 1 for an agent, if the distance between the robot and the wall is less than 1.8 m, or when the robot topples over, which can be described as

where z torso,t +1 and x torso,t +1 is the robot's torso's z-coordinate and x-coordinate at time t +1 , z rot is the robot's rotation's z-coordinate and x wall denotes the x-coordinate of the wall.

HalfCheetah task In these maps, the HalfCheetah agents move inside a corridor (which constraints their movement, but does not induce costs). Concurrently, there are pitfalls within the corridor that also move. When an agent is too close to a pitfall, specifically when the distance between an agent and a pitfall is less than 9 m, a cost of 1 will be emitted.

where the y-coordinate of the robot's torso is represented by y torso,t +1 and y obstacle denotes the y-coordinate of the moving obstacles.

F.2 Safe Multi-Agent Isaac Gym

This environment builds upon Issac Gym platform [39], renowned for its GPU-accelerated capabilities, and leverages the powerful Nvidia PhysX engine. Extending from the existing framework of DexterousHands [40], Safe MAIG requires agents to control the robot hands while optimizing both the reward and safety performance. Similarly, we also give an introduction of the specific scenarios in our experimental evaluations.

ShadowHandOver This task revolves around a dual-hand setup, with each hand occupying a fixed position. The primary objective entails the first hand, holding an object, navigating a suitable trajectory to transfer the item to the second hand while the second hand aims to acquire a successful grasp of the object. To be noted, this task incorporates safety constraints pertaining to the range of motion of one of the fingers on the first hand. Formally, the cost function can be expressed as follows:

where F a 4 ,t +1 is the first hand's fourth fingers's motion degree.

ShadowHandReOrientation Within the context of this task, both hands are equipped with two items. The fundamental objective for the agents is to execute rotational movements between these two items around each other and the safety constraints remain the same as Equation 38.

Ant 8x1

Average Episode Cost

250.0

200.0

mafocops

mappo

mappolag

happo

Average Episode Cost

20.0

250.0

Average Episode Cost

200.0

250.0

Average Episode Cost

200.0

250.0

150.0

macpo

150.0

150.0

150.0

100.0

100.0

100.0

50.0

50.0

50.0

50.0

0.0

0.0

0.0

0.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

Ant 8x1

Average Episode Reward

3000

3500.0

4000.0

Average Episode Reward

3500.0

4000.0

3000.0

Average Episode Reward

4000.0

3500.0

3000.0

Average Episode Reward

300.0

4000.0

3500.0

2500.0

2000

NN

2500.0

2000.0

2500.0

2000.0

200.0

2500.0

1500.0

1500.0

1500.0

150.0

mmmm

1000

1000.0

1000.0

1000.0

500.0

500.0

500.0

500.0

0.0

0.0

0.0

0.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

G Performance on Safe MAMuJoCo

In this section, we present the comprehensive results of experiments in Safe MAMuJoCo environment in Figure 6-9. It can be observed that our proposed MAFOCOPS consistently demonstrates superior overall performance across all tasks. Even when our method achieves similar performance compared to the other two algorithms in HalfCheetah scenarios, it still exhibits faster learning, demonstrating the advantages of our approach. As is discussed in the Experiment section, MAPPO-L algorithm always achieves the similar performance as MAPPO, except in HalfCheetah scenarios where the cost threshold is significantly smaller compared to cost achieved by HAPPO and MAPPO. This may be due to that MAPPO-Lagrangian being built upon Lagrangian multiplier combined with standard MARL algorithms, leading to a performance more similar to safety-unaware MARL algorithms. Regarding the other two hard constraint algorithm, their performance would degrade with the increasing number of agents. However, MAFOCOPS consistently outperforms MACPO, proving the effectiveness of our method. What's more, we provide additional videos of the trained policies of both our algorithm and MACPO in supplementary materials.

HalfCheetah2x3

HalfCheetah3x2

HalfCheetah 6x1

Average Episode Cost

500.0

600.0

mafocops

mappo

mappolag

Average Episode Cost

500.0

600.0

Average Episode Cost

500.0

600.0

happo

400.0

macpo

400.0

400.0

300.0

300.0

300.0

200.0

200.0

200.0

100.0

100.0

100.0

0.0

0.0

0.0

2

0.0

2.0

4.0

6.0

8.0

10.0

0.0

2.0

4.0

6.0

8.0

10.0

0.0

2.0

4.0

6.0

8.0

10.0

Environment steps (million)

Environment steps (million)

Environment steps (million)

HalfCheetah2x3

HalfCheetah3x2

HalfCheetah 6x1

Average Episode Reward

4000.0

Average Episode Reward

4000.0

Average Episode Reward

4000.0

3000.0

3000.0

300.0

338

2000.0

200.0

2000.0

100.0

1000.0

1000.0

0.0

00

0.0

0.0

2.0

4.0

6.0

8.0

10.0

0.0

2.0

4.0

6.0

8.0

10.0

0.0

2.0

4.0

6.0

8.0

10.0

Environment steps (million)

Environment steps (million)

Environment steps (million)

ManyAgent Ant2x3

ManyAgent Ant3x2

ManyAgentAnt6x1

600.0

mappolag

mafocops

mappo

500.0

0°009

Average Episode Cost

600.0

500.0

400.0

happo

macpo

400.0

400.0

300.0

300.0

300.0

200.0

200.0

200.0

100.0

0.0

100.0

0.0

100.0

0.0

0.0

20

4.0

6.0

8.0

10.0

0.0

2.0

4.0

6.0

8.0

10.0

0.0

2.0

4.0

6.0

8.0

10.0

Environment steps (million)

Environment steps (million)

Environment steps (million)

ManyAgentAnt2x3

ManyAgentAnt3x2

ManyAgent Ant 6x1

3000.0

0000

3000.0

Average Episode Reward

200.0

2500.0

Average Episode Reward

2500.0

2000.0

Average Episode Reward

2500.0

2000.0

MAOA

1500.0

1500.0

1500.0

W

1000.0

1000.0

1000.0

500.0

500.0

500.0

00

0.0

0.0

-500.0

-500.0

-500.0

0.0

20

Environment steps (million)

4.0

6.0

8.0

10.0

0.0

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

00

2.0

Environment steps (million)

4.0

6.0

8.0

10.0

ManyAgent Ant 2x4

ManyAgent Ant 8x1

Average Episode Cost

400.0

300.0

happo

mafocops

mappolag

macpo

mappo

Average Episode Cost

250.0

400.0

350.0

300.0

38/P8555

Average Episode Cost

400.0

300.0

200.0

V

200.0

200.0

W

150.0

100.0

100.0

50.0

0.0

0.0

0.0

0.0

20

4.0

6.0

8.0

10.0

0.0

20

4.0

6.0

8.0

10.0

00

2.0

4.0

6.0

8.0

10.0

Environment steps (million)

Environment steps (million)

Environment steps (million)

ManyAgentAnt2x4

ManyAgentAnt4x2

ManyAgentAnt8x1

Average Episode Reward

3000.0

4000.0

Average Episode Reward

3000.0

4000.0

Average Episode Reward

3000.0

4000.0

2000.0

2000.0

200.0

1000.0

1000.0

100.0

0.0

0.0

00

-1000.0

-1000.0

-1000.0

0.0

2.0

4.0

6.0

8.0

10.0

0.0

2.0

4.0

6.0

8.0

10.0

0.0

2.0

4.0

6.0

8.0

10.0

Environment steps (million)

Environment steps (million)

Environment steps (million)

H Efficiency Analysis

In this section, we evaluate the training efficiency, which is measured by frame per second (FPS), and the memory cost between MACPO and our MAFOCOPS. To be specific, we record the time and samples spent for each update to calculate average FPS and employ memory monitor tools to track memory utilization after 200000 samples. To ensure a fair comparison, both algorithms are executed on the same GPU device, thereby minimizing the influence of other variables. The results obtained from these evaluations are presented in Table 1 and Table 2 with a precision of two decimal places. Based on the obtained results, it is evident that an increase in the number of agents leads to a noticeable escalation in computational cost for MACPO. Whereas, our algorithm showcases substantial improvement in computational efficiency and demonstrates the ability to effectively conserve memory resources, especially in scenarios involving a larger number of agents.

I Sensitivity Analysis

We test the sensitivity of our algorithm to hyperparameters, i.e., λ j and ν max , as well as the safety bound. To be noted, because the benchmarks that we adopt only involve a single cost, we only need to set one value for λ j and ν max . In future works, we may explore the performance of our method in

environments with multiple costs. We choose several scenarios in Safe MAMuJoCo to conduct the ablation studies.

The sensitivity to the hyperparameters is evaluated across several different values for λ j and ν max while keeping all other parameters fixed. For ease of comparison, we normalized the results based on the return and cost achieved by [18], namely if our method yields a return of x and HAPPO achieves a return of y , the normalized result is reported as x y . The results report the final performance of the models after training for 10 million steps and are showcased in Table 3 and Table 4 with a precision of three decimal places. Given the complexity inherent in multi-agent environments, it is difficult to delineate the correlation between the performance of our method and the hyperparameters λ j and ν max . It can be observed that different scenarios have different sensitivity to these hyperparameters from the results. Overall, the reward achieved under different settings is relatively insensitive as even setting these parameters across a broad range only leads to an average degradation of less than 10%. On the other hand, the cost may be more sensitive to these parameters, highlighting the inherent challenges in ensuring safety guarantees in safe multi-agent reinforcement learning to some degree.

Furthermore, we select some maps to examine the sensitivity of our algorithm to the safety bound. To be mentioned, hyperparameters in this experiment remain unchanged as the main experiments. From the results depicted in Figure 10, we can see that setting the safety bound too high could lead to increased oscillations in cost performance, although it may yield better reward performance. This observation is reasonable since higher safety bounds may allow agents to explore actions with potentially higher returns but less safety. If the safety bound is low enough, the agents will take actions that are definitely safe, leading to less vibration. Whereas, from a global perspective, the effectiveness of our algorithm remains consistent across these different safety levels. In general, we

Ant2x4d

Ant4x2

175.0

bound c=25

175.0

175.0

Cost

150.0

125.0

bound c=50

Average Episode Cost

150.0

125.0

Average EpisodeCost

150.0

125.0

100.0

100.0

100.0

75.0

75.0

75.0

50.0

MMMI

50.0

50.0

25.0

25.0

25.0

0.0

0.0

0.0

0.0

20

4.0

6.0

8.0

10.0

0.0

20

4.0

6.0

8.0

10.0

0.0

20

4.0

6.0

8.0

10.0

Environment steps (million)

Environment steps (million)

Environment steps (million)

Ant4x2

4000.0

4000.0

4000.0

eward

3500.0

Reward

3500.0

Reward

3500.0

3000.0

3000.0

3000.0

Episode

2500.0

2500.0

Average EpisodeR

2500.0

2000.0

2000.0

2000.0

1500.0

1500.0

1500.0

1000.0

1000.0

1000.0

500.0

500.0

500.0

0.0

00

0.0

00

2.0

4.0

6.0

8.0

10.0

0.0

20

4.0

6.0

8.0

10.0

00

20

4.0

6.0

8.0

10.0

Environment steps (million)

Environment steps (million)

Environment steps (million)

0.0

2.0

4.0

6.0

8.0

10.0

Environment steps (million)

need to strike a balance between ensuring safety and achieving a good reward performance when applying safe RL algorithms.

J Details of Settings for Experiments

The majority of settings have been described in detail in the Experiments section; however, we provide some additional information here. As our implementation is based on the codebase provided by MACPO [24], and thus most hyperparameters remain consistent with their original values. For MAFOCOPS, the Lagrange multipliers, namely λ and ν max , we utilize are 2.2 and 1.3, respectively, which can found in Table 3 and 4. For the other two safe MARL algorithms, MACPO and MAPPOL, we modify the relevant hyperparameters to ensure their compatibility with the safety bound As mentioned in the Experiments section, for the two benchmarks, we adopt distinct hyperparameters

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a fresh approach to tackle the problem of safe Multi-Agent Reinforcement Learning (MARL) within a fully-cooperative multi-agent environment where all agents share a common reward function. The authors propose a novel algorithm known as Multi-Agent First Order Constrained Optimization in Policy Space (MAFOCOPS). This algorithm aims to maximize the expected total reward while ensuring each agent adheres to its safety constraint. The authors claim that MAFOCOPS is easy to implement and provides an approximate upper bound for worst-case constraint violation. To demonstrate its effectiveness, the paper conducts experiments on two safe MARL benchmarks, Safe MAMuJoCo and Safe MAIG. The results show that MAFOCOPS outperforms MACPO.\n\nSTRENGTHS:\n1.\tThe paper introduces a unique approach to multi-agent reinforcement learning, focusing on a fully-cooperative setting. It provides a new perspective on the problem by considering the influence of an agent's action on the total costs, even if it doesn't directly impact the costs of other agents. This approach captures the realistic multi-agent interactions in the real world, such as the disruption in traffic flow caused by a car running a red light.\n\n2.\tThe paper is well-structured and provides a clear explanation of the proposed method. It also includes a proof in the appendix, demonstrating the mathematical rigor of the work. The paper also compares its method with other algorithms, showing that it maintains a soft safety awareness, unlike other algorithms that reach safety via hard constraints.\n\n3.\tThe approach is significant as it provides a new perspective on multi-agent reinforcement learning, considering the influence of an agent's action on the total costs. The empirical results demonstrate the outstanding performance and computational efficiency of the proposed method, compared to more intricate second-order methods. The paper also suggests future work, planning to test the approach in more environments and physical settings.\n\nWEAKNESSES:\nThe paper does not evaluate the performance of the algorithm across multiple costs. The benchmarks adopted only return one cost for agents, which limits the comprehensiveness of the evaluation. This could potentially mask some weaknesses or limitations of the proposed method when dealing with multiple costs.\n\nQUESTIONS:\n1.\tThe paper acknowledges that solving the dual problems presented in the optimization process is computationally challenging when dealing with large state/action spaces. Calculating the partition function often involves evaluating a high-dimensional integral or sum, which can be computationally intensive. Moreover, the parameters λj and νj are dependent on the iteration k and need to be adjusted at every iteration to ensure the effectiveness of optimization. This adds to the complexity of the implementation. It would be beneficial if the authors could provide more details on how they plan to address these challenges in practical implementations. Are there any strategies or methods that could be used to reduce the computational complexity?\n\n2.\tThe paper mentions that the proposed method only employs first-order approximations, making it straightforward to implement. However, could the use of only first-order approximations limit the accuracy or effectiveness of the method in certain scenarios?\n\n3.\tThe paper mentions that the proposed method has an approximate upper bound for worst-case constraint violation. Could the authors elaborate more on what this upper bound is and how it was determined?\n\n4.\tThe paper discusses the use of a two-step process to solve the optimization problem. Could there be potential issues with this approach, such as the possibility of getting stuck in local optima?\n\nLIMITATIONS:\nBased on the information extracted from the paper, the authors have acknowledged some limitations of their work. They mention that due to the benchmarks they used, the performance of their algorithm across multiple costs was not evaluated. They also express their intention to test their approach in more environments and physical settings in the future.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposes a first-order method to solve the safety-constrained multi-agent policy optimization problem. The method is evaluated on two benchmarks and shows better performance over the baselines.

STRENGTHS:
- The studied problem is important and might interest the community.
-  The writing is easy to follow.

WEAKNESSES:
- My main concern is about the correctness of the derivation in the proposed method. The proof of *Throrem 1* says both *Equation (7)*, the objective and *Equation (8)*, the cost constraint, are **linear** *w.r.t.* $\pi^{i_h}$, which looks strange to me. The equations involve advantage estimation which definitely has a non-linear relationship with $\pi^{i_h}$. Why would it be linear?


- When solving the optimization problem, in *Corollary 2*, the expectation of the advantage over the optimal policy $\pi^{i_h\ast}$ is assumed to be zero. The advantage means the performance improvement of the optimal policy. How can it be reasonable to assume it to be zero?

QUESTIONS:
- Please see the above *Weakness* section.

LIMITATIONS:
- The correctness of the proposed method seems skeptical. Several important steps in the derivation are confusing or taken as granted without sufficient argument.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

comments: SUMMARY:
The paper builds over previous work (1) to construct a safe policy optimization algorithm based on first-order methods. It shows empirical improvement over some baselines.

STRENGTHS:
# quality 

The quality of this work is fairly satisfactory, as all the needed theory is introduced and a practical counterpart is proposed.

# clarity

The exposition is clear

WEAKNESSES:
# originality
The work seems not to be much original since it consists of a fair extension of a previously cited paper.

# significance

I might say that the work does not seem significant enough compared to related works. For instance, one positive fact referred to in the abstract (upper bound violation) is a direct property of using the previous algorithm with an additional constraint.

QUESTIONS:
I don't have questions

LIMITATIONS:
I didn't find any section where the limitations of the proposed method were addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method called first-order constraint optimization in multi-agent policy space (MAFOCOPS), which solves constraint optimization problems in a non-parametric policy space and then projects the updated policy back into the parametric policy space to achieve feasible strategies that meet safety constraints. Experimental results show that the proposed method has achieved remarkable performance while satisfying safe constraints on multiple safe MARL benchmarks.\n\nSTRENGTHS:\nThe paper proposes a new method called Multi-Agent First Order Constrained Optimization in Policy Space (MAFOCOPS) to address the challenge of developing safety-aware methods for multi-agent reinforcement learning (MARL). The proposed approach solves the constrained optimization problem in a non-parametric policy space and then projects the updated policy back into a parametric policy space, ensuring feasible policies that satisfy safety constraints. \n\nThe MAFOCOPS method has first-order characteristics, making it relatively easy to implement compared to more complex optimization methods. Additionally, it provides an approximate upper bound for constraint violations in worst-case scenarios. Experimental results demonstrate that the MAFOCOPS method achieves significant performance improvements in multiple safety MARL benchmarks while simultaneously satisfying safety constraints. This indicates that the proposed approach effectively balances performance and safety considerations. Unlike existing safety MARL methods that are often tailored to specific tasks or rely on specific assumptions and techniques, the MAFOCOPS method is designed to be applicable to a wide range of multi-agent systems and tasks. This generalizability enhances the practicality and versatility of the proposed approach.\n\nWEAKNESSES:\nThe performance of the method is sensitive to certain hyperparameters, such as the Lagrange multipliers and the safety bound. While the paper claims that the method is relatively insensitive to variations in these hyperparameter values, it would be helpful to provide a more detailed analysis of the sensitivity and its impact on performance. \n\nThe paper does not provide information on the implementation details of the proposed method, such as the specific algorithms used for policy optimization or the computational complexity of the method. Providing these details would help in assessing the practical feasibility and efficiency of the proposed approach.\n\nQUESTIONS:\n1. How can we select the best hyperparameter values for a new environment which hasn’t been met before, is there any search method like grid search or multi-arm bandit?\n\n2. Is it possible to introduce more baseline algorithms for comparison?\n\n3. I didn’t see a detailed explanation of whether MAFOCOPS uses Monte Carlo return or bootstrap return in the Safe-Multi-Agent-Isaac-Gym in the paper. I am curious if this factor will have a significant impact on the results. If possible, please provide experimental results using these different target returns.\n\nLIMITATIONS:\nThe penultimate section of the paper contains a discussion of the main limitations. The method itself should not have a direct negative outcome. However, note that in practical applications they could occur. Some examples and ways to address them could be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}