Multi-Agent Constrained Policy Optimisation 

1 INTRODUCTION . In recent years , reinforcement learning ( RL ) techniques have achieved remarkable successes on a variety of complex tasks ( Silver et al. , 2016 ; 2017 ; Vinyals et al. , 2019 ) . Powered by deep neural networks , deep RL enables learning sophisticated behaviours . On the other hand , deploying neural networks turns the optimisation procedure from policy space to parameter space ; this enables gradient-based methods to be applied ( Sutton et al. , 1999 ; Lillicrap et al. , 2015 ; Schulman et al. , 2017 ) . For policy gradient methods , at every iteration , the parameters of a policy network are updated in the direction of the gradient that maximises return . However , policies that are purely optimised for reward maximisation are rarely applicable to realworld problems . In many applications , an agent is often required not to visit certain states or take certain actions , which are thought of as “ unsafe ” either for itself or for other elements in the background ( Moldovan & Abbeel , 2012 ; Achiam et al. , 2017 ) . For instance , a robot carrying materials in a warehouse should not damage its parts while delivering an item to a shelf , nor should a selfdriving car cross on the red light while rushing towards its destination ( Shalev-Shwartz et al. , 2016 ) . To tackle these issues , Safe RL ( Moldovan & Abbeel , 2012 ; Garcıa & Fernández , 2015 ) is proposed , aiming to develop algorithms that learn policies that satisfy safety constraints . Despite the additional requirement of safety on solutions , algorithms with convergence guarantees have been proposed ( Xu et al. , 2021 ; Wei et al. , 2021 ) . Developing safe policies for multi-agent systems is a challenging task . Part of the difficulty comes from solving multi-agent reinforcement learning ( MARL ) problems itself ( Deng et al. , 2021 ) ; more importantly , tackling safety in MARL is hard because each individual agent has to not only consider its own safety constraints , which already may conflict its reward maximisation , but also consider the safety constraints of others so that their joint behaviour is guaranteed to be safe . As a result , there are very few solutions that offer effective learning algorithms for safe MARL problems . In fact , many of the existing methods focus on learning to cooperate ( Foerster et al. , 2018 ; Rashid et al. , 2018 ; 1Paper homepage including Videos and Code : https : //sites.google.com/view/macpo Yang & Wang , 2020 ) . However , they often require certain structures on the solution ; for example , Rashid et al . ( 2018 ) and Yang et al . ( 2020 ) adopt greedy maximisation on the local component of a monotonic joint value function , and Foerster et al . ( 2018 ) estimates the policy gradient based on the counterfactual value from a joint critic . Therefore , it is unclear how to directly incorporate safety constraints into these solution frameworks . Consequently , developing agents ’ collaborations towards reward maximisation under safety constraints remains an unsolved problem . The goal of this paper is to increase practicality of MARL algorithms through endowing them with safety awareness . For this purpose , we introduce a general framework to formulate safe MARL problems , and solve them through multi-agent policy optimisation methods . Our solutions leverage techniques from both constrained policy optimisation ( Achiam et al. , 2017 ) and multi-agent trust region learning ( Kuba et al. , 2021a ) . The resulting algorithm attains properties of both monotonic improvement guarantee and constraints satisfaction guarantee at every iteration during training . To execute the optimisation objectives , we introduce two practical deep MARL algorithms : MACPO and MAPPO-Lagrangian . As a side contribution , we also develop the first safe MARL benchmark within the MuJoCo environment , which include a variety of MARL baseline algorithms . We evaluate MACPO and MAPPO-Lagrangian on a series of tasks , and results clearly confirm the effectiveness of our solutions both in terms of constraints satisfaction and reward maximisation . To our best knowledge , MACPO and MAPPO-Lagrangian are the first safety-aware model-free MARL algorithms that work effectively in the challenging MuJoCo tasks with safety constraints . 2 RELATED WORK . Considering safety in the development of AI is a long-standing topic ( Amodei et al. , 2016 ) . When it comes to safe reinforcement learning ( Garcıa & Fernández , 2015 ) , a commonly used framework is Constrained Markov Decision Processes ( CMDPs ) ( Altman , 1999 ) . In a CMDP , at every step , in addition to the reward , the environment emits costs associated with certain constraints . As a result , the learning agent must try to satisfy those constraints while maximising the total reward . In general , the cost from the environment can be thought of as a measure of safety . Under the framework of CMDP , a safe policy is the one that explores the environment safely by keeping the total costs under certain thresholds . To tackle the learning problem in CMDPs , Achiam et al . ( 2017 ) introduced Constrained Policy Optimisation ( CPO ) , which updates agent ’ s policy under the trust region constraint ( Schulman et al. , 2015 ) to maximise surrogate return while obeying surrogate cost constraints . However , solving a constrained optimisation at every iteration of CPO can be cumbersome for implementation . An alternative solution is to apply primal-dual methods , giving rise to methods like TRPO-Lagrangian and PPO-Lagrangian ( Ray et al. , 2019 ) . Although these methods achieve impressive performance in terms of safety , the performance in terms of reward is poor ( Ray et al. , 2019 ) . Another class of algorithms that solves CMDPs is by Chow et al . ( 2018 ; 2019 ) ; these algorithms leverage the theoretical property of the Lyapunov functions and propose safe value iteration and policy gradient procedures . In contrast to CPO , Chow et al . ( 2018 ; 2019 ) can work with off-policy methods ; they also can be trained end-to-end with no need for line search . Safe multi-agent learning is an emerging research domain . Despite its importance ( Shalev-Shwartz et al. , 2016 ) , there are few solutions that work with MARL in a model-free setting . The majority of methods are designed for robotics learning . For example , the technique of barrier certificates ( Borrmann et al. , 2015 ; Ames et al. , 2016 ; Qin et al. , 2020 ) or model predictive shielding ( Zhang et al. , 2019 ) from control theory is used to model safety . These methods , however , are specifically derived for robotics applications ; they either are supervised learning based approaches , or require specific assumptions on the state space and environment dynamics . Moreover , due to the lack of a benchmark suite for safe MARL algorithms , the generalisation ability of those methods is unclear . The most related work to ours is Safe Dec-PG ( Lu et al. , 2021 ) where they used the primal-dual framework to find the saddle point between maximising reward and minimising cost . In particular , they proposed a decentralised policy descent-ascent method through a consensus network . However , reaching a consensus equivalently imposes an extra constraint of parameter sharing among neighbouring agents , which could yield suboptimal solutions ( Kuba et al. , 2021a ) . Furthermore , multi-agent policy gradient methods can suffer from high variance ( Kuba et al. , 2021b ) . In contrast , our methods employ trust region optimisation and do not assume any parameter sharing . HATRPO ( Kuba et al. , 2021a ) introduced the first multi-agent trust region method that enjoys theoretically-justified monotonic improvement guarantee . Its key idea is to make agents follow a sequential policy update scheme so that the expected joint advantage will always be positive , thus increasing reward . In this work , we show how to further develop this theory and derive a protocol which , in addition to the monotonic improvement , also guarantees to satisfy the safety constraint at every iteration during learning . The resulting algorithm ( Algorithm 1 ) successfully attains theoretical guarantees of both monotonic improvement in reward and satisfaction of safety constraints . 3 PROBLEM FORMULATION : CONSTRAINED MARKOV GAME . We formulate the safe MARL problem as a constrained Markov game 〈N , S , A , p , d0 , W , ' , C , c〉 . Here , N = { 1 , . . . , = } is the set of agents , S is the state space , A = ∏=8=1A8 is the product of the agents ’ action spaces , known as the joint action space , p : S ×A × S → ℝ is the probabilistic transition function , d0 is the initial state distribution , W ∈ [ 0 , 1 ) is the discount factor , ' : S×A → ℝ is the joint reward function , C = { 8 9 } 8∈N1≤ 9≤ < 8 is the set of sets of cost functions ( every agent 8 has < 8 cost functions ) of the form 8 9 : S ×A8 → ℝ , and finally the set of corresponding cost-constraining values is given by c = { 28 9 } 8∈N1≤ 9≤ < 8 . At time step C , the agents are in a state sC , and every agent 8 takes an action a8C according to its policy c 8 ( a8 |sC ) . Together with other agents ’ actions , it gives a joint action aC = ( a1C , . . . , a=C ) and the joint policy π ( a|s ) = ∏= 8=1 c 8 ( a8 |s ) . The agents receive the reward ' ( sC , aC ) , meanwhile each agent 8 pays the costs 89 ( sC , a8C ) , ∀ 9 = 1 , . . . , < 8 . The environment then transits to a new state sC+1 ∼ p ( ·|sC , aC ) . In this paper , we consider a fully-cooperative setting where all agents share the same reward function , aiming to maximise the expected total reward of ( π ) , s0∼d0 , a0 : ∞∼π , s1 : ∞∼p [ ∞∑ C=0 WC ' ( sC , aC ) ] , meanwhile trying to satisfy every agent 8 ’ s safety constraints , written as 89 ( π ) , s0∼d0 , a0 : ∞∼π , s1 : ∞∼p [ ∞∑ C=0 WC 89 ( sC , a8C ) ] ≤ 289 , ∀ 9 = 1 , . . . , < 8 . ( 1 ) We define the state-action value and the state-value functions in terms of reward as & π ( B , a ) , s1 : ∞∼p , a1 : ∞∼π [ ∞∑ C=0 WC ' ( sC , aC ) s0 = B , a0 = a ] , and +π ( B ) , a∼π [ & π ( B , a ) ] . The joint policies π that satisfy the Inequality ( 1 ) are referred to as feasible . Notably , in the above formulation , although the action a8C of agent 8 does not directly influence the costs { :9 ( sC , a : C ) } < : 9=1 of other agents : ≠ 8 , the action a8C will implicitly influence their total costs due to the dependence on the next state sC+1 2 . For the 9 th cost function of agent 8 , we define the 9 th state-action cost value function and the state cost value function as & 89 , π ( B , 08 ) , a−8∼π−8 , s1 : ∞∼p , a1 : ∞∼π [ ∞∑ C=0 WC 89 ( sC , a8C ) s0 = B , a80 = 08 ] , and , + 89 , π ( B ) , a∼π , s1 : ∞∼p , a1 : ∞∼π [ ∞∑ C=0 WC 89 ( sC , a8C ) s0 = B ] , respectively . Notably , the cost value functions & 8 9 , π and + 8 9 , π , although similar to traditional & π and +π , involve extra indices 8 and 9 ; the superscript 8 denotes an agent , and the subscript 9 denotes its 9 th cost . Throughout this work , we pay a close attention to the contribution to performance from different subsets of agents , therefore , we introduce the following notations . We denote an arbitrary subset 2We believe that this formulation realistically describes multi-agent interactions in the real-world ; an action of an agent has an instantaneous effect on the system only locally , but the rest of agents may suffer from its consequences at later stages . For example , consider a car that crosses on the red light , although other cars may not be at risk of riding into pedestrians immediately , the induced traffic may cause hazards soon later . { 81 , . . . , 8ℎ } of agents as 81 : ℎ ; we write −81 : ℎ to refer to its complement . Given the agent subset 81 : ℎ , we define the multi-agent state-action value function : & 81 : ℎπ ( B , a81 : ℎ ) , a−81 : ℎ∼π−81 : ℎ [ & π ( s , a81 : ℎ , a−81 : ℎ ) ] . On top of it , the multi-agent advantage function 3 is defined as follows , 81 : ℎπ ( B , a 91 : : , a81 : ℎ ) , & 91 : : ,81 : ℎπ ( B , a 91 : : , a81 : ℎ ) − & 91 : :π ( B , a 91 : : ) . ( 2 ) An interesting fact about the above multi-agent advantage function is that any advantage 81 : ℎπ can be written as a sum of sequentially-unfolding multi-agent advantages of individual agents , that is , Lemma 1 ( Multi-Agent Advantage Decomposition , Kuba et al . ( 2021b ) ) . For any state B ∈ S , subset of agents 81 : ℎ ⊆ N , and joint action a81 : ℎ , the following identity holds 81 : ℎπ ( B , a81 : ℎ ) = ℎ∑ 9=1 8 9 π ( B , a81 : 9−1 , 08 9 ) .

Summary: This paper studies the safe RL in the multi-agent setting. Specifically, the author leverage the theories from constrained policy optimization and multi-agent trust region learning to propose two algorithms: MACPO and MAPPO-Lagrangian. From the theoretical side, the author shows that in the idea setting, the proposed algorithm is guarantee to improve the objective function at iteration and the constraint satisfications can always be guaranteed. The author also demonstrate the effectiveness of their proposed algorithms in the new environment named "Safe MAMuJoCo".