Multi-Agent Constrained Policy Optimisation 

1 INTRODUCTION . In recent years , reinforcement learning ( RL ) techniques have achieved remarkable successes on a variety of complex tasks ( Silver et al. , 2016 ; 2017 ; Vinyals et al. , 2019 ) . Powered by deep neural networks , deep RL enables learning sophisticated behaviours . On the other hand , deploying neural networks turns the optimisation procedure from policy space to parameter space ; this enables gradient-based methods to be applied ( Sutton et al. , 1999 ; Lillicrap et al. , 2015 ; Schulman et al. , 2017 ) . For policy gradient methods , at every iteration , the parameters of a policy network are updated in the direction of the gradient that maximises return . However , policies that are purely optimised for reward maximisation are rarely applicable to realworld problems . In many applications , an agent is often required not to visit certain states or take certain actions , which are thought of as “ unsafe ” either for itself or for other elements in the background ( Moldovan & Abbeel , 2012 ; Achiam et al. , 2017 ) . For instance , a robot carrying materials in a warehouse should not damage its parts while delivering an item to a shelf , nor should a selfdriving car cross on the red light while rushing towards its destination ( Shalev-Shwartz et al. , 2016 ) . To tackle these issues , Safe RL ( Moldovan & Abbeel , 2012 ; Garcıa & Fernández , 2015 ) is proposed , aiming to develop algorithms that learn policies that satisfy safety constraints . Despite the additional requirement of safety on solutions , algorithms with convergence guarantees have been proposed ( Xu et al. , 2021 ; Wei et al. , 2021 ) . Developing safe policies for multi-agent systems is a challenging task . Part of the difficulty comes from solving multi-agent reinforcement learning ( MARL ) problems itself ( Deng et al. , 2021 ) ; more importantly , tackling safety in MARL is hard because each individual agent has to not only consider its own safety constraints , which already may conflict its reward maximisation , but also consider the safety constraints of others so that their joint behaviour is guaranteed to be safe . As a result , there are very few solutions that offer effective learning algorithms for safe MARL problems . In fact , many of the existing methods focus on learning to cooperate ( Foerster et al. , 2018 ; Rashid et al. , 2018 ; 1Paper homepage including Videos and Code : https : //sites.google.com/view/macpo Yang & Wang , 2020 ) . However , they often require certain structures on the solution ; for example , Rashid et al . ( 2018 ) and Yang et al . ( 2020 ) adopt greedy maximisation on the local component of a monotonic joint value function , and Foerster et al . ( 2018 ) estimates the policy gradient based on the counterfactual value from a joint critic . Therefore , it is unclear how to directly incorporate safety constraints into these solution frameworks . Consequently , developing agents ’ collaborations towards reward maximisation under safety constraints remains an unsolved problem . The goal of this paper is to increase practicality of MARL algorithms through endowing them with safety awareness . For this purpose , we introduce a general framework to formulate safe MARL problems , and solve them through multi-agent policy optimisation methods . Our solutions leverage techniques from both constrained policy optimisation ( Achiam et al. , 2017 ) and multi-agent trust region learning ( Kuba et al. , 2021a ) . The resulting algorithm attains properties of both monotonic improvement guarantee and constraints satisfaction guarantee at every iteration during training . To execute the optimisation objectives , we introduce two practical deep MARL algorithms : MACPO and MAPPO-Lagrangian . As a side contribution , we also develop the first safe MARL benchmark within the MuJoCo environment , which include a variety of MARL baseline algorithms . We evaluate MACPO and MAPPO-Lagrangian on a series of tasks , and results clearly confirm the effectiveness of our solutions both in terms of constraints satisfaction and reward maximisation . To our best knowledge , MACPO and MAPPO-Lagrangian are the first safety-aware model-free MARL algorithms that work effectively in the challenging MuJoCo tasks with safety constraints . 2 RELATED WORK . Considering safety in the development of AI is a long-standing topic ( Amodei et al. , 2016 ) . When it comes to safe reinforcement learning ( Garcıa & Fernández , 2015 ) , a commonly used framework is Constrained Markov Decision Processes ( CMDPs ) ( Altman , 1999 ) . In a CMDP , at every step , in addition to the reward , the environment emits costs associated with certain constraints . As a result , the learning agent must try to satisfy those constraints while maximising the total reward . In general , the cost from the environment can be thought of as a measure of safety . Under the framework of CMDP , a safe policy is the one that explores the environment safely by keeping the total costs under certain thresholds . To tackle the learning problem in CMDPs , Achiam et al . ( 2017 ) introduced Constrained Policy Optimisation ( CPO ) , which updates agent ’ s policy under the trust region constraint ( Schulman et al. , 2015 ) to maximise surrogate return while obeying surrogate cost constraints . However , solving a constrained optimisation at every iteration of CPO can be cumbersome for implementation . An alternative solution is to apply primal-dual methods , giving rise to methods like TRPO-Lagrangian and PPO-Lagrangian ( Ray et al. , 2019 ) . Although these methods achieve impressive performance in terms of safety , the performance in terms of reward is poor ( Ray et al. , 2019 ) . Another class of algorithms that solves CMDPs is by Chow et al . ( 2018 ; 2019 ) ; these algorithms leverage the theoretical property of the Lyapunov functions and propose safe value iteration and policy gradient procedures . In contrast to CPO , Chow et al . ( 2018 ; 2019 ) can work with off-policy methods ; they also can be trained end-to-end with no need for line search . Safe multi-agent learning is an emerging research domain . Despite its importance ( Shalev-Shwartz et al. , 2016 ) , there are few solutions that work with MARL in a model-free setting . The majority of methods are designed for robotics learning . For example , the technique of barrier certificates ( Borrmann et al. , 2015 ; Ames et al. , 2016 ; Qin et al. , 2020 ) or model predictive shielding ( Zhang et al. , 2019 ) from control theory is used to model safety . These methods , however , are specifically derived for robotics applications ; they either are supervised learning based approaches , or require specific assumptions on the state space and environment dynamics . Moreover , due to the lack of a benchmark suite for safe MARL algorithms , the generalisation ability of those methods is unclear . The most related work to ours is Safe Dec-PG ( Lu et al. , 2021 ) where they used the primal-dual framework to find the saddle point between maximising reward and minimising cost . In particular , they proposed a decentralised policy descent-ascent method through a consensus network . However , reaching a consensus equivalently imposes an extra constraint of parameter sharing among neighbouring agents , which could yield suboptimal solutions ( Kuba et al. , 2021a ) . Furthermore , multi-agent policy gradient methods can suffer from high variance ( Kuba et al. , 2021b ) . In contrast , our methods employ trust region optimisation and do not assume any parameter sharing . HATRPO ( Kuba et al. , 2021a ) introduced the first multi-agent trust region method that enjoys theoretically-justified monotonic improvement guarantee . Its key idea is to make agents follow a sequential policy update scheme so that the expected joint advantage will always be positive , thus increasing reward . In this work , we show how to further develop this theory and derive a protocol which , in addition to the monotonic improvement , also guarantees to satisfy the safety constraint at every iteration during learning . The resulting algorithm ( Algorithm 1 ) successfully attains theoretical guarantees of both monotonic improvement in reward and satisfaction of safety constraints . 3 PROBLEM FORMULATION : CONSTRAINED MARKOV GAME . We formulate the safe MARL problem as a constrained Markov game 〈N , S , A , p , d0 , W , ' , C , c〉 . Here , N = { 1 , . . . , = } is the set of agents , S is the state space , A = ∏=8=1A8 is the product of the agents ’ action spaces , known as the joint action space , p : S ×A × S → ℝ is the probabilistic transition function , d0 is the initial state distribution , W ∈ [ 0 , 1 ) is the discount factor , ' : S×A → ℝ is the joint reward function , C = { 8 9 } 8∈N1≤ 9≤ < 8 is the set of sets of cost functions ( every agent 8 has < 8 cost functions ) of the form 8 9 : S ×A8 → ℝ , and finally the set of corresponding cost-constraining values is given by c = { 28 9 } 8∈N1≤ 9≤ < 8 . At time step C , the agents are in a state sC , and every agent 8 takes an action a8C according to its policy c 8 ( a8 |sC ) . Together with other agents ’ actions , it gives a joint action aC = ( a1C , . . . , a=C ) and the joint policy π ( a|s ) = ∏= 8=1 c 8 ( a8 |s ) . The agents receive the reward ' ( sC , aC ) , meanwhile each agent 8 pays the costs 89 ( sC , a8C ) , ∀ 9 = 1 , . . . , < 8 . The environment then transits to a new state sC+1 ∼ p ( ·|sC , aC ) . In this paper , we consider a fully-cooperative setting where all agents share the same reward function , aiming to maximise the expected total reward of ( π ) , s0∼d0 , a0 : ∞∼π , s1 : ∞∼p [ ∞∑ C=0 WC ' ( sC , aC ) ] , meanwhile trying to satisfy every agent 8 ’ s safety constraints , written as 89 ( π ) , s0∼d0 , a0 : ∞∼π , s1 : ∞∼p [ ∞∑ C=0 WC 89 ( sC , a8C ) ] ≤ 289 , ∀ 9 = 1 , . . . , < 8 . ( 1 ) We define the state-action value and the state-value functions in terms of reward as & π ( B , a ) , s1 : ∞∼p , a1 : ∞∼π [ ∞∑ C=0 WC ' ( sC , aC ) s0 = B , a0 = a ] , and +π ( B ) , a∼π [ & π ( B , a ) ] . The joint policies π that satisfy the Inequality ( 1 ) are referred to as feasible . Notably , in the above formulation , although the action a8C of agent 8 does not directly influence the costs { :9 ( sC , a : C ) } < : 9=1 of other agents : ≠ 8 , the action a8C will implicitly influence their total costs due to the dependence on the next state sC+1 2 . For the 9 th cost function of agent 8 , we define the 9 th state-action cost value function and the state cost value function as & 89 , π ( B , 08 ) , a−8∼π−8 , s1 : ∞∼p , a1 : ∞∼π [ ∞∑ C=0 WC 89 ( sC , a8C ) s0 = B , a80 = 08 ] , and , + 89 , π ( B ) , a∼π , s1 : ∞∼p , a1 : ∞∼π [ ∞∑ C=0 WC 89 ( sC , a8C ) s0 = B ] , respectively . Notably , the cost value functions & 8 9 , π and + 8 9 , π , although similar to traditional & π and +π , involve extra indices 8 and 9 ; the superscript 8 denotes an agent , and the subscript 9 denotes its 9 th cost . Throughout this work , we pay a close attention to the contribution to performance from different subsets of agents , therefore , we introduce the following notations . We denote an arbitrary subset 2We believe that this formulation realistically describes multi-agent interactions in the real-world ; an action of an agent has an instantaneous effect on the system only locally , but the rest of agents may suffer from its consequences at later stages . For example , consider a car that crosses on the red light , although other cars may not be at risk of riding into pedestrians immediately , the induced traffic may cause hazards soon later . { 81 , . . . , 8ℎ } of agents as 81 : ℎ ; we write −81 : ℎ to refer to its complement . Given the agent subset 81 : ℎ , we define the multi-agent state-action value function : & 81 : ℎπ ( B , a81 : ℎ ) , a−81 : ℎ∼π−81 : ℎ [ & π ( s , a81 : ℎ , a−81 : ℎ ) ] . On top of it , the multi-agent advantage function 3 is defined as follows , 81 : ℎπ ( B , a 91 : : , a81 : ℎ ) , & 91 : : ,81 : ℎπ ( B , a 91 : : , a81 : ℎ ) − & 91 : :π ( B , a 91 : : ) . ( 2 ) An interesting fact about the above multi-agent advantage function is that any advantage 81 : ℎπ can be written as a sum of sequentially-unfolding multi-agent advantages of individual agents , that is , Lemma 1 ( Multi-Agent Advantage Decomposition , Kuba et al . ( 2021b ) ) . For any state B ∈ S , subset of agents 81 : ℎ ⊆ N , and joint action a81 : ℎ , the following identity holds 81 : ℎπ ( B , a81 : ℎ ) = ℎ∑ 9=1 8 9 π ( B , a81 : 9−1 , 08 9 ) .

Summary: This paper considers the multi-agent reinforcement learning (MARL) problem with safety constraints. The authors proposed two methods, Multi-Agent Constrained Policy Optimisation (MACPO) and MAPPO-Lagrangian, by leveraging the theories from both constrained policy optimization and multi-agent trust-region learning. Their method is shown valid both theoretically and empirically. My main concerns lie in their novelty compared with existing literature and the problem set.