Learning Safe Multi-agent Control with Decentralized Neural Barrier Certificates

1 INTRODUCTION . Machine learning ( ML ) has created unprecedented opportunities for achieving full autonomy . However , learning-based methods in autonomous systems ( AS ) can and do fail due to the lack of formal guarantees and limited generalization capability , which poses significant challenges for developing safety-critical AS , especially large-scale multi-agent AS , that are provably dependable . On the other side , safety certificates ( Chang et al . ( 2019 ) ; Jin et al . ( 2020 ) ; Choi et al . ( 2020 ) ) , which widely exist in control theory and formal methods , serve as proofs for the satisfaction of the desired properties of a system , under certain control policies . For example , once found , a Control Barrier Function ( CBF ) ensures that the closed-loop system always stays inside some safe set ( Wieland & Allgöwer , 2007 ; Ames et al. , 2014 ) with a CBF Quadratic Programming ( QP ) supervisory controller . However , it is extremely difficult to synthesize CBF by hand for complex dynamic systems , which stems a growing interest in learning-based CBF ( Saveriano & Lee , 2020 ; Srinivasan et al. , 2020 ; Jin et al. , 2020 ; Boffi et al. , 2020 ; Taylor et al. , 2020 ; Robey et al. , 2020 ) . However , all of these studies only concern single-agent systems . How to develop learning-based approaches for safe multi-agent control that are both provably dependable and scalable remains open . In multi-agent control , there is a constant dilemma : centralized control strategies can hardly scale to a large number of agents , while decentralized control without coordination often misses safety and performance guarantees . In this work , we propose a novel learning framework that jointly designs multi-agent control policies and safety certificate from data , which can be implemented in a decentralized fashion and scalable to an arbitrary number of agents . Specifically , we first introduce the notion of decentralized CBF as safety certificates , then propose the framework of learning decentralized CBF , with generalization error guarantees . The decentralized CBF can be seen as a 1https : //realm.mit.edu/blog/learning-safe-multi-agent-control-decentralized-neural-barrier-certificates contract among agents , which allows agents to learn a mutual agreement with each other on how to avoid collisions . Once such a controller is achieved through the joint-learning framework , it can be applied on an arbitrarily number of agents and in scenarios that are different from the training scenarios , which resolves the fundamental scalability issue in multi-agent control . We also propose several effective techniques in Section 4 to make such a learning process even more scalable and practical , which are then validated extensively in Section 5 . Experimental results are indeed promising . We study both 2D and 3D safe multi-agent control problems , each with several distinct environments and complex nonholonomic dynamics . Our jointlearning framework performs exceptionally well : our control policies trained on scenarios with 8 agents can be used on up to 1024 agents while maintaining low collision rates , which has notably pushed the boundary of learning-based safe multi-agent control . Speaking of which , 1024 is not the limit of our approach but rather due to the limited computational capability of our laptop used for the experiments . We also compare our approach with both leading learning-based methods ( Lowe et al. , 2017 ; Zhang & Bastani , 2019 ; Liu et al. , 2020 ) and traditional planning methods ( Ma et al. , 2019 ; Fan et al. , 2020 ) . Our approach outperforms all the other approaches in terms of both completing the tasks and maintaining safety . Contributions . Our main contributions are three-fold : 1 ) We propose the first framework to jointly learning safe multi-agent control policies and CBF certificates , in a decentralized fashion . 2 ) We present several techniques that make the learning framework more effective and scalable for practical multi-agent systems , including the use of quantity-permutation invariant neural network architectures in learning to handle the permutation of neighbouring agents . 3 ) We demonstrate via extensive experiments that our method significantly outperforms other leading methods , and has exceptional generalization capability to unseen scenarios and an arbitrary number of agents , even in quite complex multi-agent environments such as ground robots and drones . The video that demonstrates the outstanding performance of our method can be found in the supplementary material . Related Work . Learning-Based Safe Control via CBF . Barrier certificates ( Prajna et al. , 2007 ) and CBF ( Wieland & Allgöwer , 2007 ) is a well-known effective tool for guaranteeing the safety of nonlinear dynamic systems . However , the existing methods for constructing CBFs either rely on specific problem structures ( Chen et al. , 2017b ) or do not scale well ( Mitchell et al. , 2005 ) . Recently , there has been an increasing interest in learning-based and data-driven safe control via CBFs , which primarily consist of two categories : learning CBFs from data ( Saveriano & Lee , 2020 ; Srinivasan et al. , 2020 ; Jin et al. , 2020 ; Boffi et al. , 2020 ) , and CBF-based approach for controlling unknown systems ( Wang et al. , 2017 ; 2018 ; Cheng et al. , 2019 ; Taylor et al. , 2020 ) . Our work is more pertinent to the former and is complementary to the latter , which usually assumes that the CBF is provided . None of these learning-enabled approaches , however , has addressed the multi-agent setting . Multi-Agent Safety Certificates and Collision Avoidance . Restricted to holonomic systems , guaranteeing safety in multi-agent systems has been approached by limiting the velocities of the agents ( Van den Berg et al. , 2008 ; Alonso-Mora et al. , 2013 ) . Later , Borrmann et al . ( 2015 ) Wang et al . ( 2017 ) have proposed the framework of multi-agent CBF to generate collision-free controllers , with either perfectly known system dynamics ( Borrmann et al. , 2015 ) , or with worst-case uncertainty bounds ( Wang et al. , 2017 ) . Recently , Chen et al . ( 2020 ) has proposed a decentralized controller synthesized approach under this CBF framework , which is scalable to an arbitrary number of agents . However , in Chen et al . ( 2020 ) the CBF controller relies on online integration of the dynamics under the backup strategy , which can be computationally challenging for complex systems . Due to space limit , we omit other non-learning multi-agent control methods but acknowledge their importance . Safe Multi-Agent ( Reinforcement ) Learning ( MARL ) . Safety concerns have drawn increasing attention in MARL , especially with the applications to safety-critical multi-agent systems ( Zhang & Bastani , 2019 ; Qie et al. , 2019 ; Shalev-Shwartz et al. , 2016 ) . Under the CBF framework , Cheng et al . ( 2020 ) considered the setting with unknown system dynamics , and proposed to design robust multiagent CBFs based on the learned dynamics . This mirrors the second category mentioned above in single-agent learning-based safe control , which is perpendicular to our focus . RL approaches have also been applied for multi-agent collision avoidance ( Chen et al. , 2017a ; Lowe et al. , 2017 ; Everett et al. , 2018 ; Zhang et al. , 2018 ) . Nonetheless , no formal guarantees of safety were established in these works . One exception is Zhang & Bastani ( 2019 ) , which proposed a multi-agent model predictive shielding algorithm that provably guarantees safety for any policy learned from MARL , which differs from our multi-agent CBF-based approach . More importantly , none of these MARL- based approaches scale to a massive number of , e.g. , thousands of agents , as our approach does . The most scalable MARL platform , to the best of our knowledge , is Zheng et al . ( 2017 ) , which may handle a comparable scale of agents as ours , but with discrete state-action spaces . This is in contrast to our continuous-space models that can model practical control systems such as robots and drones . 2 PRELIMINARIES . 2.1 CONTROL BARRIER FUNCTIONS AS SAFETY CERTIFICATES . One common approach for ( single-agent ) safety certificate is via control barrier functions ( Ames et al. , 2014 ) , which can enforce the states of dynamic systems to stay in the safe set . Specifically , let S ⊂ Rn be the state space , Sd ⊂ S is the dangerous set , Ss = S\Sd is the safe set , which contains the set of initial conditions S0 ⊂ Ss . Also define the space of control actions as U ⊂ Rm . For a dynamic system ṡ ( t ) = f ( s ( t ) , u ( t ) ) , a control barrier function h : Rn 7→ R satisfies : ( ∀s ∈ S0 , h ( s ) ≥ 0 ) ∧ ( ∀s ∈ Sd , h ( s ) < 0 ) ∧ ( ∀ s ∈ { s | h ( s ) ≥ 0 } , ∇sh · f ( s , u ) + α ( h ) ≥ 0 ) , ( 1 ) where α ( · ) is a class-K function , i.e. , α ( · ) is strictly increasing and satisfies α ( 0 ) = 0 . For a control policy π : S → U and CBF h , it is proved in Ames et al . ( 2014 ) that if s ( 0 ) ∈ { s | h ( s ) ≥ 0 } and the three conditions in ( 1 ) are satisfied with u = π ( x ) , then s ( t ) ∈ { s | h ( s ) ≥ 0 } for ∀t ∈ [ 0 , ∞ ) , which means the state would never enter the dangerous set Sd under π . 2.2 SAFETY OF MULTI-AGENT DYNAMIC SYSTEMS . Consider a multi-agent system with N agents , the joint state of which at time t is denoted by s ( t ) = { s1 ( t ) , s2 ( t ) , · · · , sN ( t ) } where si ( t ) ∈ Si ⊂ Rn denotes the state of agent i at time t. The dynamics of agent i is ṡi ( t ) = fi ( si ( t ) , ui ( t ) ) where ui ( t ) ∈ Ui ⊂ Rm is the control action of agent i . The overall state space and input space are denoted as S .= N⊗ i=1 Si , U .= N ⊗ i=1 Ui . For each agent i , we define Ni ( t ) as the set of its neighborhood agents at time t. Let oi ( t ) ∈ Rn×|Ni ( t ) | be the local observation of agent i , which is the states of |Ni ( t ) | neighborhood agents . Notice that the dimension of oi ( t ) is not fixed and depends on the quantity of neighboring agents . We assume that the safety of agent i is jointly determined by si and oi . Let Oi be the set of all possible observations and Xi : = Si ×Oi be the state-observation space that contains the safe set Xi , s , dangerous set Xi , d and initial conditions Xi,0 ⊂ Xi , s . Let d : Xi → R describe the minimum distance from agent i to other agents that it observes , d ( si , oi ) < κs implies collision . Then Xi , s = { ( si , oi ) |d ( si , oi ) ≥ κs } and Xi , d = { ( si , oi ) |d ( si , oi ) < κs } . Let d̄i : S → R be the lifting of d from Xi to S , which is welldefined since there is a surjection from S to Xi . Then define Ss . = { s ∈ S|∀i = 1 , ... , N , d̄i ( s ) ≥ κs } . The safety of a multi-agent system can be formally defined as follows : Definition 1 ( Safety of Multi-Agent Systems ) . If the state-observation satisfies d ( si , oi ) ≥ κs for agent i and time t , then agent i is safe at time t. If for ∀i , agent i is safe at time t , then the multi-agent system is safe at time t , and s ∈ Ss . A main objective of this paper is to learn the control policy πi ( si ( t ) , oi ( t ) ) for ∀i such that the multi-agent system is safe . The control policy is decentralized ( i.e. , each agent has its own control policy and there does not exist a central controller to coordinate all the agents ) . In this way , our decentralized approach has the hope to scale to very a large number of agents .

Summary: This paper extends the neural barrier certificate method from reinforcement learning to  the decentralized multi-agent reinforcement learning setting. In particular,  it borrows the idea of the control barrier function, which enforces the states of the dynamic system to stay in the safe set. It is known that in the single agent system if the control barrier function h satisfies the equation (1), the agent will never enter the dangerous set under policy pi. A straightforward way to extend the control barrier function in the MARL setting is to replace the state-action pair by  the joint state-action pair  in equation (1). However it may suffer from the exponentially large state and action space. Therefore, the author proposes a decentralized setting, where each agent just maintains its own control barrier function. To learn the policy and function h simultaneously, the author penalizes the violation of equation (2). At last, they test the proposed method in Navigation, Predator-Prey, ground robots and Drones compare it with several baselines.