Multi-agent Reinforcement Learning for Networked System Control

This paper considers multi-agent reinforcement learning ( MARL ) in networked system control . Specifically , each agent learns a decentralized control policy based on local observations and messages from connected neighbors . We formulate such a networked MARL ( NMARL ) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent . Further , we propose a new differentiable communication protocol , called NeurComm , to reduce information loss and non-stationarity in NMARL . Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control , an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms , while NeurComm outperforms existing communication protocols in both learning efficiency and control performance . 1 INTRODUCTION . Reinforcement learning ( RL ) , formulated as a Markov decision process ( MDP ) , is a promising data-driven approach for learning adaptive control policies ( Sutton & Barto , 1998 ) . Recent advances in deep neural networks ( DNNs ) further enhance its learning capacity on complex tasks . Successful algorithms include deep Q-network ( DQN ) ( Mnih et al. , 2015 ) , deep deterministic policy gradient ( DDPG ) ( Lillicrap et al. , 2015 ) , and advantage actor critic ( A2C ) ( Mnih et al. , 2016 ) . However , RL is not scalable in many real-world control problems . This scalability issue is addressed in multi-agent RL ( MARL ) , where each agent learns its individual policy from only local observations . However , MARL introduces new challenges in model training and execution , due to non-stationarity and partial observability in a decentralized MDP from the viewpoint of each agent . To address these challenges , various learning methods and communication protocols are proposed to stabilize training and improve observability . This paper considers networked MARL ( NMARL ) in the context of networked system control ( NSC ) , where agents are connected via a communication network for a cooperative control objective . Each agent performs decentralized control based on its local observations and messages from connected neighbors . NSC is extensively studied and widely applied . Examples include connected vehicle control ( Jin & Orosz , 2014 ) , traffic signal control ( Chu et al. , 2019 ) , distributed sensing ( Xu et al. , 2018 ) , and networked storage operation ( Qin et al. , 2016 ) . We expect an increasing trend of NMARL based controllers in the near future , after the development of advanced communication technologies such as 5G and Internet-of-Things . Recent works studied decentralized NMARL under assumptions of global observations and local rewards ( Zhang et al. , 2018 ; Qu et al. , 2019 ) , which are reasonable in multi-agent gaming but not suitable in NSC . First , the control infrastructures are distributed in a wide region , so collecting global observations in execution increases communication delay and failure rate , and hurts the robustness . Second , online learning is not common due to safety and efficiency concerns . Rather , each model is trained offline and tested extensively before field deployment . In online execution , the model only runs forward propagation , and its performance is constantly monitored for triggering re-training . To reflect these practical constraints in NSC , we assume 1 ) each agent is connected to a limited number of neighbors and communication is restricted to its neighborhood , and 2 ) training is offline and global information is available in rollout training minibatches , despite a decentralized training process . The contributions of this paper are three-fold . First , we formulate NMARL under the aforementioned NSC assumptions as a decentralized spatiotemporal MDP , and introduce a spatial discount factor to stabilize training , especially for non-communicative algorithms . Second , we propose a new neural communication protocol , called NeurComm , to adaptively share information on both system states and agent behaviors . Third , we design and simulate realistic NMARL environments to evaluate and compare our approaches against recent MARL baselines . 1 2 RELATED WORK . MARL works can be classified into four groups based on their communication methods . The first group is non-communicative and focuses on stabilizing training with advanced value estimation methods . In MADDPG , each action-value is estimated by a centralized critic based on global observations and actions ( or inferred actions ) ( Lowe et al. , 2017 ) . COMA extends the same idea to A2C and estimates each advantage using a centralized critic and a counterfactual baseline ( Foerster et al. , 2018 ) . In Dec-HDRQN ( Omidshafiei et al. , 2017 ) and PS-TRPO ( Gupta et al. , 2017 ) , the centralized critic takes local observations , but the parameters are shared globally . In the NMARL work of Zhang et al . ( 2018 ) , the critic is fully decentralized but each takes global observations and performs consensus updates . In this paper , we empirically confirm that a spatial discount factor helps stabilize the training of non-communicative algorithms under neighborhood observation . The second group considers heuristic communication protocols or direct information sharing . Foerster et al . ( 2017 ) shows performance gains with directly-shared low dimensional policy fingerprints from other agents . Similarly , mean field MARL takes the average of neighbor policies for informed action-value estimation ( Yang et al. , 2018 ) . The major disadvantage of this group is that , unlike NeurComm , the communication is not explicitly designed for performance optimization , which may cause inefficient and redundant communications in execution . The third group proposes learnable communication protocols . In DIAL , the message is generated together with action-value estimation by each DQN agent , then it is encoded and summed with other input signals at the receiver side ( Foerster et al. , 2016 ) . CommNet is a more general communication protocol , but it calculates the mean of all messages instead of encoding them ( Sukhbaatar et al. , 2016 ) . Both works , especially CommNet , incur an information loss due to aggregation on input signals . Another collection of works focuses on communications in strategy games . In BiCNet ( Peng et al. , 2017 ) , a bi-directional RNN is used to enable flat communication among agents , while in MasterSlave ( Kong et al. , 2017 ) , two-way message passing is utilized in a hierarchical RNN architecture of master and slave agents . In contrast to existing protocols , NeurComm 1 ) encodes and concatenates signals , instead of aggregating them , to minimize information loss , and 2 ) includes policy fingerprints in communication to reduce non-stationarity . The fourth group focuses on communication attentions to selectively send messages . ATOC ( Jiang & Lu , 2018 ) learns a soft attention which allocates a communication probability to each other agent , while IC3Net ( Singh et al. , 2018 ) learns a hard binary attention which decides communicating or not . These works are especially useful when each agent has to prioritize the communication targets . NMARL is less likely the case since the communication range is restricted to small neighborhoods . 3 SPATIOTEMPORAL RL . This section formulates the NMARL problem as a decentralized spatiotemporal MDP , and introduces the spatial discount factor to reduce its learning difficulty . To simplify the notation , we assume the true system state is observable , and use “ state ” and “ observation ” interchangeably . This does not affect the validity of proposed methods in practice . To save space , all proofs are deferred to A . 1Code link : https : //github.com/cts198859/deeprl_network . 3.1 NETWORKED MARL . The networked system is represented by a graph G ( V , E ) where i ∈ V is each agent and ij ∈ E is each communication link . The corresponding MDP is characterized as ( G , { Si , Ai } i∈V , p , r ) where Si and Ai are the local state space and action space of agent i . Let S : = ×i∈VSi and A : = ×i∈VAi be the global state space and action space , MDP transitions follow a stationary probability distribution p : S × A × S → [ 0 , 1 ] , and global step rewards be denoted by r : S × A → R. In a multi-agent MDP , each agent i follows a decentralized policy πi : Si × Ai → [ 0 , 1 ] to chose its own action ai , t ∼ πi ( ·|si , t ) at time t. The MDP objective is to maximize E [ Rπ0 ] , where Rπt = ∑T τ=t γ τ−trτ is the long-term global return with discount factor γ . Here the expectation is taken over the global policy π : S×A → [ 0 , 1 ] , the initial distribution st ∼ ρ , and the transition sτ+1 ∼ p ( ·|sτ , aτ ) , regarding the step reward rτ = r ( sτ , aτ ) , ∀τ < T , and the terminal reward rT = rT ( sT ) 2 . The same system can be formulated as a centralized MDP . Defining V π ( s ) = E [ Rπt |st = s ] as the state-value function and Qπ ( s , a ) = E [ Rπt |st = s , at = a ] as the action-value function , we have E [ Rπ0 ] = ∑ s∈S ρ ( s ) V π ( s ) , V π ( s ) = ∑ a∈A π ( a|s ) Qπ ( s , a ) , and the advantage function Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) . MARL provides a scalable solution for controlling networked systems , but it introduces partial observability and non-stationarity in decentralized MDP of each agent , leading to inefficient and unstable learning performance . To see this , note si , t ∈ Si ⊆ S does not provide sufficient information for πi . Even assuming si , t = st , the transition pi ( si , t+1|si , t , ai , t ) = ∑ a−i , t∈A−i π−i ( a−i , t|st ) · p ( st+1|st , ai , t , a−i , t ) is non-stationary if the behavior policies of other agents π−i : = { πj } j∈V\ { i } are evolving over time . In this paper , we enforce practical constraints and only allow local observations and neighborhood communications , which makes MARL even more challenging . Definition 3.1 ( Networked Multi-agent MDP with Neighborhood Communication ) . In a networked cooperative multi-agent MDP ( G , { Si , Ai } i∈V , { Mij } ij∈E , p , { ri } i∈V ) with the message spaceM , the global reward is defined as r = 1|V| ∑ i∈V ri . All local rewards are shared globally , whereas the communication is limited to neighborhoods , that is , each agent i observes s̃i , t : = si , t ∪mNii , t . Here Ni : = { j ∈ V|ji ∈ E } , mNii , t : = { mji , t } j∈Ni , and each message mji , t ∈Mji is derived from all the available information at that neighbor . 3.2 SPATIOTEMPORAL RL . Definition 3.2 ( Spatiotemporal MDP ) . We assume local transitions are independent of other agents given the neighboring agents , that is , pi ( si , t+1|sVi , t , ai , t ) = ∑ aNi , t∈ANi ∏ j∈Ni πj ( aj , t|s̃j , t ) · p ( si , t+1|sVi , t , ai , t , aNi , t ) , ( 1 ) where Vi : = Ni ∪ { i } is the closed neighborhood , and p is abused to denote any stationary transition . Then from the viewpoint of each agent i , Definition 3.1 is equivalent to a decentralized spatiotemporal MDP , characterized as ( Si , Ai , { Mji } j∈Ni , pi , { ri } i∈V ) , by optimizing the discounted return Rπi , t = T∑ τ=t γτ−t ∑ j∈V αdijrj , t , ( 2 ) where 0 ≤ α ≤ 1 is the spatial discount factor , and dij is distance between agents i and j . The major assumption in Definition 3.2 is that the Markovian property holds both temporally and spatially , so that the next local state depends on the neighborhood states and policies only . This assumption is valid in most networked control systems such as traffic and wireless networks , as well as the power grid , where the impact of each agent is spread over the entire system via controlled flows , or chained local transitions . Note in NSC , each agent is connected to a limited number of neighbors ( the degree of G is low ) . So spatiotemporal MDP is decentralized during model execution , and it naturally extends properties of MDP . To reduce the learning difficulty of spatiotemporal MDP , a spatiotemporally discounted return is introduced in Eq . ( 2 ) to scale down reward signals further away ( which are more difficult to fit using local information ) . When α→ 0 , 2In infinite MDP , rT ( s ) = E [ ∑∞ t=T γ t−T rt ∣∣∣∣sT = s ] . each agent performs local greedy control ; when α → 1 , each agent performs global coordination and Rπi , t = R π t , ∀i ∈ V . Further , we have Qπi ( s , a ) = Qπi ( s , aVi ) = E [ Rπi , t|st = s , aVi , t = aVi ] , and V πi ( s , a−i ) = V π i ( s , aNi ) = ∑ ai∈Ai πi ( ai|s̃i ) Q π i ( s , aVi ) , since the immediate local reward of each agent is only affected by controls within its closed neighborhood . Now we assume each agent is A2C , with parametric models πθi ( s̃i ) and Vωi ( s̃i , aNi ) for fitting the optimal policy π∗i and value function V πi . Note if s̃i is able to provide global information through cascaded neighborhood communications , both πθi and Vωi are able to fit return R π i , t . Also , global and future information , such as Rπi , τ and aNi , τ , are always available from each rollout minibatch in offline training . In contrast , only local information s̃i , t is allowed in online execution of policy πθi . Proposition 3.1 ( Spatiotemporal RL with A2C ) . Let { πθi } i∈V and { Vωi } i∈V be the decentralized actor-critics , and { ( si , τ , mNii , τ , ai , τ , ri , τ ) } i∈V , τ∈B be the on-policy minibatch from spatiotemporal MDPs under stationary policies { πθi } i∈V . Then each actor and critic are updated by losses L ( θi ) = 1 |B| ∑ τ∈B ( − log πθi ( ai , τ |s̃i , τ ) Âπi , τ + β ∑ ai∈Ai πθi ( ai|s̃i , τ ) log πθi ( ai|s̃i , τ ) ) , ( 3 ) L ( ωi ) = 1 |B| ∑ τ∈B ( R̂πi , τ − Vωi ( s̃i , τ , aNi , τ ) ) 2 , ( 4 ) where Âπi , τ = R̂ π i , τ − vi , τ is the estimated advantage , R̂πi , τ = ∑τB−1 τ ′=τ γ τ ′−τ ( ∑ j∈V α dijrj , τ ′ ) + γτB−τvi , τB is the sampled action-value , vi , τ = Vω−i ( s̃i , τ , aNi , τ ) is the estimated state-value , and β is the coefficient of the entropy loss .

Summary: The authors use decentralized MARL for networked system control. Each agent might control a traffic light (exp 1) or a car in traffic (exp 2). Some features of their approach are a spatial Markov assumption (only neighborhood matters), a spatial discount factor, and NeurComm: a general message passing scheme between agent policies. The authors compare their method with CommNet (averages messages before broadcast), DIAL (small-scale direct communication), etc.