Multi-agent Reinforcement Learning for Networked System Control

This paper considers multi-agent reinforcement learning ( MARL ) in networked system control . Specifically , each agent learns a decentralized control policy based on local observations and messages from connected neighbors . We formulate such a networked MARL ( NMARL ) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent . Further , we propose a new differentiable communication protocol , called NeurComm , to reduce information loss and non-stationarity in NMARL . Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control , an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms , while NeurComm outperforms existing communication protocols in both learning efficiency and control performance . 1 INTRODUCTION . Reinforcement learning ( RL ) , formulated as a Markov decision process ( MDP ) , is a promising data-driven approach for learning adaptive control policies ( Sutton & Barto , 1998 ) . Recent advances in deep neural networks ( DNNs ) further enhance its learning capacity on complex tasks . Successful algorithms include deep Q-network ( DQN ) ( Mnih et al. , 2015 ) , deep deterministic policy gradient ( DDPG ) ( Lillicrap et al. , 2015 ) , and advantage actor critic ( A2C ) ( Mnih et al. , 2016 ) . However , RL is not scalable in many real-world control problems . This scalability issue is addressed in multi-agent RL ( MARL ) , where each agent learns its individual policy from only local observations . However , MARL introduces new challenges in model training and execution , due to non-stationarity and partial observability in a decentralized MDP from the viewpoint of each agent . To address these challenges , various learning methods and communication protocols are proposed to stabilize training and improve observability . This paper considers networked MARL ( NMARL ) in the context of networked system control ( NSC ) , where agents are connected via a communication network for a cooperative control objective . Each agent performs decentralized control based on its local observations and messages from connected neighbors . NSC is extensively studied and widely applied . Examples include connected vehicle control ( Jin & Orosz , 2014 ) , traffic signal control ( Chu et al. , 2019 ) , distributed sensing ( Xu et al. , 2018 ) , and networked storage operation ( Qin et al. , 2016 ) . We expect an increasing trend of NMARL based controllers in the near future , after the development of advanced communication technologies such as 5G and Internet-of-Things . Recent works studied decentralized NMARL under assumptions of global observations and local rewards ( Zhang et al. , 2018 ; Qu et al. , 2019 ) , which are reasonable in multi-agent gaming but not suitable in NSC . First , the control infrastructures are distributed in a wide region , so collecting global observations in execution increases communication delay and failure rate , and hurts the robustness . Second , online learning is not common due to safety and efficiency concerns . Rather , each model is trained offline and tested extensively before field deployment . In online execution , the model only runs forward propagation , and its performance is constantly monitored for triggering re-training . To reflect these practical constraints in NSC , we assume 1 ) each agent is connected to a limited number of neighbors and communication is restricted to its neighborhood , and 2 ) training is offline and global information is available in rollout training minibatches , despite a decentralized training process . The contributions of this paper are three-fold . First , we formulate NMARL under the aforementioned NSC assumptions as a decentralized spatiotemporal MDP , and introduce a spatial discount factor to stabilize training , especially for non-communicative algorithms . Second , we propose a new neural communication protocol , called NeurComm , to adaptively share information on both system states and agent behaviors . Third , we design and simulate realistic NMARL environments to evaluate and compare our approaches against recent MARL baselines . 1 2 RELATED WORK . MARL works can be classified into four groups based on their communication methods . The first group is non-communicative and focuses on stabilizing training with advanced value estimation methods . In MADDPG , each action-value is estimated by a centralized critic based on global observations and actions ( or inferred actions ) ( Lowe et al. , 2017 ) . COMA extends the same idea to A2C and estimates each advantage using a centralized critic and a counterfactual baseline ( Foerster et al. , 2018 ) . In Dec-HDRQN ( Omidshafiei et al. , 2017 ) and PS-TRPO ( Gupta et al. , 2017 ) , the centralized critic takes local observations , but the parameters are shared globally . In the NMARL work of Zhang et al . ( 2018 ) , the critic is fully decentralized but each takes global observations and performs consensus updates . In this paper , we empirically confirm that a spatial discount factor helps stabilize the training of non-communicative algorithms under neighborhood observation . The second group considers heuristic communication protocols or direct information sharing . Foerster et al . ( 2017 ) shows performance gains with directly-shared low dimensional policy fingerprints from other agents . Similarly , mean field MARL takes the average of neighbor policies for informed action-value estimation ( Yang et al. , 2018 ) . The major disadvantage of this group is that , unlike NeurComm , the communication is not explicitly designed for performance optimization , which may cause inefficient and redundant communications in execution . The third group proposes learnable communication protocols . In DIAL , the message is generated together with action-value estimation by each DQN agent , then it is encoded and summed with other input signals at the receiver side ( Foerster et al. , 2016 ) . CommNet is a more general communication protocol , but it calculates the mean of all messages instead of encoding them ( Sukhbaatar et al. , 2016 ) . Both works , especially CommNet , incur an information loss due to aggregation on input signals . Another collection of works focuses on communications in strategy games . In BiCNet ( Peng et al. , 2017 ) , a bi-directional RNN is used to enable flat communication among agents , while in MasterSlave ( Kong et al. , 2017 ) , two-way message passing is utilized in a hierarchical RNN architecture of master and slave agents . In contrast to existing protocols , NeurComm 1 ) encodes and concatenates signals , instead of aggregating them , to minimize information loss , and 2 ) includes policy fingerprints in communication to reduce non-stationarity . The fourth group focuses on communication attentions to selectively send messages . ATOC ( Jiang & Lu , 2018 ) learns a soft attention which allocates a communication probability to each other agent , while IC3Net ( Singh et al. , 2018 ) learns a hard binary attention which decides communicating or not . These works are especially useful when each agent has to prioritize the communication targets . NMARL is less likely the case since the communication range is restricted to small neighborhoods . 3 SPATIOTEMPORAL RL . This section formulates the NMARL problem as a decentralized spatiotemporal MDP , and introduces the spatial discount factor to reduce its learning difficulty . To simplify the notation , we assume the true system state is observable , and use “ state ” and “ observation ” interchangeably . This does not affect the validity of proposed methods in practice . To save space , all proofs are deferred to A . 1Code link : https : //github.com/cts198859/deeprl_network . 3.1 NETWORKED MARL . The networked system is represented by a graph G ( V , E ) where i ∈ V is each agent and ij ∈ E is each communication link . The corresponding MDP is characterized as ( G , { Si , Ai } i∈V , p , r ) where Si and Ai are the local state space and action space of agent i . Let S : = ×i∈VSi and A : = ×i∈VAi be the global state space and action space , MDP transitions follow a stationary probability distribution p : S × A × S → [ 0 , 1 ] , and global step rewards be denoted by r : S × A → R. In a multi-agent MDP , each agent i follows a decentralized policy πi : Si × Ai → [ 0 , 1 ] to chose its own action ai , t ∼ πi ( ·|si , t ) at time t. The MDP objective is to maximize E [ Rπ0 ] , where Rπt = ∑T τ=t γ τ−trτ is the long-term global return with discount factor γ . Here the expectation is taken over the global policy π : S×A → [ 0 , 1 ] , the initial distribution st ∼ ρ , and the transition sτ+1 ∼ p ( ·|sτ , aτ ) , regarding the step reward rτ = r ( sτ , aτ ) , ∀τ < T , and the terminal reward rT = rT ( sT ) 2 . The same system can be formulated as a centralized MDP . Defining V π ( s ) = E [ Rπt |st = s ] as the state-value function and Qπ ( s , a ) = E [ Rπt |st = s , at = a ] as the action-value function , we have E [ Rπ0 ] = ∑ s∈S ρ ( s ) V π ( s ) , V π ( s ) = ∑ a∈A π ( a|s ) Qπ ( s , a ) , and the advantage function Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) . MARL provides a scalable solution for controlling networked systems , but it introduces partial observability and non-stationarity in decentralized MDP of each agent , leading to inefficient and unstable learning performance . To see this , note si , t ∈ Si ⊆ S does not provide sufficient information for πi . Even assuming si , t = st , the transition pi ( si , t+1|si , t , ai , t ) = ∑ a−i , t∈A−i π−i ( a−i , t|st ) · p ( st+1|st , ai , t , a−i , t ) is non-stationary if the behavior policies of other agents π−i : = { πj } j∈V\ { i } are evolving over time . In this paper , we enforce practical constraints and only allow local observations and neighborhood communications , which makes MARL even more challenging . Definition 3.1 ( Networked Multi-agent MDP with Neighborhood Communication ) . In a networked cooperative multi-agent MDP ( G , { Si , Ai } i∈V , { Mij } ij∈E , p , { ri } i∈V ) with the message spaceM , the global reward is defined as r = 1|V| ∑ i∈V ri . All local rewards are shared globally , whereas the communication is limited to neighborhoods , that is , each agent i observes s̃i , t : = si , t ∪mNii , t . Here Ni : = { j ∈ V|ji ∈ E } , mNii , t : = { mji , t } j∈Ni , and each message mji , t ∈Mji is derived from all the available information at that neighbor . 3.2 SPATIOTEMPORAL RL . Definition 3.2 ( Spatiotemporal MDP ) . We assume local transitions are independent of other agents given the neighboring agents , that is , pi ( si , t+1|sVi , t , ai , t ) = ∑ aNi , t∈ANi ∏ j∈Ni πj ( aj , t|s̃j , t ) · p ( si , t+1|sVi , t , ai , t , aNi , t ) , ( 1 ) where Vi : = Ni ∪ { i } is the closed neighborhood , and p is abused to denote any stationary transition . Then from the viewpoint of each agent i , Definition 3.1 is equivalent to a decentralized spatiotemporal MDP , characterized as ( Si , Ai , { Mji } j∈Ni , pi , { ri } i∈V ) , by optimizing the discounted return Rπi , t = T∑ τ=t γτ−t ∑ j∈V αdijrj , t , ( 2 ) where 0 ≤ α ≤ 1 is the spatial discount factor , and dij is distance between agents i and j . The major assumption in Definition 3.2 is that the Markovian property holds both temporally and spatially , so that the next local state depends on the neighborhood states and policies only . This assumption is valid in most networked control systems such as traffic and wireless networks , as well as the power grid , where the impact of each agent is spread over the entire system via controlled flows , or chained local transitions . Note in NSC , each agent is connected to a limited number of neighbors ( the degree of G is low ) . So spatiotemporal MDP is decentralized during model execution , and it naturally extends properties of MDP . To reduce the learning difficulty of spatiotemporal MDP , a spatiotemporally discounted return is introduced in Eq . ( 2 ) to scale down reward signals further away ( which are more difficult to fit using local information ) . When α→ 0 , 2In infinite MDP , rT ( s ) = E [ ∑∞ t=T γ t−T rt ∣∣∣∣sT = s ] . each agent performs local greedy control ; when α → 1 , each agent performs global coordination and Rπi , t = R π t , ∀i ∈ V . Further , we have Qπi ( s , a ) = Qπi ( s , aVi ) = E [ Rπi , t|st = s , aVi , t = aVi ] , and V πi ( s , a−i ) = V π i ( s , aNi ) = ∑ ai∈Ai πi ( ai|s̃i ) Q π i ( s , aVi ) , since the immediate local reward of each agent is only affected by controls within its closed neighborhood . Now we assume each agent is A2C , with parametric models πθi ( s̃i ) and Vωi ( s̃i , aNi ) for fitting the optimal policy π∗i and value function V πi . Note if s̃i is able to provide global information through cascaded neighborhood communications , both πθi and Vωi are able to fit return R π i , t . Also , global and future information , such as Rπi , τ and aNi , τ , are always available from each rollout minibatch in offline training . In contrast , only local information s̃i , t is allowed in online execution of policy πθi . Proposition 3.1 ( Spatiotemporal RL with A2C ) . Let { πθi } i∈V and { Vωi } i∈V be the decentralized actor-critics , and { ( si , τ , mNii , τ , ai , τ , ri , τ ) } i∈V , τ∈B be the on-policy minibatch from spatiotemporal MDPs under stationary policies { πθi } i∈V . Then each actor and critic are updated by losses L ( θi ) = 1 |B| ∑ τ∈B ( − log πθi ( ai , τ |s̃i , τ ) Âπi , τ + β ∑ ai∈Ai πθi ( ai|s̃i , τ ) log πθi ( ai|s̃i , τ ) ) , ( 3 ) L ( ωi ) = 1 |B| ∑ τ∈B ( R̂πi , τ − Vωi ( s̃i , τ , aNi , τ ) ) 2 , ( 4 ) where Âπi , τ = R̂ π i , τ − vi , τ is the estimated advantage , R̂πi , τ = ∑τB−1 τ ′=τ γ τ ′−τ ( ∑ j∈V α dijrj , τ ′ ) + γτB−τvi , τB is the sampled action-value , vi , τ = Vω−i ( s̃i , τ , aNi , τ ) is the estimated state-value , and β is the coefficient of the entropy loss .

Summary: This paper is concerned with network multi-agent RL (N-MARL), where agents need to update their policy based on messages obtained only from neighboring nodes. This is done under sensible restrictions on the state transition distribution, which can be claimed to hold true in realistic networked settings. The authors argue that introducing a spatial discount factor (along a temporal one), where neighboring nodes have a small distance, stabilizes learning. Also, they provide a way of learning a networked communication protocol. Experiments are done on somewhat realistic simulations of traffic.