Graph Convolutional Reinforcement Learning

1 INTRODUCTION . Cooperation is a widespread phenomenon in nature from viruses , bacteria , and social amoebae to insect societies , social animals , and humans ( Melis & Semmann , 2010 ) . Human exceeds all other species in terms of range and scale of cooperation . The development of human cooperation is facilitated by the underlying graph of human societies ( Ohtsuki et al. , 2006 ; Apicella et al. , 2012 ) , where the mutual interplay between humans is abstracted by their relations . It is crucially important to enable agents to learn to cooperate in multi-agent environments for many applications , e.g. , autonomous driving ( Shalev-Shwartz et al. , 2016 ) , traffic light control ( Wiering , 2000 ) , smart grid control ( Yang et al. , 2018a ) , and multi-robot control ( Matignon et al. , 2012 ) . Multiagent reinforcement learning ( MARL ) facilitated by communication ( Sukhbaatar et al. , 2016 ; Peng et al. , 2017 ; Jiang & Lu , 2018 ) , mean field theory ( Yang et al. , 2018b ) , and causal influence ( Jaques et al. , 2019 ) have been exploited for multi-agent cooperation . However , communication among all agents ( Sukhbaatar et al. , 2016 ; Peng et al. , 2017 ) makes it hard to extract valuable information for cooperation , while communication with only nearby agents ( Jiang & Lu , 2018 ) may restrain the range of cooperation . MeanField ( Yang et al. , 2018b ) captures the interplay of agents by mean action , but the mean action eliminates the difference among agents and thus incurs the loss of important information that could help cooperation . Causal influence ( Jaques et al. , 2019 ) is a measure of action influence , which is the policy change of an agent in the presence of an action of another agent . However , causal influence is not directly related to the reward of the environment and thus may not encourage cooperation . Unlike existing work , we consider the underlying graph of agents , which could potentially help understand agents ’ mutual interplay and promote their cooperation as it does in human cooperation ( Ohtsuki et al. , 2006 ; Apicella et al. , 2012 ) . In this paper , we propose graph convolutional reinforcement learning , where the multi-agent environment is modeled as a graph . Each agent is a node , the encoding of local observation of agent is the feature of node , and there is an edge between a node and its each neighbor . We apply convolution to the graph of agents . By employing multi-head attention ( Vaswani et al. , 2017 ) as the convolution kernel , graph convolution is able to extract the relation representation between nodes and convolve the features from neighboring nodes just like a neuron in a convolutional neural network ( CNN ) . Latent features extracted from gradually increased receptive fields are exploited to learn cooperative policies . Moreover , the relation representation is temporally regularized to help the agent develop consistent cooperative policy . ∗Work done at Peking University . †Correspondence to Zongqing Lu < zongqing.lu @ pku.edu.cn > . Graph convolutional reinforcement learning , namely DGN , is instantiated based on deep Q network and trained end-to-end . DGN shares weights among all agents , making it easy to scale . DGN abstracts the mutual interplay between agents by relation kernels , extracts latent features by convolution , and induces consistent cooperation by temporal relation regularization . We empirically show the learning effectiveness of DGN in jungle and battle games and routing in packet switching networks . We demonstrate that DGN agents are able to develop cooperative and sophisticated strategies and DGN outperforms existing methods in a large margin . By ablation studies , we confirm the following . Graph convolution greatly enhances the cooperation of agents . Unlike other parameter-sharing methods , graph convolution allows the policy to be optimized by jointly considering the agents in the receptive field of an agent , promoting the mutual help . Relation kernels that are independent from the input order of features can effectively capture the interplay between agents and abstract relation representation to further improve cooperation . Temporal regularization , which minimizes the KL divergence of relation representations in successive timesteps , boosts the cooperation , helping the agent to form a long-term and consistent policy in the highly dynamic environment with many moving agents . 2 RELATED WORK . MARL . MADDPG ( Lowe et al. , 2017 ) and COMA ( Foerster et al. , 2018 ) are actor-critic models for the settings of local reward and shared reward , respectively . A centralized critic that takes as input the observations and actions of all agents are used in both , which makes them hard to scale . PS-TRPO ( Gupta et al. , 2017 ) solves problems that were previously considered intractable by most MARL algorithms via sharing of policy parameters that also improves multi-agent cooperation . However , the cooperation is still limited without sharing information among agents . Sharing parameters of value function among agents is considered in ( Zhang et al. , 2018 ) and convergence guarantee is provided for linear function approximation . However , the proposed algorithms and their convergence are established only in fully observable environments . Value propagation is proposed in ( Qu et al. , 2019 ) for networked MARL , which uses softmax temporal consistency to connect value and policy updates . However , this method only works on networked agents with static connectivity . CommNet ( Sukhbaatar et al. , 2016 ) and BiCNet ( Peng et al. , 2017 ) communicate the encoding of local observation among agents . ATOC ( Jiang & Lu , 2018 ) and TarMAC ( Das et al. , 2019 ) enable agents to learn when to communicate and who to send messages to , respectively , using attention mechanism . These communication models prove that communication does help for cooperation . However , full communication is costly and inefficient , while restrained communication may limit the range of cooperation . Graph Convolution and Relation . Many important real-world applications come in the form of graphs , such as social networks ( Kipf & Welling , 2017 ) , protein-interaction networks ( Duvenaud et al. , 2015 ) , and 3D point cloud ( Charles et al. , 2017 ) . Several frameworks ( Henaff et al. , 2015 ; Niepert et al. , 2016 ; Kipf & Welling , 2017 ; Velickovic et al. , 2017 ) have been architected to extract locally connected features from arbitrary graphs . A graph convolutional network ( GCN ) takes as input the feature matrix that summarizes the attributes of each node and outputs a node-level feature matrix . The function is similar to the convolution operation in CNNs , where the kernels are convolved across local regions of the input to produce feature maps . Using GCNs , interaction networks can reason the objects , relations and physics in complex systems , which has been proven difficult for CNNs . A few interaction frameworks have been proposed to predict the future states and underlying properties , such as IN ( Battaglia et al. , 2016 ) , VIN ( Watters et al. , 2017 ) , and VAIN ( Hoshen , 2017 ) . Relational reinforcement learning ( RRL ) ( Zambaldi et al. , 2018 ) embeds multi-head dotproduct attention ( Vaswani et al. , 2017 ) as relation block into neural networks to learn pairwise interaction representation of a set of entities in the agent ’ s state , helping the agent solve tasks with complex logic . Relational Forward Models ( RFM ) ( Tacchetti et al. , 2019 ) use supervised learning to predict the actions of all other agents based on global state . However , in partially observable environments , it is hard for RFM to learn to make accurate prediction with only local observation . MAGnet ( Malysheva et al. , 2018 ) learns relevance information in the form of a relevance graph , where relation weights are learned by pre-defined loss function based on heuristic rules , but relation weights in DGN are learned by directly minimizing the temporal-difference error of value function end-to-end . Agarwal et al . ( 2019 ) used attention mechanism for communication and proposed a curriculum learning for transferable cooperation . However , these two methods require the objects in the environment are explicitly labeled , which is infeasible in many real-world applications . 3 METHOD . We construct the multi-agent environment as a graph , where agents in the environment are represented by the nodes of the graph and each node i has a set of neighbors , Bi , which is determined by distance or other metrics , depending on the environment , and varies over time ( e.g. , the agents in i ’ s communication range or local observation ) . Moreover , neighboring nodes can communicate with each other . The intuition behind this is neighboring agents are more likely to interact with and affect each other . In addition , in many multi-agent environments , it may be costly and less helpful to take all other agents into consideration , because receiving a large amount of information requires high bandwidth and incurs high computational complexity , and agents can not differentiate valuable information from globally shared information ( Tan , 1993 ; Jiang & Lu , 2018 ) . As convolution can gradually increase the receptive field of an agent1 , the scope of cooperation is not restricted . Therefore , it is efficient and effective to consider only neighboring agents . Unlike the static graph considered in GCNs , the graph of multi-agent environment is dynamic and continuously changing over time as agents move or enter/leave the environment . Therefore , DGN should be able to adapt to the dynamics of the graph and learn as the multi-agent environment evolves . 3.1 GRAPH CONVOLUTION The problem is formulated as Decentralized Partially Observable Markov Decision Process ( DecPOMDP ) , where at each timestep t each agent i receives a local observation oti , which is the property of node i in the graph , takes an action ati , and gets an individual reward rti . The objective is to maximize the sum of all agents ’ expected returns . DGN consists of three types of modules : observation encoder , convolutional layer andQ network , as illustrated in Figure 1 . The local observation oti is encoded into a feature vector hti by MLP for low-dimensional input or CNN for visual input . The convolutional layer integrates the feature vectors in the local region ( including node i and its neighbors Bi ) and generates the latent feature vector h ′t i . By stacking more convolutional layers , the receptive field of an agent gradu- ally grows , where more information is gathered , and thus the scope of cooperation can also increase . That is , by one convolutional layer , node i can directly acquire the feature vectors from the encoders of nodes in one-hop ( i.e. , Bi ) . By stacking two layers , node i can get the output of the first convolutional layer of the nodes in one-hop , which contains the information from nodes in two-hop . However , regardless of how many convolutional layers are stacked , node i only communicates with its neighbors . This makes DGN practical in real-world applications , where each agent has limited communication range . In addition , details of the convolution kernel will be discussed in next subsection . As the number and position of agents vary over time , the underlying graph continuously changes , which brings difficulties to graph convolution . To address the issue , we merge all agents ’ feature vectors at time t into a feature matrix F t with size N×L in the order of index , where N is the number of agents and L is the length of feature vector . Then , we construct an adjacency matrix Cti with size ( |Bi| + 1 ) × N for agent i , where the first row is the one-hot representation of the index of node i , and the jth row , j = 2 , . . . , |Bi| + 1 , is the one-hot representation of the index of the ( j − 1 ) th neighbor . Then , we can obtain the feature vectors in the local region of node i by Cti × F t. Inspired by DenseNet ( Huang et al. , 2017 ) , for each agent , the features of all the preceding layers are concatenated and fed into the Q network , so as to assemble and reuse the observation representation 1The receptive field of an agent at a convolutional layer is its perceived agents at that layer . and features from different receptive fields , which respectively have distinctive contributions to the strategy that takes the cooperation at different scopes into consideration . During training , at each timestep , we store the tuple ( O , A , O′ , R , C ) in the replay buffer , where O = { o1 , · · · , oN } is the set of observations , A = { a1 , · · · , aN } is the set of actions , O′ = { o′1 , · · · , o′N } is the set of next observations , R = { r1 , · · · , rN } is the set of rewards , and C = { C1 , · · · , CN } is the set of adjacency matrix . Note that we drop time t in the notations for simplicity . Then , we sample a random minibatch of size S from the replay buffer and minimize the loss L ( θ ) = 1 S ∑ S 1 N N∑ i=1 ( yi −Q ( Oi , C , ai ; θ ) ) 2 , ( 1 ) where yi = ri + γmaxa′Q ( O′i , C , a ′ i ; θ ′ ) , Oi , C ⊆ O denotes the set of observations of the agents in i ’ s receptive fields determined by C , γ is the discount factor , and Q function , parameterized by θ , takes Oi , C as input and outputs Q value for agent i . The action of agent can change the graph at next timestep . Ideally , Q function should be learned on the changing graph . However , the graph may change quickly , which makes Q network difficult to converge . Thus , we keep C unchanged in two successive timesteps when computing the Q-loss in training to ease this learning difficulty . The gradients of Q-loss of all agents are accumulated to update the parameters . Then , we softly update the target network as θ′ = βθ + ( 1− β ) θ′ . Like CommNet ( Sukhbaatar et al. , 2016 ) , DGN can also be seen as a factorization of a centralized policy that outputs actions for all the agents to optimize the average expected return . The factorization is that all agents share θ and the model of each agent is connected to its neighbors , dynamically determined by the graph of agents at each timestep . More convolutional layers ( i.e. , larger receptive field ) yield a higher degree of centralization that mitigates non-stationarity . In addition , unlike other methods with parameter-sharing , e.g. , DQN , that sample experiences from individual agents , DGN samples experiences based on the graph of agents , not individual agents , and thus takes into consideration the interactions between agents . Nevertheless , the parameter-sharing of DGN does not prevent the emergence of sophisticated cooperative strategies , as we will show in the experiments . Note that during execution each agent only requires the ( latent ) features from its neighbors ( e.g. , via communication ) regardless of the number of agents , which makes DGN easily scale .

Summary: This paper proposes an algorithm allowing "cooperation" between agents in multi-agent reinforcement learning, modeling agents as nodes in a graph. Each agent having only a partial view of the environment, the proposed algorithm uses multi-head attention as a (graph) convolution kernel but otherwise remains similar to the DQN algorithm. Performance is evaluated on three tasks using the MAgent framework.