Graph Convolutional Reinforcement Learning

1 INTRODUCTION . Cooperation is a widespread phenomenon in nature from viruses , bacteria , and social amoebae to insect societies , social animals , and humans ( Melis & Semmann , 2010 ) . Human exceeds all other species in terms of range and scale of cooperation . The development of human cooperation is facilitated by the underlying graph of human societies ( Ohtsuki et al. , 2006 ; Apicella et al. , 2012 ) , where the mutual interplay between humans is abstracted by their relations . It is crucially important to enable agents to learn to cooperate in multi-agent environments for many applications , e.g. , autonomous driving ( Shalev-Shwartz et al. , 2016 ) , traffic light control ( Wiering , 2000 ) , smart grid control ( Yang et al. , 2018a ) , and multi-robot control ( Matignon et al. , 2012 ) . Multiagent reinforcement learning ( MARL ) facilitated by communication ( Sukhbaatar et al. , 2016 ; Peng et al. , 2017 ; Jiang & Lu , 2018 ) , mean field theory ( Yang et al. , 2018b ) , and causal influence ( Jaques et al. , 2019 ) have been exploited for multi-agent cooperation . However , communication among all agents ( Sukhbaatar et al. , 2016 ; Peng et al. , 2017 ) makes it hard to extract valuable information for cooperation , while communication with only nearby agents ( Jiang & Lu , 2018 ) may restrain the range of cooperation . MeanField ( Yang et al. , 2018b ) captures the interplay of agents by mean action , but the mean action eliminates the difference among agents and thus incurs the loss of important information that could help cooperation . Causal influence ( Jaques et al. , 2019 ) is a measure of action influence , which is the policy change of an agent in the presence of an action of another agent . However , causal influence is not directly related to the reward of the environment and thus may not encourage cooperation . Unlike existing work , we consider the underlying graph of agents , which could potentially help understand agents ’ mutual interplay and promote their cooperation as it does in human cooperation ( Ohtsuki et al. , 2006 ; Apicella et al. , 2012 ) . In this paper , we propose graph convolutional reinforcement learning , where the multi-agent environment is modeled as a graph . Each agent is a node , the encoding of local observation of agent is the feature of node , and there is an edge between a node and its each neighbor . We apply convolution to the graph of agents . By employing multi-head attention ( Vaswani et al. , 2017 ) as the convolution kernel , graph convolution is able to extract the relation representation between nodes and convolve the features from neighboring nodes just like a neuron in a convolutional neural network ( CNN ) . Latent features extracted from gradually increased receptive fields are exploited to learn cooperative policies . Moreover , the relation representation is temporally regularized to help the agent develop consistent cooperative policy . ∗Work done at Peking University . †Correspondence to Zongqing Lu < zongqing.lu @ pku.edu.cn > . Graph convolutional reinforcement learning , namely DGN , is instantiated based on deep Q network and trained end-to-end . DGN shares weights among all agents , making it easy to scale . DGN abstracts the mutual interplay between agents by relation kernels , extracts latent features by convolution , and induces consistent cooperation by temporal relation regularization . We empirically show the learning effectiveness of DGN in jungle and battle games and routing in packet switching networks . We demonstrate that DGN agents are able to develop cooperative and sophisticated strategies and DGN outperforms existing methods in a large margin . By ablation studies , we confirm the following . Graph convolution greatly enhances the cooperation of agents . Unlike other parameter-sharing methods , graph convolution allows the policy to be optimized by jointly considering the agents in the receptive field of an agent , promoting the mutual help . Relation kernels that are independent from the input order of features can effectively capture the interplay between agents and abstract relation representation to further improve cooperation . Temporal regularization , which minimizes the KL divergence of relation representations in successive timesteps , boosts the cooperation , helping the agent to form a long-term and consistent policy in the highly dynamic environment with many moving agents . 2 RELATED WORK . MARL . MADDPG ( Lowe et al. , 2017 ) and COMA ( Foerster et al. , 2018 ) are actor-critic models for the settings of local reward and shared reward , respectively . A centralized critic that takes as input the observations and actions of all agents are used in both , which makes them hard to scale . PS-TRPO ( Gupta et al. , 2017 ) solves problems that were previously considered intractable by most MARL algorithms via sharing of policy parameters that also improves multi-agent cooperation . However , the cooperation is still limited without sharing information among agents . Sharing parameters of value function among agents is considered in ( Zhang et al. , 2018 ) and convergence guarantee is provided for linear function approximation . However , the proposed algorithms and their convergence are established only in fully observable environments . Value propagation is proposed in ( Qu et al. , 2019 ) for networked MARL , which uses softmax temporal consistency to connect value and policy updates . However , this method only works on networked agents with static connectivity . CommNet ( Sukhbaatar et al. , 2016 ) and BiCNet ( Peng et al. , 2017 ) communicate the encoding of local observation among agents . ATOC ( Jiang & Lu , 2018 ) and TarMAC ( Das et al. , 2019 ) enable agents to learn when to communicate and who to send messages to , respectively , using attention mechanism . These communication models prove that communication does help for cooperation . However , full communication is costly and inefficient , while restrained communication may limit the range of cooperation . Graph Convolution and Relation . Many important real-world applications come in the form of graphs , such as social networks ( Kipf & Welling , 2017 ) , protein-interaction networks ( Duvenaud et al. , 2015 ) , and 3D point cloud ( Charles et al. , 2017 ) . Several frameworks ( Henaff et al. , 2015 ; Niepert et al. , 2016 ; Kipf & Welling , 2017 ; Velickovic et al. , 2017 ) have been architected to extract locally connected features from arbitrary graphs . A graph convolutional network ( GCN ) takes as input the feature matrix that summarizes the attributes of each node and outputs a node-level feature matrix . The function is similar to the convolution operation in CNNs , where the kernels are convolved across local regions of the input to produce feature maps . Using GCNs , interaction networks can reason the objects , relations and physics in complex systems , which has been proven difficult for CNNs . A few interaction frameworks have been proposed to predict the future states and underlying properties , such as IN ( Battaglia et al. , 2016 ) , VIN ( Watters et al. , 2017 ) , and VAIN ( Hoshen , 2017 ) . Relational reinforcement learning ( RRL ) ( Zambaldi et al. , 2018 ) embeds multi-head dotproduct attention ( Vaswani et al. , 2017 ) as relation block into neural networks to learn pairwise interaction representation of a set of entities in the agent ’ s state , helping the agent solve tasks with complex logic . Relational Forward Models ( RFM ) ( Tacchetti et al. , 2019 ) use supervised learning to predict the actions of all other agents based on global state . However , in partially observable environments , it is hard for RFM to learn to make accurate prediction with only local observation . MAGnet ( Malysheva et al. , 2018 ) learns relevance information in the form of a relevance graph , where relation weights are learned by pre-defined loss function based on heuristic rules , but relation weights in DGN are learned by directly minimizing the temporal-difference error of value function end-to-end . Agarwal et al . ( 2019 ) used attention mechanism for communication and proposed a curriculum learning for transferable cooperation . However , these two methods require the objects in the environment are explicitly labeled , which is infeasible in many real-world applications . 3 METHOD . We construct the multi-agent environment as a graph , where agents in the environment are represented by the nodes of the graph and each node i has a set of neighbors , Bi , which is determined by distance or other metrics , depending on the environment , and varies over time ( e.g. , the agents in i ’ s communication range or local observation ) . Moreover , neighboring nodes can communicate with each other . The intuition behind this is neighboring agents are more likely to interact with and affect each other . In addition , in many multi-agent environments , it may be costly and less helpful to take all other agents into consideration , because receiving a large amount of information requires high bandwidth and incurs high computational complexity , and agents can not differentiate valuable information from globally shared information ( Tan , 1993 ; Jiang & Lu , 2018 ) . As convolution can gradually increase the receptive field of an agent1 , the scope of cooperation is not restricted . Therefore , it is efficient and effective to consider only neighboring agents . Unlike the static graph considered in GCNs , the graph of multi-agent environment is dynamic and continuously changing over time as agents move or enter/leave the environment . Therefore , DGN should be able to adapt to the dynamics of the graph and learn as the multi-agent environment evolves . 3.1 GRAPH CONVOLUTION The problem is formulated as Decentralized Partially Observable Markov Decision Process ( DecPOMDP ) , where at each timestep t each agent i receives a local observation oti , which is the property of node i in the graph , takes an action ati , and gets an individual reward rti . The objective is to maximize the sum of all agents ’ expected returns . DGN consists of three types of modules : observation encoder , convolutional layer andQ network , as illustrated in Figure 1 . The local observation oti is encoded into a feature vector hti by MLP for low-dimensional input or CNN for visual input . The convolutional layer integrates the feature vectors in the local region ( including node i and its neighbors Bi ) and generates the latent feature vector h ′t i . By stacking more convolutional layers , the receptive field of an agent gradu- ally grows , where more information is gathered , and thus the scope of cooperation can also increase . That is , by one convolutional layer , node i can directly acquire the feature vectors from the encoders of nodes in one-hop ( i.e. , Bi ) . By stacking two layers , node i can get the output of the first convolutional layer of the nodes in one-hop , which contains the information from nodes in two-hop . However , regardless of how many convolutional layers are stacked , node i only communicates with its neighbors . This makes DGN practical in real-world applications , where each agent has limited communication range . In addition , details of the convolution kernel will be discussed in next subsection . As the number and position of agents vary over time , the underlying graph continuously changes , which brings difficulties to graph convolution . To address the issue , we merge all agents ’ feature vectors at time t into a feature matrix F t with size N×L in the order of index , where N is the number of agents and L is the length of feature vector . Then , we construct an adjacency matrix Cti with size ( |Bi| + 1 ) × N for agent i , where the first row is the one-hot representation of the index of node i , and the jth row , j = 2 , . . . , |Bi| + 1 , is the one-hot representation of the index of the ( j − 1 ) th neighbor . Then , we can obtain the feature vectors in the local region of node i by Cti × F t. Inspired by DenseNet ( Huang et al. , 2017 ) , for each agent , the features of all the preceding layers are concatenated and fed into the Q network , so as to assemble and reuse the observation representation 1The receptive field of an agent at a convolutional layer is its perceived agents at that layer . and features from different receptive fields , which respectively have distinctive contributions to the strategy that takes the cooperation at different scopes into consideration . During training , at each timestep , we store the tuple ( O , A , O′ , R , C ) in the replay buffer , where O = { o1 , · · · , oN } is the set of observations , A = { a1 , · · · , aN } is the set of actions , O′ = { o′1 , · · · , o′N } is the set of next observations , R = { r1 , · · · , rN } is the set of rewards , and C = { C1 , · · · , CN } is the set of adjacency matrix . Note that we drop time t in the notations for simplicity . Then , we sample a random minibatch of size S from the replay buffer and minimize the loss L ( θ ) = 1 S ∑ S 1 N N∑ i=1 ( yi −Q ( Oi , C , ai ; θ ) ) 2 , ( 1 ) where yi = ri + γmaxa′Q ( O′i , C , a ′ i ; θ ′ ) , Oi , C ⊆ O denotes the set of observations of the agents in i ’ s receptive fields determined by C , γ is the discount factor , and Q function , parameterized by θ , takes Oi , C as input and outputs Q value for agent i . The action of agent can change the graph at next timestep . Ideally , Q function should be learned on the changing graph . However , the graph may change quickly , which makes Q network difficult to converge . Thus , we keep C unchanged in two successive timesteps when computing the Q-loss in training to ease this learning difficulty . The gradients of Q-loss of all agents are accumulated to update the parameters . Then , we softly update the target network as θ′ = βθ + ( 1− β ) θ′ . Like CommNet ( Sukhbaatar et al. , 2016 ) , DGN can also be seen as a factorization of a centralized policy that outputs actions for all the agents to optimize the average expected return . The factorization is that all agents share θ and the model of each agent is connected to its neighbors , dynamically determined by the graph of agents at each timestep . More convolutional layers ( i.e. , larger receptive field ) yield a higher degree of centralization that mitigates non-stationarity . In addition , unlike other methods with parameter-sharing , e.g. , DQN , that sample experiences from individual agents , DGN samples experiences based on the graph of agents , not individual agents , and thus takes into consideration the interactions between agents . Nevertheless , the parameter-sharing of DGN does not prevent the emergence of sophisticated cooperative strategies , as we will show in the experiments . Note that during execution each agent only requires the ( latent ) features from its neighbors ( e.g. , via communication ) regardless of the number of agents , which makes DGN easily scale .

Summary: This paper introduces Graph Convolutional Reinforcement Learning (referred to as DGN). DGN is a Deep Q-Learning (DQN) agent structured as a graph neural network / graph convolutional network with multi-head dot product attention as a message aggregation function. Graphs are obtained based on spatial neighborhoods (e.g. k nearest neighbors) or based on network structure in the domain. DGN considers a multi-agent setting with a centralized learning algorithm and shared parameters across all (controlled) agents, but individually allocated reward. Further, the paper considers environments where other non-learning agents are present which follow a pre-trained, stationary policy. In addition to the attention-based multi-agent architecture, the paper introduces a regularizer on attention weights similar to the use of target networks in DQN, to stabilize training. Results demonstrate that the proposed model architecture outperforms related earlier agent architectures that do not use attention or use a fully-connected graph.