Learning Structured Communication for Multi-agent Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) has achieved remarkable success in solving single-agent sequential decision problems under interactive and complicated environments , such as games ( Mnih et al. , 2015 ; Silver et al. , 2016 ) and robotics ( Lillicrap et al. , 2016 ) . In many real world applications such as intelligent transportation systems ( Adler & Blue , 2002 ) and unmanned systems ( Semsar-Kazerooni & Khorasani , 2009 ) , not only one , but usually a large number of agents are involved in the learning tasks . Such a setting naturally leads to the popular multi-agent reinforcement learning ( MARL ) problems , where the key research challenges include how to design scalable and efficient learning schemes under an unstationary environment ( caused by partial observation and/or the dynamics of other agents ’ policies ) , with large and/or dynamic problem dimension , and complicated and uncertain relationship between agents . Learning to communicate among agents has been regarded as an effective manner to strengthen the inter-agent collaboration and ultimately improve the quality of policies learned by MARL . Various communication-based MARL algorithms have been devised recently , e.g. , DIAL ( Foerster et al. , 2016 ) , CommNet ( Sukhbaatar et al. , 2016 ) , ATOC ( Jiang & Lu , 2018 ) , IC3Net ( Singh et al. , 2019 ) and TarMAC ( Das et al. , 2019 ) . These schemes aim to improve the inter-agent collaboration by learning communication strategy to exchange information between agents . However , there are still two bottlenecks unresolved , especially when faced a large number of agents . One bottleneck lies in that achieving effective communication and global collaboration is difficult with limited resources , such as narrow communication bandwidth and energy . In particular , DIAL and TarMAC require each agent to communicate with all the other agents , i.e. , a fully-connected communication network ( Figure 1 ( a ) ) , which is not feasible for large scale scenarios with geographically apart agents . CommNet and IC3 assume a star network ( Figure 1 ( b ) ) with a central node coordinating the global collaboration of agents , which again does not allow large scale scenarios with long range communications . ATOC introduces an interesting attention scheme to build a tree communication network ( Figure 1 ( c ) ) . While the tree network can be scaled , global collaboration has to be realized through inefficient multi-hop and sequential communications . In a word , improper communication topologies will limit the cooperation ability in large scale scenarios . Another bottleneck is the difficulty of extracting essential information to exchange between agents for achieving high-performance MARL , especially when the number of agents grows . Most of the existing works simply concatenate , take the mean or use the LSTM to extract information to be exchanged . First two lack in considering the inter-relationship between agents , and LSTM assumes that there is a fixed sequence of message passing between agents , that is , the relationship between agents is predefined . Recently , TarMAC utilized an attention scheme to aggregate messages by considering the relationship from each agent to all others . However , the improper communication topology still hinders the information extraction . The communication structure needs to be jointly designed with the information extraction scheme to achieve further improved learning performance . To address the above two issues , we propose a novel structured communication-based algorithm , called learning structured communication ( LSC ) . Our LSC combines a structured communication network module and a communication-based policy module , which aims to establish a scalable hierarchically structured network and information exchange scheme for large scale MARL . In particular , a hierarchically structured communication network ( Figure 1 ( d ) ) is dynamically learned based on local partial observations of agents . In the hierarchically structured network , all agents are grouped into clusters , where global collaboration can be achieved via intra-group and inter-group communications . In contrast to the other three types in Figure 1 , the proposed hierarchical communication network is more flexible and scalable , with fewer resources needed to achieve long-range and global collaboration . The procedure to establish such a hierarchically structured communication network is shown in Figure 2 . To better utilize the relationship between agents given the hierarchically structured communication network and obtain more effective information extraction , graph neural network ( GNN ) ( Scarselli et al. , 2008 ) is employed . In GNN , each communication step involves information embedding and aggregation . Benefiting from the unordered aggregation power and the dynamic graph adaptability of GNN , the proposed LSC algorithm can extract valuable information effectively . The GNN-based information extraction procedure is depicted in Figure 3 . This paper is devoted to the learning of communication structure among agents . To our knowledge , this is the first work of hierarchical structured learning to communication for MARL . It allows to learn communication structure adaptively instead of using predefined forms . Specifically : i ) To improve scalability for a large number of agents , a hierarchical structure is devised that divides the agents into higher-level central agents and sub-level normal ones . As such , the communication network is sparsified . While it still allows for more effective global cooperation via message passing among the central agents , compared with the star/tree structures . ii ) For effective communication and global cooperation , the message representation learning is deeply integrated into the information aggregating and permeating through the network , via graph neural network ( GNN ) , which is a natural combination with the hierarchical communication structure . iii ) Extensive experiments on both MAgent and StarCraft2 show our approach achieves state-of-theart scalability and effectiveness on large-scale MARL problems . 2 RELATED WORK AND PRELIMINARIES . Many multi-agent reinforcement learning algorithms without communication in the inference procedure have experienced fast development . Recent works like MADDPG ( Lowe et al. , 2017 ) , QMIX ( Rashid et al. , 2018 ) , COMA ( Foerster et al. , 2018 ) and MAAC ( Iqbal & Sha , 2019 ) adopt a centralized training and decentralized implementing framework . All agents ’ local observations and actions are considered to improve the learning stability . These algorithms are generally not suitable for large-scale case due to explosive growing number of agents . Communication-based MARL algorithms have been showed effective for large-scale agent cooperation . Earlier works assume that all agents need to communicate with each other . DIAL ( Foerster et al. , 2016 ) learns to communication through back-propagating all other agents ’ gradients to the message generator network . Similarly , CommNet ( Sukhbaatar et al. , 2016 ) sends all agents ’ hidden states to the shared communication channel and further learns the message based on the average of all other hidden states . MFRL ( Yang et al. , 2018 ) approximates the influence of other agents by averaging the actions of surrounding neighbor agents , which could mitigate the dimensional disaster for large-scale cases . However , this can be considered as a predefined communication pattern , which is unable to adapt to complex large-scale scenarios . Communication between all agents will lead to high communication complexity and difficulty of useful information extraction . DGN ( Jiang et al. , 2018 ) employs graph convolution network ( GCN ) to extract relationships between agents which could result in better collaboration . However , it considers all agents equivalently and assumes the communication of each agent has to involve all neighbor agents which limits to adapt to more practical bandwidth-limited environments . IC3Net ( Singh et al. , 2019 ) uses a communication gate to decide whether to communicate with the center , but adopt the same star structure like CommNet which requires high bandwidth and can hard to extract valuable information with only one center . ATOC ( Jiang & Lu , 2018 ) and TarMAC ( Das et al. , 2019 ) introduce the attention mechanism to determine when to communicate and whom to communicate with , respectively . TarMAC focuses more on message aggregation rather than the communication structure . SchedNet ( Kim et al. , 2019 ) aims to learn a weight-based scheduler to determine the communication sequence and priority . From the perspective of employing GNN into MARL , MAGNet ( Malysheva et al. , 2018 ) that utilizes a relevance graph representation of the environment and a message passing mechanism to help agents learning . However , it requires heuristic rules to establish the graph which is hard to achieve in complex environments . RFM ( Tacchetti et al. , 2019 ) use graph to represent the relationship between different entities , aiming to provide interpretable representations . Before the main method , we introduce some preliminaries to facilitate the presentation . Partial Observable Stochastic Games . In stochastic games , agents learn policies by maximizing their cumulative rewards through interacting with the environment and other agents . The partial observable stochastic games ( POSG ) can be characterized as a tuple 〈 I , S , b0 , A , O , P , Pe , R 〉 where I denotes the set of agents indexed from 1 to n ; S denotes the finite set of states ; b0 represents the initial state distribution and A denotes the set of joint actions . Ai is the action space of agent i , a = 〈a1 , · , an〉 denotes a joint action ; O denotes the joint observations and Oi is the observation space for agent i , o = 〈o1 , · , on〉 denotes a joint observation ; P denotes the Markovian transition distribution with P ( s̃ , o ∣∣s , a ) as the probability of state s transit to s̃ and result o after taking action a. Pe ( o|s ) is the Markovian observation emission probability function . R : S × A → Rn means the reward function for agents . The overall task of the MARL problem can be solved by properly objective function modeling , which also indicates the relationship among agents , e.g. , cooperation , competition or mixed . Graph Neural Network . Graph neural network ( GNN ) ( Scarselli et al. , 2008 ) is a deep embedding framework to handle graph-based data on a graph G = ( V , E ) . vi denotes the node feature vector for node vi ∈ V ( for Nv nodes ) , ek denotes the edge feature vector for edge ek ∈ E ( for Ne edges ) with rk , sk be the receiver and sender of edge ek respectively . The vector u denotes the global feature . The graph network framework in ( Battaglia et al. , 2018 ) is employed , which divides computation on graph data to several blocks to gain flexible processing ability . Each block introduces the aggregation and embedding functions to handle graph data . There are many variants of GNN , like messagepassing neural network ( Gilmer et al. , 2017 ) and non local neural networks ( Wang et al. , 2018 ) . By treating every agent as a node and each communication message exchanging as the edge in a graph , the observations and messages as the attributes of nodes and edges , respectively . The whole communication process can be formulated to a graph neural network . The relationships among agents can be effectively extracted to enable efficient communication message learning . Independent Deep Q-Learning . Deep Q-Network ( DQN ) ( Mnih et al. , 2015 ) is popular in deep reinforcement learning , which is one of the few RL algorithms applicable for large-scale MARL . In each step , each agent observes state s and takes an action a based on policy π . It receives reward r and next state s̃ from environment . To maximize the cumulative reward R = ∑ t rt , DQN learns the action-value function Qπ ( s , a ) = Es∼P , a∼π ( s ) [ Rt|st = s , at = a ] by minimizing L ( θ ) = Es , a , r , s̃ [ ỹ −Q ( s , a ; θ ) ] , where ỹ = r + γmaxãQ ( s̃ , ã ; θ ) . The agent follows -greedy policy , that is , selects the action that maximizes the Q-value with probability 1- or randomly . The Independent Deep Q-Learning ( IDQN ) ( Tampuu et al. , 2017 ) is an extension of DQN by ignoring the influence of other agents for multi-agent case . Every agent learns a Q-function Qa ( ua|s ; θa ) based on its own observation and received reward . Our algorithm employs DQN as the basic RL algorithm based on the following two considerations : 1 ) our algorithm is dedicated to discuss the learning communication mechanism in largescaleMARL scenarios , as a result we can choose a concise and effective basic RL algorithm like thewell-known DQN ; 2 ) data collection in large-scale MARL environments is extremely inefficiently , while DQN has excellent data efficiency as an offline RL algorithm .

Summary: This paper proposes a method of learning a hierarchical communication graph for improving collaborative multi-agent reinforcement learning, particularly with large numbers of agents. The method is compared to a suitable range of baseline approaches across two complex environments. The initial results presented seem promising, but further work is needed to ensure the results are reproducible and repeatable.