Communication in Multi-Agent Reinforcement Learning: Intention Sharing

1 INTRODUCTION . Reinforcement learning ( RL ) has achieved remarkable success in various complex control problems such as robotics and games ( Gu et al . ( 2017 ) ; Mnih et al . ( 2013 ) ; Silver et al . ( 2017 ) ) . Multi-agent reinforcement learning ( MARL ) extends RL to multi-agent systems , which model many practical real-world problems such as connected cars and smart cities ( Roscia et al . ( 2013 ) ) . There exist several distinct problems in MARL inherent to the nature of multi-agent learning ( Gupta et al . ( 2017 ) ; Lowe et al . ( 2017 ) ) . One such problem is how to learn coordinated behavior among multiple agents and various approaches to tackling this problem have been proposed ( Jaques et al . ( 2018 ) ; Pesce & Montana ( 2019 ) ; Kim et al . ( 2020 ) ) . One promising approach to learning coordinated behavior is learning communication protocol among multiple agents ( Foerster et al . ( 2016 ) ; Sukhbaatar et al . ( 2016 ) ; Jiang & Lu ( 2018 ) ; Das et al . ( 2019 ) ) . The line of recent researches on communication for MARL adopts end-to-end training based on differential communication channel ( Foerster et al . ( 2016 ) ; Jiang & Lu ( 2018 ) ; Das et al . ( 2019 ) ) . That is , a message-generation network is defined at each agent and connected to other agents ’ policies or critic networks through communication channels . Then , the message-generation network is trained by using the gradient of other agents ’ policy or critic losses . Typically , the message-generation network is conditioned on the current observation or the hidden state of a recurrent network with observations as input . Thus , the trained message encodes the past and current observation information to minimize other agents ’ policy or critic loss . It has been shown that due to the capability of sharing observation information , this kind of communication scheme has good performance as compared to communication-free MARL algorithms such as independent learning , which is widely used in MARL , in partially observable environments . In this paper , we consider the following further question for communication in MARL : ” How to harness the benefit of communication beyond sharing partial observation. ” We propose intention of each agent as the content of message to address the above question . Sharing intention using communication has been used in natural multi-agent systems like human society . ∗Corresponding author For example , drivers use signal light to inform other drivers of their intentions . A car driver may slow down if a driver in his or her left lane turns the right signal light on . In this case , the signal light encodes the driver ’ s intention , which indicates the driver ’ s future behavior , not current or past observation such as the field view . By sharing intention using signal light , drivers coordinate their drive with each other . In this paper , we formalize and propose a new communication scheme for MARL named Intention sharing ( IS ) in order to go beyond existing observation-sharing schemes for communication in MARL . The proposed IS scheme allows each agent to share its intention with other agents in the form of encoded imagined trajectory . That is , each agent generates an imagined trajectory by modeling the environment dynamics and other agents ’ actions . Then , each agent learns the relative importance of the components in the imagined trajectory based on the received messages from other agents by using an attention model . The output of the attention model is an encoded imagined trajectory capturing the intention of the agent and used as the communication message . We evaluate the proposed IS scheme in several multi-agent environments requiring coordination among agents . Numerical result shows that the proposed IS scheme significantly outperforms other existing communication schemes for MARL including the state-of-the-art algorithms such as ATOC and TarMAC . 2 RELATED WORKS . Under the asymmetry in learning resources between the training and execution phases , the framework of centralized training and decentralized execution ( CTDE ) , which assumes the availability of all system information in the training phase and distributed policy in the execution phase , has been adopted in most recent MARL researches ( Lowe et al . ( 2017 ) ; Foerster et al . ( 2018 ) ; Iqbal & Sha ( 2018 ) ; Kim et al . ( 2020 ) ) . Under the framework of CTDE , learning communication protocol has been considered to enhance performance in the decentralized execution phase for various multi-agent tasks ( Foerster et al . ( 2016 ) ; Jiang & Lu ( 2018 ) ; Das et al . ( 2019 ) ) . For this purpose , Foerster et al . ( 2016 ) proposed Differentiable Inter-Agent Learning ( DIAL ) . DIAL trains a message-generation network by connecting it to other agents ’ Q-networks and allowing gradient flow through communication channels in the training phase . Then , in the execution phase the messages are generated and passed to other agents through communication channels . Jiang & Lu ( 2018 ) proposed an attentional communication model named ATOC to learn when to communicate and how to combine information received from other agents through communication based on attention mechanism . Das et al . ( 2019 ) proposed Targeted Multi-Agent Communication ( TarMAC ) to learn the message-generation network in order to produce different messages for different agents based on a signature-based attention model . The message-generation networks in the aforementioned algorithms are conditioned on the current observation or a hidden state of LSTM . Under partially observable environments , such messages which encode past and current observations are useful but do not capture any future information . In our approach , we use not only the current information but also future information to generate messages and the weight between the current and future information is adaptively learned according to the environment . This yields further performance enhancement , as we will see in Section 5 . In our approach , the encoded imagined trajectory capturing the intention of each agent is used as the communication message in MARL . Imagined trajectory was used in other problems too . Racanière et al . ( 2017 ) used imagined trajectory to augment it into the policy and critic for combining modelbased and model-free approaches in single-agent RL . It is shown that arbitrary imagined trajectory ( rolled-out trajectory by using a random policy or own policy ) is useful for single-agent RL in terms of performance and data efficiency . Strouse et al . ( 2018 ) introduced information-regularizer to share or hide agent ’ s intention to other agents for a multi-goal MARL setting in which some agents know the goal and other agents do not know the goal . By maximizing ( or minimizing ) the mutual information between the goal and action , an agent knowing the goal learns to share ( or hide ) its intention to other agents not knowing the goal in cooperative ( or competitive ) tasks . They showed that sharing intention is effective in the cooperative case . In addition to our approach , Theory of Mind ( ToM ) and Opponent Modeling ( OM ) use the notion of intention . Rabinowitz et al . ( 2018 ) proposed the Theory of Mind network ( ToM-net ) to predict other agents ’ behaviors by using meta-learning . Raileanu et al . ( 2018 ) proposed Self Other-Modeling ( SOM ) to infer other agents ’ goal in an online manner . Both ToM and OM take advantage of predicting other agents ’ behaviors capturing the intention . One difference between our approach and the aforementioned two methods is that we use communication to share the intention instead of inference . That is , the agents in our approach allow other agents to know their intention directly through communication , whereas the agents in ToM and OM should figure out other agents ’ intention by themselves . Furthermore , the messages in our approach include future information by rolling out the policy , whereas ToM and CM predict only the current or just next time-step information . 3 SYSTEM MODEL . We consider a partially observable N -agent Markov game ( Littman ( 1994 ) ) and assume that communication among agents is available . At time step t , Agent i observes its own observation oit , which is a part of the global environment state st , and selects action ait ∈ Ai and messagemit ∈Mi based on its own observation oit and its own previous time step message m i t−1 plus the received messages from other agents , i.e. , mt−1 = ( m1t−1 , · · · , mNt−1 ) . We assume that the message mit of Agent i is sent to all other agents and available at other agents at the next time step , i.e. , time step t + 1 . The joint actions at = ( a1t , · · · , aNt ) yield the next environment state st+1 and rewards { rit } Ni=1 according to the transition probability T : S × A × S → [ 0 , 1 ] and the reward function Ri : S × A → R , respectively , where S and A = ∏N i=1Ai are the environment state space and the joint action space , respectively . The goal of Agent i is to find the policy πi that maximizes its discounted return Rit = ∑∞ t′=t γ t′rit′ . Hence , the objective function of Agent i is defined as Ji ( π i ) = Eπ [ Ri0 ] , where π = ( π1 , · · · , πN ) and γ ∈ [ 0 , 1 ] are the joint policy and the discounting factor , respectively . 4 THE PROPOSED INTENTION SHARING SCHEME . The key idea behind the IS scheme is that multiple agents communicate with other agents by sending their implicit future plans , which carry their intention . The received messages capturing the intention of other agents enable the agent to coordinate its action with those of other agents . We now describe the architecture of the proposed IS scheme . At time step t , Agent i selects an action ait ∼ πi ( ·|oit , mt−1 ) and a message mit = MGN i ( oit , mt−1 , πi ) based on its own observation oit and received messages mt−1 , where MGN i is the message-generation network ( MGN ) of Agent i . The MGN consists of two components : Imagined trajectory generation module ( ITGM ) and attention module ( AM ) . Each agent generates an imagined trajectory by using ITGM and learns the importance of each imagined step in the imagined trajectory by using AM . The output of AM is an encoded imagined trajectory reflecting the importance of imagined steps and is used as the communication message . The overall architecture of the proposed IS scheme is shown in Fig . 1 . In the following we describe the detail of each module . 4.1 IMAGINED TRAJECTORY GENERATION MODULE ( ITGM ) . The role of ITGM is to produce the next imagined step . ITGM takes the received messages , observation , and action as input and yields the predicted next observation and predicted action as output . By stacking ITGMs , we generate an imagined trajectory , as shown in Fig . 1 . For Agent i at time step t , we define an H-length imagined trajectory as τ i = ( τ it , τ̂ i t+1 , · · · , τ̂ it+H−1 ) , ( 1 ) where τ̂ it+k = ( ô i t+k , â i t+k ) is the imagined step at time step t + k. Note that τ i t = ( o i t , a i t ) is the true values of observation and action , but the imagined steps except τ it are predicted values . ITGM consists of a roll-out policy and two predictors : Other agents ’ action predictor f ia ( o i t ) ( we will call this predictor simply action predictor ) and observation predictor f io ( o i t , a i t , a −i t ) . First , we model the action predictor which takes the observation as input and produces other agents ’ predicted actions . The output of the action predictor is given by f ia ( o i t ) = ( â 1 t , · · · , âi−1t , âi+1t , · · · , âN ) = : â−it ( 2 ) Note that the action predictor can be trained by the previously proposed opponent modeling method ( Rabinowitz et al . ( 2018 ) ; Raileanu et al . ( 2018 ) ) and can take the received messages as input . Next , we model the observation predictor f io ( o i t , a i t , â −i t ) which is conditioned on the observation o i t , own action ait , and the output of the action predictor â −i t . Here , we adopt the dynamics function that predicts the difference between the next observation and the current observation , i.e. , oit+1 − oit instead of the next observation oit+1 proposed in ( Nagabandi et al . ( 2018 ) ) in order to reduce model bias in the early stage of learning . Hence , the next observation can be written as ôit+1 = o i t + f i o ( o i t , a i t , â −i t ) . ( 3 ) By injecting the predicted next observation and the received messages into the roll-out policy in ITGM , we obtain the predicted next action âit+1 = π i ( ôit+1 , mt−1 ) . Here , we use the current policy as the roll-out policy . Combining ôit+1 and â i t+1 , we obtain next imagined step at time step t + 1 , τ it+1 = ( ô i t+1 , â i t+1 ) . In order to produce an H-length imagined trajectory , we inject the output of ITGM and the received messages mt−1 into the input of ITGM recursively . Note that we use the received messages at time step t , mt−1 , in every recursion of ITGM.1

Summary: Paper proposed to generate the communication message in MARL with the predicted trajectories of all the agents (include the agent itself). An extra self-attention model is also stacked over the trajectories to trade off the length of prediction and the possible explaining away issue.  The whole model is trained via a canonical MARL objective while the trajectory prediction model utilizes direct supervision collected from the environments. Experiments on several toy MARL benchmark demonstrates the effectiveness of the proposed method.