Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning

1 INTRODUCTION . Humans are highly social creatures and spend vast amounts of time coordinating , collaborating and communicating with others . In contrast to these , at least partially , cooperative settings most progress on AI in games has been in zero-sum games where agents compete against each other , typically rendering communication futile . This includes examples such as Go ( Silver et al. , 2016 ; 2017 ; 2018 ) , poker ( Brown & Sandholm , 2017 ; Moravčík et al. , 2017 ; Brown & Sandholm , 2019 ) and chess ( Campbell et al. , 2002 ) . This narrow focus is unfortunate , since communication and coordination require unique abilities . In order to enable smooth and efficient social interactions of groups of people , it is commonly required to reason over the intents , points of views and beliefs of other agents from observing their actions . For example , a driver can reasonably infer that if a truck in front of them is slowing down when approaching an intersection , then there is likely an obstacle ahead . Furthermore , humans are both able to interpret the actions of others and can act in a way that is informative when their actions are being observed by others , capabilities that are commonly called theory of Mind ( ToM ) , ( Baker et al. , 2017 ) . Importantly , in order to carry out this kind of reasoning , an agent needs to consider why a given action is taken and what this decision indicates about the state of the world . Simply observing what other agents are doing is not sufficient . While these abilities are particularly relevant in partially observable , fully cooperative multi-agent settings , ToM reasoning clearly matters in a variety of real world scenarios . For example , autonomous cars will likely need to understand the point of view , intents and beliefs of other traffic participants in order to deal with highly interactive settings such as 4-way crossing or dense traffic in cities . Hanabi is a fully cooperative , partially-observable card game that has recently been proposed as a new benchmark challenge problem for AI research ( Bard et al. , 2019 ) to fill the gap around ToM . In Hanabi , players need to find conventions that allow them to effectively exchange information from their local observations through their actions , taking advantage of the fact that actions are observed by all team mates . Most prior state-of-the-art agents for Hanabi were developed using handcrafted algorithms , which beat off-the-shelf deep multi-agent RL methods by a large margin . This makes intuitive sense : Beyond the “ standard ” multi-agent challenges of credit assignment , nonstationarity and joint exploration , learning an informative policy presents an additional fundamentally new conflict . On the one hand , an RL agent needs to explore in order to discover good policies through trial and error . On the other hand , when carried out naively , this exploration will add noise to the policy of the agent during the training process , making their actions strictly less informative to their team mates . One possible solution to this is to explore in the space of deterministic partial policies , rather than actions , and sample these policies from a distribution that conditions on a common knowledge Bayesian belief . This is successfully carried out in the Bayesian Action Decoder ( BAD ) ( Foerster et al. , 2019 ) , the only previous Deep RL method to achieve a state-of-the-art in Hanabi . While this is a notable accomplishment , it comes at the cost of simplicity and generality . For a start , BAD requires an explicit common knowledge Bayesian belief to be tracked , which not only adds computational burden due to the required sampling steps , but also uses expert knowledge regarding the game dynamics . Furthermore , BAD , as presented , is trained using actor-critic methods which are sample inefficient and suffer from local optima . In order to get around this , BAD uses population based training , further increasing the number of samples required . Lastly , BAD ’ s explicit reliance on common knowledge limits the generality of the method . In this paper we propose the Simplified Action Decoder ( SAD ) , a method that achieves a similar goal to BAD , but addresses all of the issues mentioned above . At the core of SAD is a different approach towards resolving the conflict between exploration and being interpretable , which , like BAD , relies on the centralized training with decentralized control ( CT/DC ) regime . Under CT/DC information can be exchanged freely amongst all agents during centralized training , as long as the final policies are compatible with decentralized execution . The key insight is that during training we do not have to chose between being informative , by taking greedy actions , and exploring , by taking random actions . To be informative , the greedy actions do not need to be executed by the environment , but only need to be observed by the team mates . Thus in SAD each agent takes two different actions at each time step : One greedy action , which is not presented to the environment but observed by the team mates at the next time step as an additional input , and the “ standard ” ( exploratory ) action that gets executed by the environment and is observed by the team mates as part of the environment dynamics . Importantly , during greedy execution the observed environment action can be used instead of centralized information for the additional input , since now the agent has stopped exploring . Furthermore , to ensure that these greedy actions and observations get decoded into a meaningful representation , we can optionally train an auxiliary task that predicts key hidden game properties from the action-observation trajectories . While we note that this idea is in principle compatible with any kind of model-free deep RL method with minimal modifications to the core algorithm , we use a distributed version of recurrent DQN in order to improve sample efficiency , account for partial observability and reduce the risk of local optima . We also train a joint-action Q-function that consists of the sum of per-agent Q-values to allow for off-policy learning in this multi-agent setting using Value Decomposition Networks ( VDN ) ( Sunehag et al. , 2017 ) . Using SAD we establish a new SOTA for learning methods for 2-5 players in Hanabi , with a method that not only requires less expert knowledge and compute , but is also more general than previous approaches . In order to ensure that our results can be easily verified and extended , we also evaluate our method on a proof-of-principle matrix game and open-source our training code and agents . Beyond enabling more research into the self-play aspect of Hanabi , we believe these resources will provide a much needed starting point for the ad-hoc teamwork part of the Hanabi challenge . 2 RELATED WORK . Our work relates closely to research on emergent communication protocols using deep multi-agent RL , as first undertaken by Sukhbaatar et al . ( 2016 ) and Foerster et al . ( 2016 ) . There has been a large number of follow-up papers in this area , so listing all relevant work is beyond the scope and we refer the reader to Nguyen et al . ( 2018 ) , a recent survey on deep multi-agent RL . One major difference to our work is that the environments considered typically contain a cheap-talk channel , which can be modeled as a continuous variable during the course of training . This allows agents to , for example , use differentiation across the communication channel in order to learn protocols . In contrast , in our setting agents have to communicate through the observable environment actions themselves , requiring fundamentally different methods . Furthermore , our work is an example of cooperative multi-agent learning in partially observable settings under centralized training and decentralized control . There have been a large number of papers in this space , with seminal work including MADDPG ( Lowe et al. , 2017 ) and COMA ( Foerster et al. , 2018a ) , both of which are actor-critic methods that employ a centralized critic with decentralized actors . Again , we refer the reader to Nguyen et al . ( 2018 ) for a more comprehensive survey . Until 2018 , work on Hanabi had been focused on hand-coded methods and heuristics . Some relevant examples include SmartBot ( O ’ Dwyer , 2019 ) and the so-called “ hat-coding ” strategies , as implemented by WTFWThat ( Wu , 2018 ) . These strategies use the information theoretic ideas that allow each hint to reveal information to all other agents at the same time . While they do not perform well for 2-player Hanabi due to the smaller action space , they get near perfect scores for 3-5 players . In contrast , so far learning methods have seen limited success on Hanabi . Bard et al . ( 2019 ) undertake a systematic evaluation of current Deep RL methods for 2-5 players in two different regimes and open-source the Hanabi-Learning-Environment ( HLE ) to foster research on the game . They evaluate a feed-forward version of DQN trained on 100 million samples and a recurrent actor-critic agent with population based training using 20 billion samples . Notably , while both agents achieve near 0 % win rate for 3-5 players in Hanabim , at a high level their DQN agent is a good starting point for our work . However , since the authors did not propose any specific method of accounting for the issues introduced by -greedy exploration in a ToM task , they resorted to setting to zero after a short burn-in phase . The only state-of-the-art in Hanabi established by an RL agent is from Foerster et al . ( 2019 ) which we refer to in more detail in Section 1 and Section 4 . Recently there have also been attempts to train agents that are robust to different team-mates ( Canaan et al. , 2019 ) and even to extend to human-AI collaboration ( Liang et al. , 2019 ) . For a more comprehensive review on previous results on Hanabi we refer the reader to Bard et al . ( 2019 ) . Poker is another partially observable multi-agent setting , although it is fundamentally different due to the game being zero-sum . Recent success in Poker has extensively benefited from search ( Brown et al. ) . Examples of using search in Hanabi include Goodman ( 2019 ) . 3 BACKGROUND . 3.1 SETTING . In this paper we assume a Dec-POMDP ( Oliehoek , 2012 ) , in which N agents interact in a partially observable environment . At each time step agent a ∈ 1 .. N obtains an observation , oat = O ( st , a ) , where st ∈ S is the Markov state of the system and O ( st , a ) is the deterministic observation function . Since we are interested in ToM , in our setting the observation function includes the last action of the acting agent , which is observed by all other agents at the next time step . We note that actions are commonly observable not only in board games but also in some real world multi-agent settings , such as autonomous driving . For simplicity , we restrict ourselves to turn based settings , in which at each time step only the acting agents takes an action , uat , which is sampled from their policy , u a ∼ πaθ ( ua|τa ) , while all other agents take a no-op action . Here τa is the action-observation history of agent a , τa = { oa0 , ua0 , r1 , .. rT , oaT } , T is the length of the episode and θ are the weights of a function approximator that represents the policy , in our case recurrent neural networks , such as LSTMs ( Hochreiter & Schmidhuber , 1997 ) . We further use τt to describe the state-action sequence , τ = { s0 , u0 , r1 , .. rT , sT } , where ut is the joint action of all agents . As is typical in cooperative multi-agent RL , the goal of the agents is to maximize the total expected return , Jθ = Eτ∼P ( τ |θ ) R0 ( τ ) , where R0 ( τ ) is the return of the trajectory ( in general Rt ( τ ) =∑ t′≥t γ t′−trt′ ) and γ is an optional discount factor . We have also assumed that agents are sharing parameters , θ , as is common in cooperative MARL .

Summary: This paper introduces a novel exploitation of the centralized training for decentralized execution regime for Dec-POMDPs with publicly/commonly known actions.  In particular, the paper augments independent value-based reinforcement learning by allowing each agent to announce the action it would have taken, had it acted greedily. This relaxation is consistent with decentralized execution, at which time agents always act greedily and no counterfactual announcement is required. The paper demonstrates the utility of this trick in Hanabi, where it achieves strong performance when combined with distributed Q-learning and an auxiliary task. The paper is largely well-written.