Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
Authors: Hengyuan Hu
Jakob N Foerster
Venue:  (2020)
Keywords: multi-agent RL
theory of mind

Abstract: In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e. the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with an auxiliary task for state prediction and best practices for multi-agent learning, SAD establishes a new state of the art for 2-5 players on the self-play part of the Hanabi challenge.

Published as a conference paper at ICLR 2020
SIMPLIFIED ACTION DECODER FOR DEEP MULTI-AGENT REINFORCEMENT LEARNING
Hengyuan Hu, Jakob N Foerster Facebook Al Research, CA, USA (hengyuan, jnf) & @fb.com .com
ABSTRACT
In recent years we have seen fast progress on a number of benchmark problems AL, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for Al to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e., the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present new deep multi-agent RL. method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with best practices for multi-agent learning, SAD establishes a new SOTA for learning methods for 2-5 players on the self-play part of the Hanabi challenge. Our ablations show the contributions of SAD compared with the best practice components, All of our code and trained agents are available at httt:s//ggghuuuo.ooonceeebeeeeee
Hanahiinnnaa_a.. _SD
INTRODUCTION
Humans highly social creatures and spend vast amounts of time coordinating, collaborating and communicating with others. In contrast to these, at least partially, cooperative settings most progress AI in games has been in zero-sum games where agents compete against each other, typically rendering communication futile. This includes examples such as Go (Silver et al. 2016 2017 2018). poker (Brown & Sandholm 2017 Moravčík et 2017 and chess (Campbell et al. 2002).
Brown & Sandholm 2019
This narrow focus unfortunate, since communication and coordination require unique abilities. In order to enable smooth and efficient social interactions of groups of people, it is commonly required to reason over the intents, points of views and beliefs of other agents from observing their actions. For example, driver can reasonably infer that if a truck in front of them is slowing down when approaching an intersection, then there likely an obstacle ahead. Furthermore, humans are both able to interpret the actions of others and can act in a way that is informative when their actions being observed by others, capabilities that are commonly called theory of Mind (ToM), (Baker et al.) agent needs to consider why
2017). Importantly, in order carry this kind of reasoning, given action is taken and what this decision indicates about the state of the world. Simply observing what other agents are doing is not sufficient.
While these abilities are particularly relevant in partially observable, fully cooperative multi-agent settings, ToM reasoning clearly matters in variety of real world scenarios. For example, autonomous

---

Published as a conference paper at ICLR 2020
cars will likely need to understand the point of view, intents and belief's of other traffic participants in order to deal with highly interactive settings such as 4-way crossing or dense traffic in cities.
Hanabi is fully cooperative, carteallyobseervabee card game that has recently been proposed as new benchmark challenge problem for AI research (Bard et al., 2019) to fill the gap around ToM. In Hanabi, players need to find conventions that allow them to effectively exchange information from their local observations through their actions, taking advantage of the fact that actions are observed by all team mates.
Most prior state-of-the-art agents for Hanabi were developed using handcrafted algorithms, which beat off-the-shee deep multi-agent RL methods by a large margin. This makes intuitive sense: Beyond the "standard" multi-agent challenges of credit assignment, nonstationarity and joint exploration, learning an informative policy presents an additional fundamentally new conflict. On the one hand, an RL agent needs to explore in order to discover good policies through trial and error. On the other hand, when carried out naively, this exploration will add noise to the policy of the agent during the training process, making their actions strictly less informative to their team mates.
One possible solution to this is to explore in the space of deterministic partial policies, rather than actions, and sample these policies from a distribution that conditions on a common knowledge Bayesian belief. This successfully carried out in the Bayesian Action Decoder (BAD) (Foerster) et al. 2019), the only previous Deep RL method to achieve state-of-the in Hanabi. While this is notable accomplishment, comes at the cost of simplicity and generality, For a start, BAD requires an explicit common knowledge Bayesian belief to be tracked, which not only adds computational burden due to the required sampling steps, but also uses expert knowledge regarding the game dynamics. Furthermore, BAD, as presented, is trained using actor-critic methods which are sample inefficient and suffer from local optima. In order to get around this, BAD uses population based training, further increasing the number of samples required. Lastly, BAD's explicit reliance on common knowledge limits the generality of the method.
In this paper we propose the Simplified Action Decoder (SAD), a method that achieves a similar goal to BAD, but addresses all of the issues mentioned above. At the core of SAD is different approach towards resolving the conflict between exploration and being interpretable, which, like BAD, relies on the centralized training with decentralized control (CT/DC) regime. Under CT/DC information can be exchanged freely amongst all agents during centrulized training, as long as the final policies are compatible with decentralized execution.
The key insight is that during training we do not have to chose between heing informative, by taking greedy actions, and exploring, by taking random actions. To be informative, the greedy actions do not need to be executed by the environment, but only need to be observed by the team mates. Thus in SAD each agent takes two different actions at each time step: One greedy action, which is not presented to the environment but observed by the team mates at the next time step as an additional input, and the "standard" (exploratory) action that gets executed by the environment and is observed by the team mates as part of the environment dynamics. Importantly, during greedy execution the observed environment action can be used instead of centralized information for the additional input, since now the agent has stopped exploring.
Furthermore, to ensure that these greedy actions and observations get decoded into a meaningful representation, we can optionally train an auxiliary task that predicts key hidden game properties from the oetion-observation trajectories. While we note that this idea is in principle compatible with any kind of model-free deep RL method with minimal modifications to the core algorithm, we use a distributed version of recurrent DQN in order to improve sample efficiency, account for partial observability and reduce the risk of local optima. We also train joint-action Q-function that consists of the sum of per-agent Q-values to allow for off off-policy learning in this multi-agent setting using Value Decomposition Networks (VDN) (Sunchag et al. 2017).
Using SAD we establish a new SOTA for learning methods for 2-5 players in Hanabi, with a method that not only requires less expert knowledge and compute, but is also more general than previous approaches. In order to ensure that our results can be easily verified and extended, we also evaluate our method a proof-of- ofiinciple matrix game and open-source our training code and agents. Beyond enabling more research into the self-play aspect of Hanabi, we believe these resources will provide much needed starting point for the ad- teamwork part of the Hanabi challenge.
2

---

Published as a conference paper at ICLR 2020
2 RELATED WORK
Our work relates closely to research emergent communication protocols using deep multi-agent RL, as first undertaken by SSkhbaataar et al. (016) and Foerster et al. (2016). There has been a large number of follow-up papers in this area, so listing all relevant work is beyond the scope and we refer the reader to Nguyyen et al. (2018). , a recent survey on deep multi-agent RL. One major difference to our work is that the environments considered typically contain cheap-talk channel, which can be modeled as a continuous variable during the course of training. This allows agents to, for example, use differentiation across the communication channel in order to learn protocols. In contrast, in our setting agents have to communicate through the observable environment actions themselves, requiring fundamentally different methods.
Furthermore, our work is an example of cooperative multi-agent learning in partially observable settings under centralized training and decentralized control. There have been large number of papers in this space, with seminal work including MADDPG (Lowe et al. (017)) and COMA (Foerster) al. 2018a), both of which are actor-critic methods that employ a centralized critic with decentralized actors. Again, we refer the reader to Nguyen et al. (2018) for more comprehensive survey.
Until 2018, work on Hanabi had been focused on hand-coded methods and heuristics. Some relevant examples include SmartBot Dwwer 2019) and the so-called "hat-coding" strategies, as implemented by WTFWThat (Wul 2018). These strategies use the information theoretic ideas that allow each hint to reveal information to all other agents at the same time. While they do not perform well for 2-player Hanabi due to the smaller action space, they get near perfect scores for 3-5 players.
In contrast, far learning methods have seen limited success on Hanabi. [Brrd et al. (2019) undertake a systematic evaluation of current Deep RL methods for 2-5 players in two different regimes and open-source the wawaaa---aannnn--aviinnnoo
(HI.E) to foster research on the game. They evaluate a feed-forward version of DQN trained on 100 million samples and recurrent actor-critic agent with population based training using 20 billion samples. Notably, while both agents achieve near 0% win rate for 3-5 players in Hanabim, at a high level their DQN agent is a good starting point for our work. However, since the authors did not propose any specific method of accounting for the issues introduced by e-greedy exploration in a ToM task, they resorted to setting to zero after a short burn-in phase. The only state-of-the-ee in Hanabi established by an RL agent is from Foerster et al. (2019) which we refer to in more detail in Section and Section 4 Recently there have also been attempts to train agents that are robust to different geam-mates (Canaan et al.) (019)) and even to extend to human-Al collaboration (Liang et al. 2019). For a more comprehensive review previous results on Hanabi we refer the reader to Bard et al. (2019).
Poker is another partially observable multi-agent setting, although it is fundamentally different due to the game being zero-sum. Recent success in Poker has extensively benefited from search (Brown) et )..) Examples of using search in Hanabi include Goodman (2019).
BACKGROUND
3.1 SETTING
In this paper we assume a Dec-POMDP (Olichoek 2012). in which N agents interact in a partially observable environment. At each time step agent E I..N obtains an observation, of = O(ss, a). where s is the Markov state of the system and O(s,. a) the deterministic observation function. Since we are interested in ToM, in our setting the observation function includes the last action of the acting agent, which is observed by all other agents at the next time step. We note that actions are commonly observable not only in board games but also in some real world multi-agent settings, such as autonomous driving.
For simplicity, we restrict ourselves to turn based settings, in which at each time step only the acting agents takes an action, uf, which is sampled from their policy, u² ~ x (u((r") while all other agents take no-op action. Here 7" is the citionnobservation history of agent o, T {oß ug. "g, ..r,, of T is the length of the episode and 9 are the weights of a function approximator that represents the policy, in our case recurrent neural networks, such as LSTMs (Hochreiter & Schmidhuber 1997).
3

---

Published as a conference paper at ICLR 2020
We further Ty to describe the state-action sequence, {su, Uo "1. 8T). where u, is the joint action of all agents.
As is typical in cooperative multi-agent RL, the goal of the agents is to maximize the total expected return, J0 = (
where Ro(T) is the return of the trajectory (in general R((T) =
Exsr? "rrr) and y is an optional discount factor. We have also assumed that agents are sharing parameters, 0, as is common in cooperative MARL.
3.2 DISTRIBUTED RECURRENT DQN AND AUXILIARY TASKS
In Q-learning the agent approximates the expected return for a given state action-pair, 5, u, assuming that the agent acts greedily with respect to the Q-function for all future time steps, Qss, u) E,-P(r]ss.u) (T), where 7 = {st, 24 "'+1,...,, ST,, Uy = u and ue = arg max,, Q( u Vi' A common exploration scheme is e-greedy, in which the agent takes a random action with probability and acts greedily otherwise. Importantly, the Q-function can be trained efficiently using the Bellman equation: Q(s, u) = Egg(r (re+1 + maxy Q(s', where for simplicity we have assumed a deterministic reward. In Deep Q-Learning (DQN) (Mnih et al. 2015)) the Q-function is parameterized by a deep neural network and trained with transitions sampled from experience replay.
In our work we also incorporate other best practice components of the last few years, including double- DQN (van Hasselt et al., 2015), dueling network architecture (Wang et al., 2015) and prioritized replay (Schaul et al. (015)) We also employ distributed training architecture similar to the one proposed by Horgan et al. (018)) where number of different actors with their own exploration rates collect experiences in parallel and feed them into central replay buffer. Since our setting is partially observable the natural choice for the function approximator is a recurrent neural network. A combination of these techniques was first explored by Kapturowski et al. (2019) in single agent environments such as Atari and DMLab-30.
Another common best-practice in RL are auxiliary tasks Mirowski et al. (016): Jaderberg al. (016)) in which the agent produces extra sutput-heads that are trained on supervised tasks and optimized alongside the RL loss.
3.3 CENTRALISED TRAINING, DECENTRALIZED EXECUTION AND JOINT Q-FUNCTIONS
The most straight forward application of Q-learning to multi-agent settings is Independent Q-Learning (IQL. Tan 1993) in which each agent keeps independent estimate of the expected return, treating all other agents as part of the environment. One challenge with IQL is that the exploratory behavior of other agents is not corrected for via the max operator in the bootstrap. Notably, IQL does typically not take any advantage of centralized training with decentralised control (CT/DC), a paradigm under which information can be exchanged freely amongst agents during the training phase as long as the policies rely only on local observations during execution.
There are various approaches for learning joint-Q- f-fucctions in the CT/DC regime. For example, Value- Saaiee-DccomosttionnNeeee Networrs (VDN) (Sunehag et al., 2017) represent the joint-Q-function as sum of per-agent contributions and QMIX (Rashid et al. 2018) learns non-linear monotonic combination of these contributions.
4 METHOD
4.1 THEORY OF MIND AND BAYESIAN REASONING
At the very core of interpreting the actions of another agent, and ToM in general, is Bayesian reasoning. Fundamentally, asking what a given action by another agent implies about the state of the world requires understanding of why this action was taken. To illustrate this, start out with an agent that has a given belief about the state-action action history of the world, T,, given her own action-observation history of: B(Tt)= P.
4

---

Published as a conference paper at ICLR 2020
Next the agent observes the action uff of her team mate, a', and carries out Bayesian update:
Plug (m) P(T)TT))
P(TilTi, uz
(1)
Ex P(uf' P()
n" (uf 10 T.) B(T)) Σxj x" (uf (0 B(T))
(2)
where, with a slight abuse of notation, we have used (and will keep using) O(d', .7)) for the action- observation history, 7/" that results from applying the observation function for agent a' to Tr at each time step. Note that for non-deterministic observation functions would have to marginalize over P(r (T))
Clearly, since agents have access to the policy of their teammate during centralised training, we could in principle evaluate this explicit Bayesian belief. However, beyond the practical difficulty of computing this explicit belief, when it is used as an input to the policy it will lead to prohibitively costly higher order belief'.. The typical workout for this public belief over private features which only conditions on common knowledge and can therefore be calculated by all agents individually, we refer Moravčík et al. (2017); Nayyar al. (2013); Foerster et al. (2018b) for more details.
Instead, in this work we rely on RNNs to learn implicit representations of the sufficient statistics over the distribution of the Markov state given the -otie-oobsevatiion histories, noting that they are unlikely to recover exact beliefs due to the issues mentioned above.
4.2 EXPLORATION AND BELIEFS
Next illustrate the impact of exploration on the beliefs, which we will do in the explicit (exact) case, since it serves as an upper bound on the accuracy of the implicit beliefs. Since we are looking at fully-cooperatin settings we assume that the optimal policy of the agent is deterministic and any randomness is due to exploration. Given that we are focused on value based methods we furthermore assume an e-greedy exploration scheme, noting that the same analysis can be extended to other methods. Under this exploration scheme nt (ut 10 re)) becomes:
na (ut 10(a,, T()) (1 ()l=))((""))) (T)), 24 + e/|UI.
(3)
where have used u² (Tt) to indicate the greedy action of the agent d' u' (Ta)
arg max, Q=' (u, Oa',, T:)) and I is the indicator function.
While the first part corresponds to a filtering operator, in which the indicator function only attributes finite probability to those histories that are consistent with the action taken under greedy execution, the exploration term adds a fixed (history independent) probability, which effectively 'blurs' the posterior:
P(Tl77 uf')
(4)
(5)
e/101 + -. ((I-)I((u* (T'), B(T))
1+10 B(T.)
(6)
up B(T") (11=)[(u"(>>) B(T.)
c/101 + +., ((1-e) (T'), ug B(T')
We find that the posterior includes an additional term of the form B(T)) which carries over an unfiltered density over the trajectories from the prior. We further confirm that in the limit of c 1, the posterior collapses to the prior, P(Tel7f uf') = B(T.). This can be particularly worrisome in
5

---

Published as a conference paper at ICLR 2020
the context of our training setup, whereby different agents run different, and potentially high, throughout the course of training. fundamentally makes the beliefs obtained less informative.
While not making the above argument explicitly, the Bayesian Action Decoder (BAD) (Foerster et al., 2019), resolves this issue by shifting exploration to the level of deterministic partial policies, rather than action-level, and tracking an approximate Bayesian belief. As outlined in Section [ comes at a huge cost in the complexity of the method, the computation requirements and in the loss of generality of the method.
4.3 SIMPLIFIED ACTION DECODING
In this paper we take a drastically simpler and different approach towards the issue. We note that the 'bluring', which makes decoding of action challenging, is entirely due to the e-greedy exploration term. Furthermore, in order for another agent to do an implicit Bayesian update over an action taken, it is not required that this action is executed by the environment. Indeed, if assume that other agents can observe the greedy action, u', at every time step and condition their belief update on this, the terms depending on e disappear from the Bayesian update:
I(u (T=), P(TalTf,"") u²)
((u u²) ()) B(T')
Therefore, to have our cake and eat it, in the Simplified Action Decoder (SAD) the acting agent allowed to 'take' two actions at any given time step during training. The first action, u², is the standard environment action, which gets executed as usual and is observed by all agents through the observation function at the next time step, as mentioned in Section The second action, u", the greedy action of the active agent. This action does not get executed by the environment but instead presented as an additional input to the other agents the next time step, taking advantage of the centralized training regime during which information can be exchanged freely.
Clearly we are not allowed to pass around extra information during decentralized control, but luckily this is not needed. Since we set to 0 at test time we can simply use the, now greedy, environment action obtained from the observation function as our greedy-action input.
While this is most straight forward in settings where the last action is observed by other agents directly, in principle SAD can also be extended to settings where it is indirectly observed by all agents through the environment dynamics. In these cases we can replace the greedy-action side-channel with a learned inverse model that recovers the action from the observation history during execution.
Furthermore, to encourage the agent to meaningfully decode the information contained in the greedy action, we can optionally add an auxiliary task to the training process, such as predicting unobserved information from their observation history
While this idea is compatible with any deep RL algorithm with minimal modifications, we use a recurrent version of DQN with distributed training, dueling networks and prioritized replay. We also learn a joint Q-function using VDN in order to address the challenges of multi-agent off-policy learning, please Section 3 for details all of these standard methods.
5 EXPERIMENTS
5.1 MATRIX GAME
We first verify the effectiveness of SAD in the two step, two player matrix game from Foerster et al. (2019). which replicates the communication through action challenge of Hanabi in highly simplified setting. In this fully cooperative game each player obtains a privately observed 'card', which is drawn iid from two options (1,2).
After observing her card, the first player takes one of three possible discrete actions (1, 2, 3). Crucially, the second player observes both her own private card and the team mate's action before acting herself, which establishes the opportunity to communicate. The payout is a function of both the two private cards and the two actions taken by both agents, as shown Figure]]

---

Published as a conference paper at ICLR 2020
Importantly, there are some obvious strategies that do not require any communication. For example, if both player learn to play the 2nd action, the payout is always 8 points, independent of the cards dealt. However, the players do learn to communicate it is possible to achieve 10 points for every pair of cards dealt.
Player (acts second)
Card
Card
Player action
Players action
A
10
0
0
10
Gard
a
4
4
4
8
4
5.2 HANABI
Player (acts
10
0
10
Hanabi a fully cooperative card game in which all play- work together to complete piles of cards referred to fireworks. Each card has rank, 1 to 5, and a color, G B /W/ W Y/R. Each firework (one per color) starts with and is finished once the 5 has been added. There are three 1s, one 5 and two of all other ranks for each of the colors, adding up to total of 50 cards in the deck. The twist in Hanabi that while players can observe the cards held by their team mates, they cannot observe their own cards and thus need to exchange information with each other in order to understand what cards can be played. There are two main means for doing so: First of all, players can take grounded hint actions, in which they reveal the subset of team mate's 's hand that matches specific rank or color. example hint "Your third and fifth card are 1s". These hint actions cost scarce information tokens, which can be replenished by discarding a card, an action that both removes the card from the game and makes visible to all players.
0
Card 2
8
4
10
0 0
Figure 1: Illustration matrix game Foersterr et al. (2019)
Finally players can also choose to play card. If this card is the next card for the firework of the corresponding color, it is added to the firework and the team scores point. Otherwise the card is removed from the game, the identity is made public, and the team loses one of the 3 life tokens. If the team out of life tokens before the end of the game, all points collected far are lost and the game finishes immediately. These rules result in a maximum score of 5 x 5 = 25 points in any game, which corresponds all five fireworks being completed with five cards per firework.
To ensure reproducibility and comparability of our results we use the Hanabi Learning Environment (HLE) (Bard et al. 2019) for all experimentation. For further details regarding Hanabi and the self-play part of the Hanabi challenge please see Bard et al. (2019)
5.3 ARCHITECTURE AND COMPUTATION REQUIREMENTS
We borrow some ideas and insights from prior distributed Q-learning methods while bring extensions to MARL as well innovations to improve throughput and efficiency. Following Horgan al. (2018) and Kapturowski et al. (2019), we use distributed prioritized replay buffer shared by N asynchronous actors and centralized trainer that samples mini-batches from the replay buffer to update the model. In each actor thread, we run K environments sequentially and batch their observations together. The observation batch is then fed into actor that utilize GPU to compute a batch of actions. All asynchronous actors share one GPU and the trainer uses another GPU for gradient computation and model updates. This is different from prior works which run single actor and single environment in each thread on CPU. Our method enables us to run very large number of simulations with moderate computation resources. In all Hanabi experiments. we run N 80 actor threads with K = 80 environments in each thread on single machine with 40 CPU cores and GPUs. Without this architectural improvement, may require at least few hundred CPU cores to run 6400 Hanabi environments, in which case neural network agents and simulations have to be distributed across multiple machines, greatly reducing the reproducibility and accessibility of such research. Please refer to Appendix A for implementation details and pyrer-parameters.
7

---

Published as a conference paper at ICLR 2020
6 RESULTS
10.0
6.1 MATRIX GAME
As we can see in Figure 2 even in our simple ma- trix game the greedy action input makes a drastic difference. With an average reward of around 9.5 points, tabular IQI. does well in this task, matching the BAD results from Foerster et al. (2019). How- ever, just by adding the greedy action as an addi- tional input, obtain average performance of 9.97 + 0.02. Results are averaged over 100 seeds, and shading is s.e.m. The code is available here: www. blt. 1y/2mBJLyk
9.0
8.5
IQL
8.0
SAD
BAD final
0
2000
4000
6000
8000 10000
Epoch
6.2 HANABI
Figure 2: Results for the matrix game.
As shown in Table our findings from the matrix game are for the most part confirmed on the challenging Hanabi benchmark. To illustrate the contributions of the different components, we compare average scores and win rates across 13 independent training runs of SAD and three different options: IQL is simply the recurrent DQN agent with parameter sharing. VDN is the same agent but also learns joint Q-function and finally SAD & Aux'lask is the SAD agent with the auxiliary task.
While we find that SAD significantly outperforms our baselines (IQL and VDN) for 2, 4 and 5 players in terms of average score and/or win rate, there is significant difference for 3 players, where VDN matches the performance of SAD.
Interestingly, the auxiliary task only significantly helps the 2-player performance, where substan- tially boosts the average score and win rate. In contrast, drastically hurts performance for 3-5 players, which opens an interesting avenue for future work.
For completeness we have included training curves showing average scores and s.c.m. across all training runs for all numbers of players for our methods and ablations in Appendix B We find that for 5 players the auxiliary task drastically reduces the variance of SAD and intermittently leads to higher performance during training but ultimately results in lower final performance. We can also clearly see that despite 72 hours of training and billions of samples consumed, the performance has not plateaued for 3-5 players, pointing to an obvious avenue for further improvements.
The original numbers in the Hanabi challenge and BAD used population based training (Jaderberg al. 2018), effectively reporting maximum performance across a large number of different runs. Therefore, for reproducibility purposes, report evaluations of the best model from our various training runs for each method Table 2]
As shown, under this reporting we establish new SOTA for learning methods on the self-play part of the Hanabi challenge for 2-5 players, with the most drastic improvements being achieved for 3-5 players. In particular, agent on average score, even though both of them used population based training and require more compute. We note that while we follow the counting convention proposed by the challenge paper, BAD was optimized for a different counting scheme, in which agents keep their scores when they run out of lives. This may explain the higher win rate (58.6%) of BAD combined with relatively low mean score, which is exceeded even by our baseline methods. Once again, only the performance for 2-player is significantly improved by the auxiliary task and the 3-player setting is an outlier in the sense that SAD does not improve the best performance compared to VDN,
beat both the ACHA agent from [ardd et al. (2019) and the BAD
CONCLUSION AND FUTURE WORK
In this paper we presented the Simplified Action Decoder (SAD), a novel deep multi-agent RL algorithm that allows agents to learn communication protocols in settings where no cheap-talk channel is available. On the challenging benchmark Hanabi our work substantially improves the SOTA for RL method for all numbers of players. For two players SAD establishes new high-score
8

---

Published as a conference paper at ICLR 2020
Agent
2 Players
3 Players
4 Players
5 Players
IQL
23.77 + 0.04
23.02 + 0.10
21.99 = 0.09
20.60 + 0.11
(Baseline)
43.88 1.21 %
26.16 + 2.01
10.15 + 0.86 %
2.27 = 0.32
VDN
23.83 + 0,03
23.71 + 0.06
23.03 + 0.15
21.18 + 0.12
(Baseline)
44.97 1.28 %
41.16 + 1.27
23.57 = 2.20 %
2.26 0.32
SAD
23.87 + 0.03
23.69 0.05
23.27 + 0.16
22.06 0.23
47.90 + 1.10 %
41.12 + 1.10 %
29.38 + 2.63 %
7.22 + 1.29 %
SAD AuxTask
24.02 + 0.01
23.56 + 0.07
22.78 + 0.10
21.47 + 0.08
54.38 + 0.41 %
40.77 + 1.35 %
20.80 + 1.65 %
2.92 + 0.40
Table 1: Mean performance of methods and baselines on Hanabi. We take the final models of 13 independent runs, Le. 13 models per algorithm per player setting. Each model evaluated on 100K games. Mean and s.e.m over the mean scores of the 13 models are shown in the table. The second row of each section is the win rate.
Agent
2 Players
3 Players
4 Players
5 Players
Rainbow
20.64 + 0.03
18.71 +0.01
18.00 + 0.17
15.26 + 0.18
(Brrd et al.
2019
2.5%
0.2%
0%
0%
ACHA
22.73 + 0.12 15.1%
20.24 + 0.15
21.57 + 0.12
16.80 + 0.13
Bardd et
2019
1.1%
2.4%
0%
BAD
23.92 = 0.01 58.56%
Foerster al. 2019
IQL
23.97 + 0.01 40.25%
23.69 + 0.01 19.39%
22.76 + 0.01 4.93%
21.29 + 0.01
(Baseline) 50.47%
VDN
23.96 + 0.01
23.99 + 0.01
23.79 + 0.00 38.86%
21.80 + 0.01 4.98%
(Baseline)
50.27%
50.37%
SAD
24.01 +: 0.01
23.93
0.01 48.05%
23.81
0.01
23.01
0.01
52.39%
41.45%
13.93%
SAD & AuxTask
24.08 + 0.01
23.81 + 0.01
23.47 + 0.01
22.25 + 0.01
56.09%
49.74%
33.87%
7.33%
Table 2:
Comparison between the previous SOTA learning methods and ours. We take the best model of 13 runs for each of our methods and baselines. Each model is evaluated on 100K games with different seeds. Mean and s.e.m over the 100K games are shown in the table. The s.e.m. is less than 0.01 for most models. Bold numbers are the best results achieved with learning algorithms. The second row of each section is the win rate.
across any method. Furthermore we accomplish all of this with method that is both simpler and requires less compute than previous advances. While these are encouraging steps, there clearly more work to do. In particular, there remains large performance gap between the numbers achieved by SAD and the known performance of hat-coding strategies (Wu) 2018) for 3-5 players. One possible reason is that SAD does not undertake any explicit exploration in the space of possible conventions. Another promising route for future work is to integrate search with RL, since this has produced SOTA results in number of different domains including Poker, Go and backgammon.

---

Published as a conference paper at ICLR 2020
REFERENCES
Chris I. Baker, Julian Jara-Fitinger, Rebecca Saxe, and Joshua B Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4): 0064, 2017.
Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, lain Dunning, Shibl Mourad, Hugo Larochelle, Mare G. Bellemare, and Michael Bowling. The Hanabi Challenge: A New Frontier for AI Research. arXiv: /902: /902.00506 [cs, stat], February 2019. org/abs 902. 00506
arXiv: 1902.00506.
Noam Brown and Tuomas Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, pp. eaao1733, 2017.
Noam Brown and Tuomas Sandholm. Superhuman AI for multiplayer poker. Science, pp. eaay2400, 2019.
Noam Brown, Tuomas Sandholm, and Brandon Amos. Depth-Limited Solving for Imperfect- Information Games. pp. 14.
Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep Blue. Artificial intelligence, 134 (1-2):57-83. 83, 2002.
Rodrigo Canaan, Julian Togelius, Andy Nealen, and Stefan Menzel. Diverse agents for ad-hoc cooperation in hanabi. arXiv preprint arXiv: (eti::1900:00 90990000388 2019.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent reinforcement learning. International Conference on Machine Learning, pp. 1942-1951, 2019.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Jakob N. Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning. 0rXiv:1811.01455 [cs], November 2018b. URL httt:://aax:
org//bs//8111 01458
arXiv: 1811.01458.
James Goodman. Re-determinizing information set monte carlo tree search in hanabi. arXiv preprint arXiv:1902.06075, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory, Neural Comput., 978) 1997.9.8.1735 htthttp::/
1780, November 1997. ISSN 0899-7667, doi:
/ecooocc.ooccccooooooooooeeooo.c doi. 10. 1 62 neco 1 1999.9..8.. 9.8.1735
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. CoRR, abs/1803.00033, 2018. URL
00933
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint a61iv: .05397, 2016.
Max Jaderberg, Wojeiech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Cas- taneda, Charles Beattie, Neil C. Rabinowitz. Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green. Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population- based deep reinforcement learning.
[cs, stat]. July 2018. doi: /11.///cincc..aauueoooo URI/tttp:: http://///awa..oorr///////////// 01281
arXiv: 1807.01281.
10

---

Published as a conference paper at ICLR 2020
Steven Kapturowski, Georg Ostrovski, John Quan, and Will Dabney, RECURRENT EXPERIENCE REPLAY IN DISTRIBUTED REINFORCEMENT LEARNING. pp. 1-19, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Claire Liang. Julia Proft, Erik Andersen, and Ross A Knepper. Implicit communication of actionable information in human-ai teams. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 95. ACM, 2019.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiivv environments. Advances in Neural Information Processing Systems, pp. 6379-667788 6390, 2017.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, al. Learning to navigate in complex environments. arXiv preprint 1rXiv:16/1.036// 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Na- lure, 51877540::59995333, 02 2015. UtLhttp///dx doi. org 10. 0088 http//aturrre1.e0
Matej Moravěík, Martin Schmid, Neil Burch, Viliam Lisy, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508-51 2017.
Ashutosh Nayyar, Aditya Mahajan, and Demosthenis Teneketzis. Decentralized stochastic control with partial history sharing: common information approach. IEEE Transactions on Automatic Control, 58(7): 164))164441658, 2013.
Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multi- agent systems: a review of challenges, solutions and applications. arXiv preprint 1rXiv:1812.11799 2018.
Arthur Hanabi. httmaa//eeppe//aattta....ooo///p
2019.
Frans A Olichock. Decentralized pomdps. In Reinforcement Learning, pp. 471-503, Springer, 2012.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. CoRR, abs/1883.11443 2018. URL
1803. 11485
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay. arXiv e-prints, art. arXiv::511 10:::1111.9999 1.09552, Nov 2015.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schriuwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. Nature, 529(7587):484 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1144,114, 2018.
11

---

Published as a conference paper at ICLR 2020
Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. Learning Multiagent Communica- tion with Backpropagation. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Gamett (eds.). Advances in Neural Information Processing Systems 29, Pp. 2244 2252. Curran Associates, Inc., 2016. URL 6398 - wtt.://ww....t.oooooaraaarrrrra.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zambaldi. Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomppsitionn networks for cooperative multi-agent learning. CoRR, URL /tRthttpp////xi...oom org abs////s/////
2017.
05296
Richard S. Sutton. Learning predict by the methods of temporal differences. Mach. Learn., URL https:
3(1):9-44, August 1988. ISSN 0885-6125, doi: 10.1023/A:1022633314799
7/doi.orgi 1023 1000:oo::/1..002::::11100 10013333111122233311177
Ming Tan. Multi-agent reinforcement learning: Independent cooperative agents. In Proceedings of the tenth international conference on machine learming, pp. 330-337, 1993.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. CoRR,
2015. URL U.t.htppp//aaaivr.rrr//////bb...
Ziyu Wang, Nando de Freitas, and Mare Lanctot. Ducling hucting.naawoookkaccccicicscsoon for reinfore ment learning. CoRR, https///s///gghhooooo///a//a/aaa
2015. httpg////w/g///a/////a///////aa/
Jeff Wu. Hanabi simulation in rust. 2018.
12

---

Published as a conference paper at ICLR 2020
A NETWORK ARCHITECTURE AND HYPER-PAMAMETERS FOR HANABI
Our Hanabi agent uses dueling network architecture (Wang et al. 2013). The main body of the network consists of fully connected layer of 512 units and 2 LSTM (Hochreiter & Schmidhuberi 1997) layers of 512 units, followed by two output heads for value and advantages respectively. The same network configuration is used across all Hanabi experiments. We take the default featurization of HLE and replace the card knowledge section with the V0-Belief proposed by Foerster et al. (2019). The maximum length of an episode is capped at 80 steps and the entire episode stored in the replay buffer as one training sample. This avoids the "slate hidden states" problem described in Kapturowski et al. (219)) because we can simply initialize the hidden states of L.STM as zero during training. For exploration and experience prioritization, we follow the simple strategy as in Horgan et al. (2018) and Kapturowski et al. (2019). Each actor exccutes an e-greedy policy where = x= for i E NN. N - }} but with a smaller = 0.1 and a = 7. For simplicity, all players of game use the same epsilon. The per time-step priority 6 is the TD error and per episode priority is computed following 6, = max, ó + (1 - - where r) = 0.9. Priority exponent is set to 0.9 and importance sampling exponent is set to 0.6. We use n-step return (Sutton 1988) and double Q-learning (van Hasselt et al., (013) for target computation during training. The discount factor y is set to 0.999. The network is updated using Adam optimizer (Kingma & Ba (0144) with learning rate Ir = 6.25 x 10-5 and every 10 updates and target network is synchronized with online network every 2500 updates. These hyper-panne hyper-parameters are fixed across all experiments.
1.5 x 10-5, Trainer sends its network weights to all actors
In the baseline, we use Independent Q-Learning where each player estimates the Q value and selects action independently at each time-step. Note that all players need to operate on the observations in order to update their recurrent hidden states while only the current player has non- non-trivial legal moves and other players can only select 'pass'. Each player then writes its own version of the episode into the prioritized replay buffer and they are sampled independently during training. The prioritized replay buffer contains 2 (131072) episodes. We warm up the replay buffer with 10,000 episodes before training starts. Batch size during training is 128 for games of different numbers of players.
As mentioned in Section 4 the SAD agent is built on top of joint Q-function where the Q value is the sum of the individual Q value of all players given their own actions. One episode produces only one training sample with an extra dimension for the number of players. The replay buffer size is reduced to 21 for 2-player and 3-player games and 2 for 4-player and 5-player games. The batch sizes for 2-, 3-, 4- 5-players are 64, 43, 32, 26 respectively to account for the fact that each sample contains more data.
Auxiliary task be added to the agent help it decode the greedy action more effectively. In Hanabi, the natural choice is the predict the card of player's 's own hand. In our experiments, the auxiliary task to predict the status of card, which can be playable, discardable, or unknown. The loss is the average cross entropy loss per card and is simply added to the TD-error of reinforcement learning during training.
13

---

Published as a conference paper at ICLR 2020
B LEARNING CURVES FOR HANABI
Figure shows learning curves of different algorithms averaged over 13 seeds per algorithm per player setting. Shading is error of the mean.
24
24
21
21.
VON
VON
SAD
SAD
SAD+-Aux
SAD+AAu
18.0
0.2
0.4
0.6
0.8
1.0
1.4
18.0
0.2
0.4
0.6
0.8
1.0
1.2
Epoch(Millon)
Epoch/Millon)
(a) 2-Player
(b) 3-Player
24
22
21
IQL
19
IQL
VDN
SAD
SAD
SAD+Aux
0.0
0.2
0.4
0.6
0.0
1.0
1.2
18.
0.2
0.4
0.6
0.8
1.0
1.2
EpochiMilllo)
EpochiMillnn)
(c) 4-Player
(d) S-Player
Figure 3: Learning Curves
14


Reviews:
[{"review_id": "SygklZ5DYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a novel exploitation of the centralized training for decentralized execution regime for Dec-POMDPs with publicly/commonly known actions.  In particular, the paper augments independent value-based reinforcement learning by allowing each agent to announce the action it would have taken, had it acted greedily. This relaxation is consistent with decentralized execution, at which time agents always act greedily and no counterfactual announcement is required. The paper demonstrates the utility of this trick in Hanabi, where it achieves strong performance when combined with distributed Q-learning and an auxiliary task. The paper is largely well-written.\n\nI think that the trick introduced in this paper is a valuable contribution to the community. As the paper discusses, SAD is simpler and easier to implement than BAD, the algorithm with the most comparable ambitions with respect to scalability. Also, unlike BAD, it is model-free. I think that investigating lightweight alternatives to common/public knowledge based approaches, such as SAD, is an important research direction.\n\nThat being said, I have some issues with the presentation of the content in the paper. \n\n1. Section 4 is problematic. The entirety of the section is based on the assumption that agent a’ observes the Markov state. The paper claims that this assumption is made for simplicity: “where for simplicity we have assumed that agent a’ uses a feed-forward policy and observes the Markov state of the environment.” \n    a. First, this assumption is unmet, with few exceptions. Other agents do \n        NOT in general observe the Markov state. This is a very significant \n        part of why Dec-POMDPs are so difficult. Justifying SAD in a setting \n        that so radically departs from its use case is not informative. \n    b. Second, the claim made by the paper (that the assumption is made \n        for simplicity) is misleading. The assumption is made out of \n        necessity. The cost of neglecting public/common knowledge, as SAD \n        does, is that principled Bayesian reasoning is not possible. It is \n        important that the paper acknowledges this to counterbalance its \n        (valid) criticisms that public/common knowledge based approaches \n        are difficult to scale.\n2. That the experiments are expensive is understandable and a fully complete ablation study is not expected. But the results of these experiments should be presented clearly. \n    a. In both Table 1 and Table 2, presenting the “best seed” is \n        not appropriate. The paper should report the mean or the median. \n        Given that only three seeds could be afforded, it is understandable \n        that the results would have high variance. \n    b. In Table 2, (if I am understanding the paper correctly) it is claimed \n        that the s.e.m. is less than or equal to 0.01 over a set of three seeds \n        for all of the experiments. But looking at Figure 4, this is clearly not \n        the case, especially in the four and five player settings. The intended \n        meaning should be clarified.\n    c. In Table 1, the paper should report the results for the baseline, VDN, \n        VDN with greedy input, and SAD separately, as is done in Table 2. \n        These are the main experimental results and merit space in the main \n        body of the paper. Moreover, there are a number of results that \n        deserve written attention.\n        i. It looks the baseline is very competitive with BAD in two player \n           Hanabi. This in itself is a very interesting finding and is worth \n           discussing.\n        ii. Looking at Table 2, it appears that VDN with GreedyInput \n            outperforms SAD in three of the four settings and quite \n            significantly in the five player setting. If this is also the case when \n            results are aggregated over the median or mean, it should be \n            discussed.\n3. The greedy input approach is specific to Dec-POMDPs in which actions are publicly/commonly available. This should at least be mentioned somewhere.\n\nOverall, I think the ideas and experimental findings in the paper are very interesting. However, as outlined above, there are a number of issues. At a high level, I think that the paper is too concerned with 1) justifying greedy input with Bayesian reasoning and 2) promoting state of the art results. The ideas and experimental findings are more than sufficient for strong contribution without these things.\n\nIn its current state, I feel that this work is a rejection. However, its issues are relatively easily amendable:\n1. For each algorithm and each setting, separately report median or mean results over the seeds, with uncertainty.\n2. Reallocate space to section 6.2 for a scientific discussion of the experimental results.\n3. Remove or qualify the arguments made in section 4.\n4. Mention that the greedy input approach is specific to Dec-POMDPs with publicly/commonly available actions.\n\nWere these issues addressed, my opinion would change.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g0gKQlqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents SAD (Simplified Action Decoder), a new method to address an issue in Centralized Training / Decentralized Control regimes: that exploratory actions during training can make those actions less informative for the agents who observe and attempt to learn from it. The method addresses this issue by allowing the agent to select two actions: one (the possibly exploratory action) is applied to the environment and the other (the greedy action) is presented to other agents as part of their observations for the next turn.\n\nThe authors use a distributed recurrent DQN architecture and apply the resulting agent to a toy problem and to the Hanabi Learning Environment. The authors claim that the method offers improvement over the Bayesian Action Decoder (BAD), that has been similarly applied to the same environments, being simpler, more sample efficient  and achieving overall higher scores, which is confirmed by their results: the agent outperforms BAD in 2-player Hanabi (where BAD was previously state-of-the-art) and the best scores out of any learning agent (although not as high as some non-learning agents such as WTFWThat in the 3-5 player versions. \n\nThe paper does not directly address the ad-hoc cooperation aspect of Hanabi, and it is unclear wheter the method could be used as-is for that problem, due to its reliance on centralized training. Nevertheless, the paper represents a relevant improvement to the self-play aspect of the game, and the core insight that the method leverages could conceivably be applied to minimize the noise introduced by exploration in other cooperative CT/DC problems. For this reason, I recommend the paper to be accepted.\n\nTypos/language issues:\n\nIntroduction: “spend vast amounts of time coordinate” -> coordinating\nSection 3.1: “While our method are general” -> methods\n\n\"accomplish a state of the art in Hanabi\"\nI see what you mean but this is a strange phrasing.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Bkl2ONldFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper examines the problem of epsilon-greedy exploration in cooperative multi-agent reinforcement learning. Such exploration makes actions less informative to other agents because of the noise added to greedy optimal actions. The suggested solution is to consider two actions: one action is epsilon-greedy and it is passed to the environment, while another is fully greedy according to the agent’s strategy, and it is shown to another agents. At test time these two actions are the same. This idea is applied to Hanabi game; the self-play regime is examined. The suggested method is claimed to show state-of-art results in 2-5 players Hanabi. \n\nThe Methods Section provides mathematical grounds of using Bayesian reasoning in ToM. It is shown how epsilon-greedy exploration leads to the blurring of the posterior and consequently making beliefs of other agents less informative. Thus, additional fully greedy input is motivated. However, there are still some vague parts in the description of methods used.\n1.\tIn Section 4.3 it is not clear what auxiliary supervised task is added to training. The information from Appendix should be moved to Section 4.3. Furthermore, some details on how training data for this task is collected and model is trained should be added. Also, how does this model affect the whole training procedure? \n2.\tThe description of the Q-learning in the first paragraph of Section 3.2 is too vague. In the first equation for Q(s,u), there is no u_t, u_{t’}, so it’s unclear why they pop out next. In the Bellman equation for Q(s, u) parentheses are omitted.\n\nThe advantage of the paper is that apart from the suggested additional greedy actions, several ingredients of the state-of-art approach (suggested in previous works) are tested separately, such as learning a joint Q-function and auxiliary supervised task. However, considering the results, it seems there is no clear winner for different number of players in Hanabi. From the Table 2, it is not clear whether the additional greedy input or the auxiliary task are beneficial for the best seed results. One of the three competing methods is the method named VDN in Tables 1 and 2. Is it the contribution of the paper or a previous work? If it is the contribution, the difference from the previous work should be clearly stated. If not, then it’s not fully fair to say that the new approach shows state-of-art results.\n\nAlso, is it fair to say that s.e.m. of the best seed is 0.01? Comparing the results from Figure 3 and Table 2, one can see that the order of the winners is changing significantly, so 3 seeds are not enough to reliably estimate the performance of the best seed and s.e.m. should be higher.\n\nAnother comment on the Experiment 6.1: as far as BAD results are mentioned in the text, maybe they could be put on the plot? At least final training score as horizontal line, if the whole training curve is not available.\n\nMinor comments\n1.\tPage 1: Simply observing other agents are doing -> Simply observing what other agents are doing.\n2.\tPage 3: While our method are general, we restrict ourselves to turn based settings – odd phrase.\n3.\tPage 4: A combination these techniques -> A combination of these techniques.\n4.\t\\eps-greedy is written with a hyphen in some places of the paper and somewhere without.\n5.\tSection 5.3: Compute Requirements -> Computation Requirements.\n\nUPD: score updated.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The method presented, the simplified action decoder, is a clever way of addressing the influence of exploratory actions in multi-agent RL. It's shown to enable state of the art performance in Hanabi, an interesting and relatively novel cooperative AI challenge. It seems, however, that the method has wider applicability than that.

All reviewers agree that this is good and interesting work. Reviewer 2 had some issues with the presentation of the results and certain assumptions, but the authors responded so as to alleviate any concerns.

This paper should definitely be accepted, if possible as oral.

 




Decision: Accept (Spotlight)