Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper presents SAD (Simplified Action Decoder), a new method to address an issue in Centralized Training / Decentralized Control regimes: that exploratory actions during training can make those actions less informative for the agents who observe and attempt to learn from it. The method addresses this issue by allowing the agent to select two actions: one (the possibly exploratory action) is applied to the environment and the other (the greedy action) is presented to other agents as part of their observations for the next turn.

The authors use a distributed recurrent DQN architecture and apply the resulting agent to a toy problem and to the Hanabi Learning Environment. The authors claim that the method offers improvement over the Bayesian Action Decoder (BAD), that has been similarly applied to the same environments, being simpler, more sample efficient  and achieving overall higher scores, which is confirmed by their results: the agent outperforms BAD in 2-player Hanabi (where BAD was previously state-of-the-art) and the best scores out of any learning agent (although not as high as some non-learning agents such as WTFWThat in the 3-5 player versions. 

The paper does not directly address the ad-hoc cooperation aspect of Hanabi, and it is unclear wheter the method could be used as-is for that problem, due to its reliance on centralized training. Nevertheless, the paper represents a relevant improvement to the self-play aspect of the game, and the core insight that the method leverages could conceivably be applied to minimize the noise introduced by exploration in other cooperative CT/DC problems. For this reason, I recommend the paper to be accepted.

Typos/language issues:

Introduction: “spend vast amounts of time coordinate” -> coordinating
Section 3.1: “While our method are general” -> methods

"accomplish a state of the art in Hanabi"
I see what you mean but this is a strange phrasing.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper examines the problem of epsilon-greedy exploration in cooperative multi-agent reinforcement learning. Such exploration makes actions less informative to other agents because of the noise added to greedy optimal actions. The suggested solution is to consider two actions: one action is epsilon-greedy and it is passed to the environment, while another is fully greedy according to the agent’s strategy, and it is shown to another agents. At test time these two actions are the same. This idea is applied to Hanabi game; the self-play regime is examined. The suggested method is claimed to show state-of-art results in 2-5 players Hanabi. 

The Methods Section provides mathematical grounds of using Bayesian reasoning in ToM. It is shown how epsilon-greedy exploration leads to the blurring of the posterior and consequently making beliefs of other agents less informative. Thus, additional fully greedy input is motivated. However, there are still some vague parts in the description of methods used.
1.	In Section 4.3 it is not clear what auxiliary supervised task is added to training. The information from Appendix should be moved to Section 4.3. Furthermore, some details on how training data for this task is collected and model is trained should be added. Also, how does this model affect the whole training procedure? 
2.	The description of the Q-learning in the first paragraph of Section 3.2 is too vague. In the first equation for Q(s,u), there is no u_t, u_{t’}, so it’s unclear why they pop out next. In the Bellman equation for Q(s, u) parentheses are omitted.

The advantage of the paper is that apart from the suggested additional greedy actions, several ingredients of the state-of-art approach (suggested in previous works) are tested separately, such as learning a joint Q-function and auxiliary supervised task. However, considering the results, it seems there is no clear winner for different number of players in Hanabi. From the Table 2, it is not clear whether the additional greedy input or the auxiliary task are beneficial for the best seed results. One of the three competing methods is the method named VDN in Tables 1 and 2. Is it the contribution of the paper or a previous work? If it is the contribution, the difference from the previous work should be clearly stated. If not, then it’s not fully fair to say that the new approach shows state-of-art results.

Also, is it fair to say that s.e.m. of the best seed is 0.01? Comparing the results from Figure 3 and Table 2, one can see that the order of the winners is changing significantly, so 3 seeds are not enough to reliably estimate the performance of the best seed and s.e.m. should be higher.

Another comment on the Experiment 6.1: as far as BAD results are mentioned in the text, maybe they could be put on the plot? At least final training score as horizontal line, if the whole training curve is not available.

Minor comments
1.	Page 1: Simply observing other agents are doing -> Simply observing what other agents are doing.
2.	Page 3: While our method are general, we restrict ourselves to turn based settings – odd phrase.
3.	Page 4: A combination these techniques -> A combination of these techniques.
4.	\eps-greedy is written with a hyphen in some places of the paper and somewhere without.
5.	Section 5.3: Compute Requirements -> Computation Requirements.

UPD: score updated.

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper introduces a novel exploitation of the centralized training for decentralized execution regime for Dec-POMDPs with publicly/commonly known actions.  In particular, the paper augments independent value-based reinforcement learning by allowing each agent to announce the action it would have taken, had it acted greedily. This relaxation is consistent with decentralized execution, at which time agents always act greedily and no counterfactual announcement is required. The paper demonstrates the utility of this trick in Hanabi, where it achieves strong performance when combined with distributed Q-learning and an auxiliary task. The paper is largely well-written.

I think that the trick introduced in this paper is a valuable contribution to the community. As the paper discusses, SAD is simpler and easier to implement than BAD, the algorithm with the most comparable ambitions with respect to scalability. Also, unlike BAD, it is model-free. I think that investigating lightweight alternatives to common/public knowledge based approaches, such as SAD, is an important research direction.

That being said, I have some issues with the presentation of the content in the paper. 

1. Section 4 is problematic. The entirety of the section is based on the assumption that agent a’ observes the Markov state. The paper claims that this assumption is made for simplicity: “where for simplicity we have assumed that agent a’ uses a feed-forward policy and observes the Markov state of the environment.” 
    a. First, this assumption is unmet, with few exceptions. Other agents do 
        NOT in general observe the Markov state. This is a very significant 
        part of why Dec-POMDPs are so difficult. Justifying SAD in a setting 
        that so radically departs from its use case is not informative. 
    b. Second, the claim made by the paper (that the assumption is made 
        for simplicity) is misleading. The assumption is made out of 
        necessity. The cost of neglecting public/common knowledge, as SAD 
        does, is that principled Bayesian reasoning is not possible. It is 
        important that the paper acknowledges this to counterbalance its 
        (valid) criticisms that public/common knowledge based approaches 
        are difficult to scale.
2. That the experiments are expensive is understandable and a fully complete ablation study is not expected. But the results of these experiments should be presented clearly. 
    a. In both Table 1 and Table 2, presenting the “best seed” is 
        not appropriate. The paper should report the mean or the median. 
        Given that only three seeds could be afforded, it is understandable 
        that the results would have high variance. 
    b. In Table 2, (if I am understanding the paper correctly) it is claimed 
        that the s.e.m. is less than or equal to 0.01 over a set of three seeds 
        for all of the experiments. But looking at Figure 4, this is clearly not 
        the case, especially in the four and five player settings. The intended 
        meaning should be clarified.
    c. In Table 1, the paper should report the results for the baseline, VDN, 
        VDN with greedy input, and SAD separately, as is done in Table 2. 
        These are the main experimental results and merit space in the main 
        body of the paper. Moreover, there are a number of results that 
        deserve written attention.
        i. It looks the baseline is very competitive with BAD in two player 
           Hanabi. This in itself is a very interesting finding and is worth 
           discussing.
        ii. Looking at Table 2, it appears that VDN with GreedyInput 
            outperforms SAD in three of the four settings and quite 
            significantly in the five player setting. If this is also the case when 
            results are aggregated over the median or mean, it should be 
            discussed.
3. The greedy input approach is specific to Dec-POMDPs in which actions are publicly/commonly available. This should at least be mentioned somewhere.

Overall, I think the ideas and experimental findings in the paper are very interesting. However, as outlined above, there are a number of issues. At a high level, I think that the paper is too concerned with 1) justifying greedy input with Bayesian reasoning and 2) promoting state of the art results. The ideas and experimental findings are more than sufficient for strong contribution without these things.

In its current state, I feel that this work is a rejection. However, its issues are relatively easily amendable:
1. For each algorithm and each setting, separately report median or mean results over the seeds, with uncertainty.
2. Reallocate space to section 6.2 for a scientific discussion of the experimental results.
3. Remove or qualify the arguments made in section 4.
4. Mention that the greedy input approach is specific to Dec-POMDPs with publicly/commonly available actions.

Were these issues addressed, my opinion would change.
review_title: Official Blind Review #2

Meta-review: The method presented, the simplified action decoder, is a clever way of addressing the influence of exploratory actions in multi-agent RL. It's shown to enable state of the art performance in Hanabi, an interesting and relatively novel cooperative AI challenge. It seems, however, that the method has wider applicability than that.

All reviewers agree that this is good and interesting work. Reviewer 2 had some issues with the presentation of the results and certain assumptions, but the authors responded so as to alleviate any concerns.

This paper should definitely be accepted, if possible as oral.

 




Decision: Accept (Spotlight)