reviews: 
Comments: This paper learns a generalist agent that is able to play any-card variants of Hanabi by casting the game to a text based game and utilize model architectures from language modeling to learn such policy with Q-learning
Strengths: 1. The idea of casting Hanabi as a text game for the purpose of learning a generalist agent is interesting. Language as a common interface for variants of games, or completely different games, is interesting, especially when the game primarily requires reasoning, which is the case in Hanabi.

2. If I understand correctly, the R3D2 agent is able to learn consistent policies, i.e. achieving high intra-AXP score and small gap between intra-AXP and self-play, without any additional assumptions that prior methods use. This is a cool result and probably the benefit of using language as observation and action? Although I am not sure exactly why this model is able to do that. See my question below.
Weaknesses: 1. Although this paper does a good job showing that we can train a single generalist agent for any-player Hanabi, which is an interesting contribution, one downside is that the paper is not able to have convincing results of the benefit of having a “generalist” agent, i.e. the benefit of going from R3D2-S to R3D2-M. Apart from the fact that the generalist agent does better in Figure 6, which it should because the single-variant agents are not trained on other variants of Hanabi, I only found this single comment on the benefit of R3D2-M:
> Interestingly, R3D2-M learns decent strategies for all settings, despite receiving the same training budget as single-setting algorithms. This shows that knowledge from one setting is useful for other ones, and that the textual representation allows for effective transfer of kenowledge across settings.

However, I don’t find this to be convincing. If the generalist agent is given the same amount of data as the sum of the 4 single-variant agents, will it outperform the single agents? I think this is crucial for making such a claim and I do believe that would make this paper stronger to the general audience beyond MARL.

\
2. The high inter-AXP results are not super exciting because:
* IQL is known to produce strange conventions that is hard to coordinate with;
* OP has significantly higher inter-AXP score with IQL than any other methods, indicating that the reproduced OP is probably as brittle as the IQL agents, which is totally possible as some OP implementation uses tricks such as hiding the last action from the observation to regularize more;
* OBL coordinates badly with the OP agents, indicating that they are very different;

With the arguments above, a high inter-AXP score here would just translate to better coordination with  IQL/OP than OBL, which is a useful result but not necessarily an important result because eventually we want policies that coordinates better with human-like policies (OBL) rather than inhuman policies (IQL). Having a human study as in [1], a paper in a comparable conference (ICML) on Hanabi that this paper cited, would greatly address this.

[1] Language Instructed RL for Human-AI Coordination.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3