reviews: 
Comments: The paper considers the problem of generalization and coordination in multi-agent reinforcement learning (MARL) within the game of Hanabi, focusing on enabling agents to adapt across different game settings and learn to collaborate with unfamiliar partners. To tackle this, the paper makes two primary contributions: first, it reformulates the Hanabi environment using a text-based representation, which enables use of language model to process state and action space, thereby providing a consistent observation and action space across varying player configurations; second, it extends the distributed training regimen of Recurrent Replay Distributed DQN (R2D2) by combing language model with Deep Recurrent Relevance Q-network,  creating what they call Recurrent Replay Relevance Distributed DQN (R3D2), an agent architecture that can process dynamic observation and action spaces in a multi-agent environment. Experiments assess R3D2’s ability to coordinate with both familiar and unfamiliar agents in zero-shot settings and to transfer strategies across different player configurations, evaluating performance across various player numbers and agent types. The authors propose that this approach could support further exploration of generalization in MARL for complex cooperative games.
Strengths: - The paper considers the important problem of enabling agents to collaborate across different player configurations and game settings, a key challenge for advancing multi-agent reinforcement learning. This focus addresses a critical gap in MARL research, as agents must be able to generalize and coordinate flexibly in dynamic, multi-agent environments, an area relevant to both academic research and real-world applications.

- The key contribution of the paper can be seen as creating a text-based version of Hanabi game. Consequently, the exploration in Section 5 of how current language models perform in the Hanabi environment is a valuable addition, as it highlights the capabilities and limitations of large language models (LLMs) in multi-agent coordination tasks. This analysis opens pathways for further investigation into LLM limitations, potentially contributing to future improvements in language model-based reinforcement learning.

- Within the Hanabi environment, the results indicate that the proposed R3D2 approach achieves effective transfer of strategies across different game settings, demonstrating the agent’s adaptability to variable player configurations.
Weaknesses: - The authors make several broad claims in the introduction about multi-agent reinforcement learning (MARL) and the adaptability of agents, but these claims lack supporting evidence or citations. For example, in the following statements: “Artificial agents should do the same  for successful collaboration of artificial and hybrid systems” (line 34-35), “… players are required to infer the beliefs and intentions of their counterparts through theory of mind reasoning” (line 42-43), and “…resulting in misunderstandings and thus a drop in cooperation …” (line 48-49), the authors do not provide empirical data or references to substantiate these claims, limiting the rigor of their introductory motivation.

- The paper does not sufficiently establish why Hanabi is an appropriate testbed for studying adaptability to new settings and collaborators, nor does it explain how findings in Hanabi might translate to real-world collaborative tasks. The link between Hanabi’s in-game interactions and broader applications remains unclear, and this omission leaves open questions about the broader relevance and applicability of the research.

- While the authors cite work by Hu and Sadigh as justification for their choice of Hanabi and as a baseline for comparison, this prior research does not fully support the tasks explored here. A central focus of Hu and Sadigh’s work was on human-AI coordination, which this paper does not directly address. This gap makes the comparison less relevant, as the primary objectives differ significantly.

- The authors critique Hu and Sadigh's approach for requiring an explicit specification of the expected behavior of the learning agent. However, the proposed R3D2 method also requires specific environment settings, which could present a similar or greater burden. The need to configure a consistent environment specification might, in some ways, be less useful than specifying interaction strategies, as discussed in later sections.

- A significant oversight in this paper is the lack of reference to the extensive body of literature on unsupervised environment design in both single-agent and multi-agent settings [1,2]. This research area directly addresses the problem of adaptability and generalization in agent training, which the authors claim to investigate. For the paper’s contributions to be meaningful within the field, it is essential for the authors to engage with this related work and to include comparative empirical evaluations.

- While creating a text-based version of Hanabi contributes to the approach, it is unclear how this addresses the problem of adaptability that the authors set out to solve. Converting the environment to a text-based form, and leveraging symmetries within Hanabi, simplifies many of the original environment’s complexities. As a result, adaptability may not be fully tested here since the text-based version removes many elements that would typically require generalization, making it a fundamentally easier learning problem.
- The approach also necessitates adapting the replay buffer to handle a variable number of players, which could be a limiting factor. Although this modification works here because Hanabi caps players at five, it may not scale effectively for environments requiring more flexibility or involving larger numbers of agents.

- The empirical evaluation of collaboration with other agents is limited to two-player settings, as the other agents require training specific to each number of players. This narrow scope restricts the evaluation, as it remains unclear how these methods would perform in settings with more players. Additionally, because the baseline agents are not trained using a text-based representation, this could lead to an unfair comparison between approaches.

- The authors indicate that all agents were trained using identical hyperparameters (line 374-375), despite the agents having different architectures and representations. Given these differences, it is crucial to tune hyperparameters individually for each agent to ensure a fair comparison, as uniform parameters may disadvantage some agents more than others.

- The explanation given for the relatively low performance of OP (line 410) lacks clarity, and the observed performance of OBL seems promising in comparison to R3D2. However, the current experimental setup does not provide sufficient data to substantiate why R3D2 should be chosen over OBL or how these results might generalize across settings.

- The passage on lines 422-425 presents an unclear and convoluted explanation of the comparison between methods, and the reported marginal improvements in R3D2’s performance relative to other settings are not fully justified within the results.

- The insights gained from the results presented in Figure 6 are unclear. Given that R3D2 was trained across all game environments, it is unsurprising that it performs adequately across settings. Without a clearer interpretation, the results lack novelty, as the outcomes align with the fact that R3D2 has been exposed to all environments during training.

[1] Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, Dennis et. al. NeurIPS 2020

[2] MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning, Samvelyan et. al. ICLR 2023
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 1