A Generalist Hanabi Agent
Authors: Arjun V Sudhakar
Hadi Nekoei
Mathieu Reymond
Miao Liu
Janarthanan Rajendran
Sarath Chandar
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Agent Reinforcement Learning (MARL)
Cooperative game
Multi Agent Text-based game
Primary area: reinforcement learning
TL;DR: First MARL agent that can play different Hanabi settings and cooperate with other algorithmic agents

Abstract: Traditional multi-agent reinforcement learning (MARL) systems can develop cooperative strategies through repeated interactions. However, these systems are unable to perform well on any other setting than the one they have been trained on, and struggle to successfully cooperate with unfamiliar collaborators. This is particularly visible in the Hanabi benchmark, a popular 2-to-5 player cooperative card-game which requires complex reasoning and precise assistance to other agents. Current MARL agents for Hanabi can only learn one specific game-setting (e.g., 2-player games), and play with the same algorithmic agents. This is in stark contrast to humans, who can quickly adjust their strategies to work with unfamiliar partners or situations. In this paper, we introduce Recurrent Replay Relevance Distributed DQN (R3D2), a generalist agent for Hanabi, designed to overcome these limitations. We reformulate the task using text, as language has been shown to improve transfer. We then propose a distributed MARL algorithm that copes with the resulting dynamic observation- and action-space. In doing so, our agent is the first that can play all game settings concurrently, and extend strategies learned from one setting to other ones. As a consequence, our agent also demonstrates the ability to collaborate with different algorithmic agents ---agents that are themselves unable to do so.

Published as a conference paper at ICLR 2025
A GENERALIST HANABI AGENT
Arjun Vaithilingam Sudhakar*1.2 Hadi Nekoei Mathieu Reymond 1,2,3
Janarthanan Rajendran" Miao Liu Sarath Chandar1.237
Chandar Research Lab Universitéé de Montréal
2 Milaa Quebec Al Institute *Polytechnique Montréal
IBM Research
DDahhousie University
"Canada CIFAR Al Chair
ABSTRACT
Traditional multi-agent reinforcement learning (MARL) systems can develop co- operative strategies through repeated interactions. However, these systems are unable to perform well on any other setting than the one they have been trained on, and struggle to successfully cooperate with unfamiliar collaborators. This is particularly visible in the Hanabi benchmark, a popular 2-to-5 player coopera- tive card-game which requires complex reasoning and precise assistance to other agents, Current MARL agents for Hanabi can only learn one specific game-sctting (e.g., 2-player games), and play with the same algorithmic agents. This is in stark contrast to humans, who can quickly adjust their strategies to work with unfamiliar partners situations. In this paper, introduce Recurrent Replay Relevance Distributed DQN (R3D2), a generalist agent for Hanabi, designed to overcome these limitations. We reformulate the task using text, language has been shown to improve transfer. We then propose a distributed MARL algorithm that copes with the resulting dynamic observation- and action-space. In doing so, our agent is the first that can play all game settings concurrently, and extend strategies learned from one setting to other ones. As a consequence, our agent also demonstrates the ability to collaborate with different algorithmic agents agents that are themselves unable do so. The implementation code available at:
INTRODUCTION
Humans were able to thrive society through their ability to cooperate. Interactions among multi- ple people or agents are essential components of various aspects of our lives, ranging from everyday activities like commuting to work, to the functioning fundamental institutions like governments and economic markets. Through repeated interactions, humans can understand their partners, and learn to reason from their perspective. Crucially, humans can generalize their reasonings towards novel partners, in different situations. Artificial agents should be able to do the same for the suc- cessful collaboration of artificial and hybrid systems (Dafoe et al.) 2020). This is why defining the problem of multi-agent cooperation nicely fits the multi-agent reinforcement learning (MARL) paradigm, as artificial agents learn to collaborate together through repeated interactions, in the same principled manner humans would.
In MARL, the game of Hanabi has emerged as popular benchmark to assess the cooperative abil- ities of learning agents (Bard et al. 2020). Hanabi a partially-observabble card game designed for 2 to 5 players, with approximately 20d unique player hands in the 5-player setting. Progressing in the game requires intricate skills, including long-term planning, precise assistance through clues to other agents, and complex reasoning. Adding to the complexity, players are required to infer the be- liefs and intentions of their counterparts through theory of mind reasoning (Bard et al.) 22020). All of these characteristics are required in real-world multi-agent interactions, making Hanabi challeng- ing and relevant testbed for MARL, Moreover, Hu al. (2020); Hu Sadigh (2023) have shown that agents performing well in Hanabi, particularly in zero- shot coordination, demonstrate improved capabilities in human-AI collaborative scenarios.
'Fqual contribution. Corresponding authors: arjun. vva thi ingam saannnuudhaaaaaa k @mi 1a quebec, n he@mi quebec

---

Published as a conference paper at ICLR 2025
A straightforward way to learn to play is through self (Tan 1993 Tampuu al.) 2017 Foerster et al. (211)). By repeatedly playing with oneself, an agent can learn conventions and effectively play the game. Sadly, these conventions often do not apply to others, resulting in misunderstandings and thus a drop in cooperation capabilities when paired with novel agents (Carroll et al.) 2019). This is of course undesired behavior. An important aspect of assessing an artificial agent's 's performance is thus to evaluate its cooperative abilities when paired with agent it has not trained with, i.c., zero-shot coordination (ZSC). This means agents require to have solid understanding of the game, and need robust strategies that cope with unexpected decisions of their teammates. Moreover, if strategy robust enough, it should provide a solid basis for variations of the task at hand. In Hanabi, for example, the optimal strategy for a 3-player game is different from the one for a 2-player game, even though the rules remain unchanged. However, using a robust 2-player strategy on a 3-player game should still yield solid performance, even if not optimal.
Learning such robust strategies is precisely the goal of this paper. By sacrificing some of the perfor- mance gains that come with learning a highly specialized but inflexible strategy, we design an agent that can not only generalize across different types of partners, but also to different game settings.
A centerpiece of our work lies in the realisation that the representation of the Hanabi environment the agents use to make decisions is highly structured, and thus inflexible. Changing the game setting results in a completely different structure, severely hampering the potential transfer of knowledge from one setting to another. This is the case for both observations of the game's state an abstract encoding of bits- and actions the agent perform one-hot encoding for all action-combinattins. Thus, as a first contribution, we modify the representations of both the observation- and action- spaces of Hanabi to make more suitable for knowledge transfer. For this, we propose use natural language as a backbone for our representation. Language has been shown to be a successful medium for transfer (Radford et al. 2019b] Brown al.] 2020), and using text for observations and actions results in a representation that becomes agnostic to number of partners in play (i.e., the setting of the game). This results in agents that learn on similar data-distributions, regardless of the number of teammates in play.
Next, we propose novel neural network architecture that combines language models (Devlin al,, 2018b) and Deep Recurrent Relevance Q-network (DRRN) (He et al.) 2015) to create agent robust towards the dynamic textual observation- and action-spaces. Integrating this architecture with distributed training regimen results in the Recurrent Replay Relevance Distributed DQN (R3D2) algorithm, our main contribution. What is remarkable is that, even though R3D2 learns the game of Hanabi through self-play, the simple fact of using more abstract game representation and well-suited network architecture results in robust strategies that to not only successfully cooperate with unseen R3D2 agents, but also and perhaps more importantly with completely different algorithmic agents. Moreover, due to their dynamic network architecture, R3D2 agents that have been trained on different player settings are able to collaborate together, even though they have leamed different strategies.
Finally, because of the player-agnostee nature of R3D2, R3D2 agents can change the number of players in a game while they are learning. effectively enabling what we call variable-player learn- ing, a aulti-agent-specific variant of multi-task learning. By training with multiple combinations of number of agents, R3D2 can extend the simpler 2-player strategies to the hardest 5-player setting. In doing so, we have developed the first generalist Hanabi agent. To the best of our knowledge, this is the first time that generalization across game settings has been investigated for Hanabi. We argue this to be an essential component that defines robustness of behavior, and believe that general- ization across game- settensttings is promising research direction to evaluate policy robustness. While we demonstrate our approach on Hanabi, our core technical contributions text-based representa- tion for better transfer, architecture for dynamic action/state spaces, and variable-player learning domain-agnostic advances that could benefit MARI. applications in general.
RELATED WORK
MARL for Hanabi For successful collaboration, MARL agents require specific skills, such as dealing with imperfect information, predicting the intentions of partners, and communicating valu- able information to others. All of these skills are required to play the game of Hanabi, which is why (aard al. (2020) proposed the Hanabi challenge new frontier for AI research. The first
2

---

Published as a conference paper at ICLR 2025
deep RL methods to learn winning strategies for Hanabi use self-play, combined with either a public belief state (Foerster et al. 2019). or by explicitly providing additional information during train- ing (Hu & Foerster, 2019). However, as observed by Huu et al. (020)), self-play agents learn highly specialized conventions that are not transferable to novel partners. They introduce the concept of zero-shot coordination (ZSC), where AI agents need to coordinate with partners they have never seen before, and propose Other-Play for ZSC. Other-Play develops more robust strategies by leveraging the presence of known symmetries in the underlying problem. However, the ZSC performance is evaluated through cross-play, i.e., agents of the same learning algorithm but from independent train- ing runs. While it is an important step towards generalization, cross-play is cheap proxy to ZSC with completely different Al agents or even human players. Multiple works have since proposed ZSC-capable learners. Lupu et al. (02T); )); Nekoei et al. (202T) use population training with diverse policies so agents train with large pool of agents, thus encountering diverse behaviors and creating robust strategies that generalize across the population. Lucas & Allen (2022) use intrinsic rewards as alternative way to promote diverse behaviors. As an alternative to diversity-based approaches, Cui et al. (202T) propose to ground policies using hierarchies of agents, where a level-k agent learns the best-response strategy to level-k 1 agent. Similarly, (Hu et al. (2021b) propose iteratively optimize a policy against the optimal policy of the previous iteration. Through these hierarchies, the agents observe different levels of reasoning and can fall back lower-level reasoning when playing with unknown partners. Both these works focus on the 2-player setting of Hanabi, and are unable to generalize to higher player settings. Finally, Hu & Sadigh (2023) propose a framework where the partner specifies what kind of strategy it expects the learning agent to play. Using pretrained large language models (LL.Ms), a prior policy is generated conditioned on this specification, such that the agent learns strategy that is aligned with this policy. However, this prior policy is only conditioned on other agents' actions and heuristic instructions in contrast to our work where the language model operates on both observations and action avoiding the need to design hand-coded instructions. In all these works, agents need to learn how to play with others, on the same game-setting. In contrast, our R3D2 agent can play on multiple game-settings at once, and play with agents trained on other game-settings.
RL for text-based games Recent successes in RL and natural language processing (NLP) have resulted in a surge of interest for developing RL agents based on text-based games. Examples in- clude story-based games (Hausknecht et al. 2020), adventure games (Yin & May 2019), and many others (Yuan et al. 2018; Murugesan et al., 2020: Wang et al. 2022). In such environments, the agent is presented with textual description of goal (Osborne et al.) (022). These interactive envi- ronments offer challenging and realistic training that requires a solid understanding of the language and the task. Moreover, connecting language with the physical world is critical to solving the task (Bisk et al. 20200] Bender & Koller (020). To address these, researchers have developed several RL-based agents that operate on text (He et al. 2015) Jain et al. 2019: Xu et al. 2020 Yuan et al,, 2018). Language models have been used to propose action candidates Jang al. 202 Yao al.
2020 Singh et al. [022]] Sudhakar et al. 2023). While these environments allow for many different and diverse behaviors and tasks, they focus single-agent learning. For multi-agent systems, Park let al. 2023) investigate generative agents interacting with each other and the world through text. However, the focus of this work is to observe believable individual and emergent social behaviors from these agents, no learning is involved.
Transfer Learning and Generalization in RL Transfer learning and generalization in multi- agent reinforcement learning present unique challenges due to complex agent interactions. While single-agent approaches focus on task-invarian features (Taylor & Stone (009): Finn et al., 2017) and curriculum learning (Bengio et al.] 2009 Narvekar et al., 2020). multi-agent systems face addi- tional challenges in adapting to different interaction patterns (Carbonell & Veloso 2006 Da Silva & Costa, 2019). Recent work has explored environment design, with Team et al. (2021) and Dennis et al. (2020) demonstrating zero-shot generalization through procedurally generated environments, and (Samvelyan et al. (2023) extending this to multi multi-agent coordination. Our work diverges from these approaches by using language to reformulate environments for consigonfignration generaliza-
tion, building on language-based transfer advances (Andreas al. 2017
Lee et al. 2022).
3

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
3.1 THE GAMR OF HANABI
In Hanabi, 2-to-5 players work cooperatively to arrange cards in ascending order (from 1 to 5) for each color (typically five colors: red, blue, green, white, and yellow). Each player is dealt hand of cards, but the twist is that they cannot see their own cards, only their teammates can see them. Players take turns, and on each turn a player must choose to perform one of three possible actions. Either it gives a clue, plays a card, or discards a card. If the player gives a clue, it can provide one piece of information to another player about their hand (e.g.. "These two cards are blue"). Giving a clue costs a hint token, and there are 8 hint tokens available. Alternatively, the player can decide to play a card, hoping can be added to the sequence of cards on the table. If the card correct (i.e., it extends one of the color sequences in ascending order), it is successfully played. If the card is wrong, it is discarded, and the team loses one of its 3 life tokens. Finally, the player can discard card from their hand to gain hint token back. This action is necessary when hint tokens run out, but it comes with the risk of discarding a crucial card. Each correctly arranged card results in 1 point, for maximum of 25 points (all cards of all 5 colors have been arranged correctly). The game ends either when all cards are arranged, all life tokens have been used, or the deck is empty.
3.2 MULTI-AGENT REINFORCEMENT LEARNING
In Hanabi, each player is dealt number of cards, which only the other players can see. As such, Hanabi is a partially observable game: not all information is available to the player to make decision. Each turn, player can decide to either play discard card, or to give clue. Players need to make decisions alone, making this a decentralized game. By playing cards in a specific order, players gain points team. Hanabi is thus fully cooperative. In the MARL literature, such Sertialll-Observabbb Markov Decision Process DeccPOMMD ) (Bernstein et al., 2002 Nair et al. 2003), formally defined as tuple G = (, P. R. Ω2 O,N, 7), with the set of states S, the set of actions A, the transition function P, the reward function R, the set of observations for each agent 2 the observation function O, the number of agents N, and as the discount factor. The game is partially observable, with of ~ O(o i, i, as agent 's observation of the global state, sampled from the (stochastic) observation function O, The game is also fully cooperative, thus agents share the same reward r = R(s, a), conditioned on the joint action a = [a']
setting is modeled as a Decentralized
N and the global state 8. At each timestep i,, all agents are at the state 8t. Each agent has an action-observattee observation history (AOH) 7/ = {ob ab. . and selects action of using stochastic policy of the form T (a' (7). The transition function P ((s' |ss,ar)) at), conditioned on the joint action and the global state, transitions to the next state St+1- Under the joint policy m, we call (s) = E E, (n'r, I & = s] the value, i.e., the expected sum of discounted rewards (or return). The policy that maximizes the value is said to be the optimal policy x² = max, V Closely related to the value is the Q-value (*((s.) a) = E Σ, IY'r,
TT s,
al a).
In single-agen agent RL, Deep Q-Networks (DQN) Mnih et al.) imation of Q" with neural network parametrized by 0. Qo is learned by minimizing (Qo(s.a) (r + y max, Qa (s', , d')))² where Qo is a copy of Qu updated regularly stabilize learning. To extend DQN to the partially observable setting, Hausknecht & Stone (2013) propose Deep Recurrent Q-Network (DRON), which uses recurrent neural networks to estimate Q-values based on the AOH, i.e., QT((T, a) = E (r'rr T. = T, a). This, combined with several modern best practices on top of DON, including double-DQN (Van Hasselt et al.] 2016), a duel- ing network architecture (Wang et al., (016), prioritized experience replay (Schaul et al.) 2016), and a distributed training setup with parallel running environments (Horgan et al. 2018) result in Recurrent Replay Distributed Deep Q-Networks (R2D2) (Kapturowski et al. 2019). Our proposed algorithm, which we explain in section 4 uses R2D2 as its foundation. In MARL, a straightforward strategy is to represent each agent as an independent single-agent learner, e.g., using R2D2. An
2015) learns Qo. an approx-
agent can then be trained through self-play (SP) with copies of itself (Tan) 1993] Tampuu et al. 2017). Our proposed algorithm, R3D22 also uses SP to learn cooperative strategies.
4

---

Published as a conference paper at ICLR 2025
4 METHODOLOGY
We present Recurrent Replay Relevance Distributed DQN (R3D2), a generalist Hanabi agent that adapts to novel partners and game-settings. R3D2 observes and interacts with its environment through natural language, which is known to improve transfer of learned behavior to other tasks. Through its network architecture R3D2 handles dynamic observation- and action-spaces. Combined with distributed training regimen, it allows R3D2 to jointly learn diverse strategies for 2-to-5 player games, resulting in robust, adaptive artificial player.
4.1 HANABI AS A TEXT-BASED GAME
As our first contribution, frame Hanabi text-based game, due to recent successes of using language as medium for transfer learning (Radford et al. 2019b; Brown et al. 2020).
In the original Hanabi environment (Bard ct al. 2020), the observation is represented as a bitstring encoding, i.c., a concatenation of bits representing the observation. To encode the game state as text, a template, of which an example can be seen in Figure [ The template starts by listing the number of life and clue tokens, followed by a listing of the arranged cards, other players' hands, and the knowledge of the player's ' own hand. It also includes the hints given to other players. While the bitstring encoding captures all the information available for optimal gameplay, it greatly changes depending on the setting (2-to-5-player games). In contrast, our text encoding appears intuitive, and requires minimal modifications switching between game settings. Thus, it allows for state-space generalization, i.e., learning policies across different settings becomes easier. For example, adding a new hand to the game results in arbitrary modifications in the original bitstring encoding. In contrast, the text encoding only appends relevant information to the observation.
Secondly, we modify the action-space. There are 3 types of actions: play card, discard a card and give a clue. Each type has a fixed number of concrete actions. player can play or discard any the cards in their hand (e.g.. "I play card 1²).. Or, player can give clue concerning specific color or rank to any other player (c.g., reveal bl ue to player 2"). In the original environment, actions are encoded as one-hot encoding of all the possible cetion-combinations. Instead, we encode the action with a keyword corresponding to the action-type, followed by the type's 's parameter (e.g. reveal blue 2). Clues are the only type of actions affected by change in the number of players. With this encoding, the agent can generalize the behavior for each action-type, and easily extend cluc-actions other player settings. This motivates generalizations to actions not seen during training.
We select this template because it includes all the crucial information contained in the vectorized observation. We also perform ablation studies on different components of the template measure its impact on the agent's capability to predict optimal actions, which we show in Appendix A.9 Using this textual version, we design an agent that takes advantage of this representation to improve generalization across players and settings.
4.2 R3D2 AGENT: HANDLING DYNAMIC STATE AND ACTION SPACE
With the dynamic representation resulting from Hanabi's 's text-based encoding, we propose an agent architecture (shown in Figure 2 that incorporates a (pretrained) language model to encode obser- vations and actions into observation- and action-embeddiings, To do so, we take inspiration from Deep Reinforcement Relevance Network (DRRN) (He et al. 2015).. which propose a neural archi- tecture for deep RL designed to handle action-spaces characterized by natural language. In contrast to R2D2, which outputs a fixed vector of size [A], DRRN encodes the action in a separate action- embedding network. The corresponding Q- Qavauu is then computed the inner product of the state- and action-embeddinng::
Q(x,a) = f(s) g(a),
where f()) is the state embedding and g(a) is the action embedding, In doing so, DRRN is able to encode an arbitrary number of actions. Ma et al. (2022) also showed recently that iieetiio-basse architectures that jointly process featurized representation of observations and actions have better inductive bias for learning intuitive policies. As a downside, |AA separate forward passes need to be performed on g compute all Q-values of specific state 8.
5

---

Published as a conference paper at ICLR 2025
the tobeez avalatle ranaiing
che tokens wvailbbee. life remaining. Fireworks dis- pla:: Red 5. Yellow 4, Green 2. White 4. Blue 4. knowledge about hand: Green Green Unknowa 5,
TRyBERT (48)
Qo
known X. Player hand: Yellow
5. White 4. White Yellow I,
Green 2. Discurd;; Green Red
card
FF
I Yellow 4 Green Red
Ling (42)
Figure 1: The template used for textual observations in Hanabi. It includes all nec- essary information to play the game, including life and clue tokens, visible hands, dis- carded cards, and hints.
Figure 2: An overview of the R3D2 architecture. R3D2 uses separate head for observations and actions. Each head starts with 2 TinyBERT layers to encode the textual representation, followed by LSTM layer encode the previous timesteps. We use elementwise multiplication to combine both embed- dings. This is then split into a separate value and advantage head, before being summed together to obtain Qo (T)
In our case, Qo depends on the AOH, resulting in sequence of embeddings. This sequence is pro- cessed by a Long Short-Term Memory (LSTM) network, followed by a two-layer perceptron (MLP) predict value and advantage estimates. These then combined to estimate Q-values. We train our R3D2 agent via self-play, similar to IQL-basee baselines the literature (Hu & Foerster (219)) 20199) minimize the TD-loss. In section | we demonstrate that using an appropriate representation one can enable self-play to Iearn cooperative behavior with other partners, without relying on complex MARL methods.
The primary trainable components of R3D2 include the language model (LM), LSTM, and MLP. We utilize a two-layer TinyBERT (Tiao et al., 2020) as the LM due its optimal balance between perfor- mance and inference time, which is crucial in RL settings with frequent interactions. Moreover, we performed preliminary experiments with different small LM confirm this choice (see section (5.4), and to analyse the impact of pretrained weights, as well as the frequency at which to update the LM (see section AA.2II) Ip. is worth noting that larger text encoders could also be integrated; however, the inference cost of large language models can become bottleneck (Kaplan et al. 2020 which
remains an open area of research.
We follow the same training procedure R2D2. by using large number of parallel environments to gather trajectories and prioritized experience replay to sample transitions to update Qs. Moreover, designed R3D2 support environments with varying numbers of players, ranging from to participants. To achieve this, multiple parallel actors take actions in these settings, while a shared replay buffer gathers the trajectories from all actors together. Although the agent itself is agnostic to the number of players, we adapt the replay buffer to handle sequences of different lengths. To address this, we pad the token trajectories with zeros, ensuring a consistent buffer structure across all trajectories, regardless of the number of players.
5 EXPERIMENTS
In this section, we analyse the generalization performance of R3D2 Hanabi. Hanabi is chal- lenging task, which requires to learn conventions with other players to reach a high score. When changing the number of players participating in game, the strategy reach high score changes, even though the rules of the game remain the same. For example, the number of allowed clues remains
2-player and 5-player setting, despite having more players that require hints in the 5-player setting. Despite requiring to change strategies, some of the learned conventions remain the same, and should be transferable from one setting to another. For example, providing the clue "this card is a card of rank 5" tells the player that they absolutely cannot discard that card, as there is only one such a card for each color, and it is required to play it to reach a maximum score. We aim to learn how specialized the conventions of agents that have played together during training are, by evaluating them in a ZSC fashion with agents they have never encountered before. We also aim to evaluate just how much these conventions agents that have been trained on different settings together. Finally, since our R3D2 agent is flexible enough to play any setting of the game, also train R3D2 agent all game settings
be carried on from one setting to another, by pairing

---

Published as a conference paper at ICLR 2025
the same time. We assess the benefits of playing more diverse types of gameplay, and how this improves the agent's cooperation abilities with others.
5.1 EXPERIMENTAL. SETUP
We train single-setting R3D2 agents for each setting, i.c., from 2-player games S-player games. We call these agents R3D2-S (single setting). Analogously, we call R3D2-M (multiple settings) the R3D2 agent trained on all game- settings concurrently. For comparison, we train different baselines on the original, vectorial version of the Hanabi environment. The first baseline is Independent Q- Learning (IQL) TTan 1993; Tampuu et al,, 2017). IQL is still frequently used as a baseline for Hanabi, as it serves as the foundation of many state-of-the-art MARL Hanabi algorithms, including the other baselines we use in this work. We call this baseline R2D2 as it is based on R2D2 agents trained independently through self-play. It shows strong performance with its training partners, having learned highly specialized conventions through self-play. However, this also means that it typically does not play well with other partners, who do not necessarily follow the same conventions. For our second baseline, we use Other-Play (OP) (Hu et al. 2020). In Hanabi, playing a game with permuted colors of the cards does not change the game, i.e., there exist symmetries in the game that the policy should be invariant to. OP learns such policies by exploiting known symmetries of the POMD.. This avoids over-specialized conventions that would break on different, but symmetrical (and thus equivalent) states of the game. We call this R2D2-based OP agent: R2D2- OP. Our final baseline is Off-Belief Learning (OBL) (Hu et al. 2021b). OBL assumes past actions where taken by a fixed policy, different from its own. OBL then converges to an optimal grounded policy, which can in turn be used to ground a higher-level policy. In our experiments, we use OBL after 4 levels of grounding, as this is the highest level of grounding used in their original work. To highlight that this baseline is also R2D2 based, we call it R2D2-OBL. All these baselines value-based, learning Qo network. To ensure fair comparison with R3D2, all baselines use R2D2 as a basis, which contains several modern best practices for learning Qp. Note that although train and test R3D2 in settings ranging from OBL checkpoints from the original paper, which focused exclusively on the 2-player setting, as it was the only configuration examined in those studies. To isolate the contributions of our two key innovations using language models for state representation and handling dynamic action spaces we created an intermediate baseline called R2D2-text. This agent combines R2D2's fixed action space with R3D2's 's text-based state representation.
players, we utilized R2D2-OP and R2D2-
All baselines use the same neural network architecture, a recurrent neural network which takes a vectorial observation as input and outputs the Q-values for all possible actions. Details about the network architecture and hyperparameters are described in Appendix [A.4 Each agent is trained for 2000 epochs with each epoch corresponding to 500 batch updates. For each experiment, we run three different seeds. For each evaluation setup, we define the performance of a team of agents as the average performance over 1000 games.
5.2 ZERO SHOT COORDINATION TO NOVEL SETTINGS
As a first set of experiments, to showcase our agent's ability to transfer policies across varying configurations, we evaluate its performance in different game settings. For a game with n-players, we select a subset i < n of players that have been trained on n-player games, and we pair them with players that have been trained on m-player games (each player is different seed). By aggregating all subsets 0 < i < n. we can assess how well the agents of m-player games generalize to n- player games in a zero-shot manner. Each combination of n and m is shown in Figure B As a reference point, we showcase the self-play (SP) performance of R3D2 (leftmost bar on each plot). Finally, we also showease the cross-play performance of n-player agents combined with R3D2-M (in blue). Finally, note that, when n m, this results in standard intra-cross-pla (e.g.. 5p-XP on the rightmost plot).
We make several observations. First, unsurprisingly, increasing the number of players results in lower overall performance, as strategies become more complex. Next, R3D2-M learns competitive strategies for all settings despite receiving the same training budget as single-setting algorithms, and is able to play well with single-setting players. This shows that knowledge from one setting is useful for other ones, and that the network architecture used by R3D2 allows for effective transfer of

---

Published as a conference paper at ICLR 2025
Eval: Players setting
Eval: Players setting
Eval: Players setting
Eval: Players setting
25
25
23
25
Random Agent
20
Zoro shot transfer
20
20
R202
R202-4ect
R302
R302-40
10
10
5
5
2P.55
M
2par
ap
4p
5p-XP
58 14
2p
3p
xP
4p-XP
Sp-kP
AP
2p-20
3p-XP
ap-4r
Sp-a
5p-5P
M.D
2p
4P
3p-4P
4p
Sp Sp-XP
Training setting
Figure 3: Policy Transfer Zeroshot setting. Each subplot shows the evaluation setting for a n-player game. Each bar combines 0 < i < agents trained on a different setting, with players trained on n-player games. R3D2 agents demonstrate strong zero-shot generalization to novel settings. Moreover, R2D2-text seems to be unable to match R3D2's transfer performance specially when transferring from a setting with large number of actions to smaller action space.
knowledge across settings, Additionally, R2D2-text is unable to match R3D2's transfer performance specially when transferring from a setting with large number of actions to smaller action space. Finally, R3D2's standard pross-play scores remain high, despite using self-play during training. The cross-play scores drop compared to self-play as the number of players increase, but that because this results in systematically has a lower cross-play score than R3D2, despite having the same action-space as during training. This shows that, while textual observations help for learning generalizable policies, incorporating dynamic action-spaces is inportant as well for same- same-setting scenarios. We refer to Appendix A.I for additional comparisons on cross-play.
unique players playing together for the first time. in comparison, R2D2-text
5.3
ZERO SHoT COORDINATION TO NOVEL PARTNERS
25
Intra-XP
IQL
22.5
Inter-XP
OP
17.5
Scores
2
15.0
15
OBL
10
S5F SL7 2
10.0
R302-5
R3D2-M
-.0
-2.
IQL
OBL R3D2-S
R3D2-M
IQL
O6L R3D2.5
R302.M
Agent
Figure 4: Selfplay, intra-XP and inter-XP performance in 2-player setting averaged across three independent seeds per method. R3D2 achieves significantly better inter-XP compared the baselines while maintaining competitive SP and intra-XP.
Figure 5: player zero-shot coordina- tion matrix between different methods and three independent seeds per method. R3D2 achieves better inter-XP with IQL, R2D2- OBL and R2D2-OP.
Now, we compare the robustness of R3D2's learned policies when partnered with novel agents, for the same game setting (i.c., cross-play) with other baselines. We not only compare the coopera- tion capabilities of different seeds of the same learning algorithm, but also of different algorithms with each other. We make 2-player teams composed of all combinations of seeds of all training algorithms, and evaluate their performance. The aggregated scores are shown in Figure 4 As a reference, we also show the performance when playing in self-play (SP) mode. The results of each individual combination of teammates results in the matrix of scores shown in figure 5 We note that the submatrices highlighted in red concern intra-algorithmic pross-play (intra-XP) (with their diagonal comparisons of the same seed, which equivalent to self-play), while all other compar- isons concern inter-algorithmic cross-payy (inter-XP). We start with the observation that R2D2 and
8

---

Published as a conference paper at ICLR 2025
R2D2-OP exhibit a lower intra-XP performance than self-play. While this is expected for R2D2, it is surprising for R2D2-OP, which is explicitly designed to cooperate well in the intra-XP setting. We believe this is because some of R2D2-OP's runs might not have exploited the environment sym- metries well enough. The inconsistent hehavior of R2D2-OP agents were also reported in [Hu et al. (2020). Since this is the way R2D2-OP learns robust policies, failing to exploit these symmetries would results in a training procedure similar to R2D2. This would explain lower intra-XP scores, and would also explain why R2D2-OP cooperates surprisingly well with R2D2 in the inter-XP set- ting. On the other hand, R3D2's intra-XP performance is on par with self-play, even though it has been trained through self-play, similarly as R2D2. While R2D2's performance quickly degrades when paired with other seeds, this is not the case for R3D2, which confirms that self-play can lead to robust strategies, provided they use the appropriate representation. We believe that R2D2-OP, by exploiting known symmetries of the game, learns that different permutations of the vectorial version of Hanabi are equivalent. Trying to make sense of this vectorial representation thus contributes to R2D2-OP's 's cross- play abilities. In contrast, in our textual representation, symmetrical observations are the same but for few tokens. Symmetries are thus implicitly tackled by R3D2.
The aggregated scores are shown in Figure 4 In general, while our agents achieve competitive per- formance in self-play and int 23D2-based agents demonstrate superior inter-XP performance when paired with R2D2- agents. This suggests R3D2 learns more robust and general strategies, in contrast to R2D2 and R2D2-OP which tend to learn brittle, specialized conventions. The strong performance with R2D2-OBL. which known for learning compaiile strategies (Hu al., 2021b), indicates that R3D2 develops more natural and transferable coordination patterns. Inter- estingly, R3D2-M seems better inter-XP than R3D2-S. Having been trained on multiple settings, R3D2-M has experienced more diverse strategies during training. We surmise that this diversity allows R3D2-M be prepared to a wider range of novel policies, leading to higher overall collab- oration. Next, we note that R2D2-OBL collaborates better with R3D2 than any other baseline. We thus ask ourselves if it R2D2-OBL that flexible enough to adapt to R3D2 or the opposite. We aim to answer this question by comparing their performance with R2D2, the least flexible policy. When R2D2-OBL is paired with R2D2, their score is lower than when R3D2-S is paired with R2D2. Thus, R3D2-S seems to have more flexible policy than R2D2-OBL. An analogous analysis can be made for R2D2-OP, where R3D2-S paired with R2D2-OP achieves a higher score than R2D2-OBL. with R2D2-OP. Results shown in Figure demonstrate that R3D2, even though trained using self-play, learns more robust policies than methods that explicitly aim to learn policies for ZSC.
5.4 LANGUAGE MODEL VARIANTS FOR HANABI
LMs have shown promising reasoning and planning capabilities Radford et al. (2019b); Brown al. (2020). Having a completely text- game, one might expect a LM should be able to play the Hanabi game successfully. Therefore, before testing our R3D2 agent, we evaluate several LMs on the Hanabi tasks with the text-based Hanabi environment, to understand the difficulty of playing the game of Hanabi for current LMs and establish a baseline for future improvements.
Prompting and low-rank adaptation fine-tuning Modern large LMs (LLMs) such as GPT- (OpenAll 2023) and LLaMA-2 (Touvron et al. 2023) showcase remarkable zero-sho shot or few-shot generalization capacities, particularly in complex natural language tasks Kojima et al. (2022). First, we prompt GPT-4 to select actions for Hanabi, providing the textual observation and legal moves as context. Although it successfully avoids losing life tokens for extended periods, it struggles with optimal planning, limiting its ability to achieve scores higher than or 4 Details about the var- ious system prompts and game scores how well a L.LM fine-tuned on expert data would perform. For this, we fine-tune I.LMA-7B us- ing low-rank adaptation (LoRA) (Hu et al. 2021a) on dataset of expert data collected using an Off-Belief Learning agent (Hu et al. 2b221b) Despite this tuning, the model performs poorly based on gameplay scores. Hu & Sadigh (223)) further supports the observation that current large LLMs are still far from independently solving Hanabi. For more detailed experimental results, we refer to Appendix 8.88 These initial experiments seem to indicate that LLMMs as-is are not sufficient to properly play Hanabi, and that learning to coordinate is required.
available in Appendix AA6] and A.8 Next, we analyze
We also tested the 03-mini model, which was released after the rebuttal, and observed significantly im- proved results. However, it still falls short of matching the performance of SOTA RI. agents in gameplay. The code is available online a notchook here).
9

---

Published as a conference paper at ICLR 2025
Full fine-tuning Next considered full fine-tuning of small LMs. Therefore, focused on two types of language models-classifiers classifiers and generative models- as agents in the text-based Hanabi environment. We compare the impact of BERT-like architectures, i... BERT (Devlin et al. 2018a) and DistiBBERT (Sanh et al. 2019), with the GPT-2 (Radford et al. 2019a) generative) architecture.
(classifier and
To benchmark the performance, we select the optimal checkpoint for each LM based on gameplay scores. The best checkpoint is then subjected to 1200 runs in the Han- abi environment to handle variance and randomness, as depicted in Figure [ Both the BERT and DistilBERT models demonstrate commendable performance in the Hanabi gameplay, achieving a maximum score of 23 out of a possible 25. Their average gameplay scores hover around 10 during the gameplay. The GPT2-generative model has better top-k test accuracy. However, fails short compared to the classification-bii based model in the overall gameplay score with ~ 4.5. We further tried using understand the role of data. Compared to 10% or less, when using 25% of the data, there a sharp increase in the gameplay score. However, the performance plateaus for both 75% and 100% are indicative of reaching a saturation point. Also, we tried different BERT variants, and all are satu- rated to the same game score irrespective of the increase in the parameter size. Finally, we also investigated the role of discarding information in the ob- servation and found it didn't help much in the gameplay score. Refer to appendix [A.9 for detailed ablation results language models. Since the BERT architecture results in higher performance than GPT-2, we use a BERT-like network in our R3D2 agent.
Game Score
BERT
DiSuBBRT
GFTz-dasitte dassilar
G9T2-gerieation
GP
different percentages of training datasets
Language Models
Figure 6: The evaluation of language models performance as an agent is con- ducted via gameplay score, measured across various language models with 1200 game runs.
CONCLUSION AND FUTURE WORK
In this work, we show that learning through self-play can lead to robust policies, provided that the learning agent is trained with an adequate representation. We propose Recurrent Replay Relevance Distributed DQN (R3D2), that plays Hanabi with a textual representation of the game, and a player- agnostic neural network architecture. R3D2's intra-algorithmic cross-play score is on par with its self-play score, first for Hanabi agents learning through self-play. Moreover, our experiments show that pairing R3D2 agents from different settings together can lead to collaborative success, with agents having been trained more complicated settings being more capable in general than agents that have been trained on simpler settings, with less players in the game. Additionally, R3D2's player-agnostic architecture facilitates variable-player learning, enabling it to generalize strategies across various settings. This opens a new research avenue for exploring generalization across game settings, in addition coordination with novel partners in MARI. for complex cooperative games such as Hanabi.
Our approach, leveraging embedding and language models, is naturally adaptable to other text-based tasks. However, we acknowledge certain limitations environments with continuous state/action spaces (like robotic control) or image-based inputs would require domain-specific adaptations. How- ever, recent advances in language models are expanding the possibilities. For instance, Llama-2's specialized tokenizer demonstrates remarkable performance on numerical tasks by decomposing numbers into digit sequences (Touvron et al., (023). An interesting direction for future work in- volves enhancing inter-setting cross-play evaluation, which we introduced and see as having signif- icant potential. This area allows for further exploration of robustness by improving agents' adapt- ability across different game settings. Expanding the evaluation to include various combinations of replaced agents and different algorithms could yield deeper insights. Additionally, while Zero- Shot Coordination serves as a useful benchmark. it may lack realism. Exploring few-shot coordina- tion (Nekoci et al. 2023) could be a promising research direction, where agents quickly adapt to new environments and partners, striving for consensus and effective collaboration in minimal episodes, offering more dynamic approach to agent interaction in complex scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their insightful comments and constructive suggestions that significantly improved the quality of this manuscript. The authors acknowledge the computational resources provided by Mila and the Digital Research Alliance of Canada. Hadi Nekoei is supported by NSERC and FRQNT Doctral scholarships. Sarath Chandar is supported by the Canada CIFAR AI Chairs program, the Canada Research Chair in Lifelong Machine Learning, and the NSERC Discovery Grant. This work was also supported by an IBM-Mila collaboration grant.
REFERENCES
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, pp. 166-175. PMLR, 2017.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 130:1033 103216, 2020.
Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and under- standing in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185-5198, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18655/112222..alllm. a0a220000.cc-mma.46666 URL https:
org/
2020.ac1 463
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference Machine Learning, pp. 41-48, 2009.
Daniel Š Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of Operations Research, 27(4): 819-840, 2002.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: //www. www. wandb. com/ Software available from wandb.com.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. Experience grounds language. In Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), pp. 8718-8735, Online, November 703. URL
2020. Association for Computational Linguisties. doi: 10.
htttp:://aclamtholloyy 20w.aaammmmooooooooo222000000 wwt//o20.2222o. 1ttp://ow..or//222mmmmmmmppppppp emnlp- ina.inn... 703
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
Jana Carbonell and Manuela Veloso. Transfer learning in multi-agent systems through parallel transfer. In International Conference on Machine Learning Workshop on Structural Knowledge Transfer for Machine Learning. 2006.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, 32, 2019.
Brandon Cui, Hengyuan Hu, Luis Pineda, and Jakob Foerster. K-level reasoning for zero-shot co- ordination in hanabi. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 8215-8228. Curran Associates, Inc., 2021. URL http://s////pcccceiiiieeooo/eeee meeeeeeedeennneeeeeeeeeeeeeeeeee neur ps. cc cc/paper/
pdf
Felipe Leno Da Silva and Anna Helena Reali Costa. A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:645-703, 2019.

---

Published as a conference paper at ICLR 2025
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.08660, 2020.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances in Neural Information Processing Systems, volume 33, pp. 13049-13061, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- traitrainin of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018a. 8a. URL
http:://axx: org/ abs/ 1810. 04805
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini arXiv:/810.0400 2018b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. CoRR, abs/1703.03400, 2017. URL http:::/arrxv..org/aa///////////
03400
Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent re- inforcement learning. International Conference on Machine Learning, pp. 1942-1951, PMLR, 2019.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Inter- active fiction games: A colossal adventure. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7903-7910 Apr. 2020. doi: 10.1609/aaai, v34i05.6297. URL https:
//ojs. wwa..aaai..cm www.aiioor/nnndeeeeeeeee
AAAI/ http://A/aaattceeeeeeeee 6297
Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language action space. arXiv: Artificial Intelligence, 2015,
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Has- selt, and David Silver. Distributed prioritized experience replay. In International Conference Learning Representations, 2018
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021a.
Hengyuan Hu and Jakob Foerster. Simplified action decoder for deep multi-agent reinforcement learning. arXiv preprint arXiv:191..02288. 2019.
Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-ai coordi- nation, 2023.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-play" for zero-shot coordination. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Muchine Learning Research, pp. 4410. PMLR, 13-18 Jul 2020.
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, and Jakob Foerster. Off- belief learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4369 4379. PMLR, 18-24 Jul 2021b. URL https:ttp//pt... http:pppp//oocceedggg... mlr ptwss/v13s/ hu21c.hta. h
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, David J. Wu, Noam Brown, and Jakob N. Foerster. Off-be learning. CoRR, abs/2103.04000 2021c. URL http:::/t//// :/ttppsi//riiv....// org/ abs/ / 21 03. 04000)
12

---

Published as a conference paper at ICLR 2025
Vishal Jain, William Fedus, H. Larochelle, Doina Precup, and Mare G. Bellemare. Algorithmic improvements for deep reinforcement learning applied to interactive fiction. In AAAI Conference on Artificial Intelligence, 2019.
Youngsoo Jang, Seokin Seo, Jongmin Lee, and Kee-Eung Kim. Monte-carlo planning and learning with language action value estimates. In International Conference on Learning Representations, 2021.
htttps://openreviee... net. htttt//foormmmaaa______ae---
Xin Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing: Findings, pp. 4163-4174, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
wwwp:xiiii.rrg/boeo wttp://pp.abb...oooooooo 08361
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent ex- perience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2019.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22221 2022.
Wenlong Lee, Abhishek Agrawal, and Jitendra Malik. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning. pp. 12648-12671. PMLR, 2022.
Keane Lucas and Ross E. Allen. Any-play: An intrinsic augmentation for zero-shot coordination, 2022. URL https: / arxiv. at//aab/2222 abs/ 2211//222111113666
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp- 7204-7213. PMLR, 18-23 24 Jul 2021. URL
mlr. wwt..ppp../.v.//.//
1upu2la 21......0 html.
Mingwei Ma, Jizhou Liu, Samuel Sokota, Max Kleiman-Weiner, and Jakob N. Foerster. Learning to coordinate with humans using action features. CoRR, htss/2011.1655,, 2022. URL. https:
12658
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. Nature, 518(7540):529-532 2015.
Keerthiram Murugesan, Mattia Atzeni, Pushkar Shukla, Mrinmaya Sachan, Pavan Kapanipathi, and Kartik Talamadupula. Enhancing text-based reinforcement learning agents with commonsense knowledge. CoRR, abs/2005.008 2020. URL httpt::/p//:attta. org/ absa 200000005.00 00811
Ranjit Nair, Milind Tambe, Makoto Yokoo, David Pynadath, and Stacy Marsella. Taming decentral- ized pomdps: Towards efficient policy computation for multiagent settings, In Proceedings of the 18th International Joint Conference Artificial Intelligence (ICAAI) pp. 705-711, 2003.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, (1(11)):1-50 2020.
Hadi Nekoci, Akilesh Badrinaaraayanan, Aaron Courville, and Sarath Chandar. Continuous coor- dination as a realistic scenario for lifelong learning. In International Conference on Machine Learming, pp. 8016-8024. PMLR, 2021.
Hadi Nekoei, Xutong Zhao, Janarthanan Rajendran, Miao Liu, and Sarath Chandar. Towards few- shot coordination: Revisiting ad- hoc teamplay challenge in the game of hanabi. In Conference on Lifelong Learning Agents, pp. 861-877. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2025
OpenAl. GPT-4 Technical Report. arXiv e-prints, art. arXiv:2303.08774, March 2023. doi: 10. 48550//rXXiv22003.88774
Philip Osborne, Heido Nömm, and André Freitas. A survey of text games for reinforcement learning informed by natural language. Transactions of the Association for Computational Linguistics, doi: 11l16//aalla.0004400 URL https: /aw..//nnhoooo..orr/////00000
LinggittiiioallLLLiniisiisiii 10:
873-887, 2022.
tacl-1.51
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Perey Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners. 2019a. URI. https: //api.
Semantiicscholann wwtp:aarr.rg//oouuoSTte
60025533
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188)99 2019b.
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette, and Tim Rocktäschel. Maestro: Open- ended environment design for multi-agent reinforcement learning. In International Conference on Learning Representations, 2023.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108, 2019. URL http://
arxiv. org. abs wassss/910 /1900.91..000 01108
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 1rXiv:151 (5110595 2016.
Ishika Singh, Gargi Singh, and Ashutosh Modi. Pre-trained language models as prior knowledge for playing text-based games. ArXiv, abs/2107.08400. 2021.
Arjun Vaithilingam Sudhakar, Prasanna Parthasarathi, Janarthanan Rajendran, and Sarath Chan- dar. Language nooel-in-thhellop: Data optimal approach to sear-tto-reoommend actions in text games, 2023.
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4):c0172395, 2017.
Ming Tan. Multi reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
OpenEnded Learning Team, Andrew K Lampinen, Stephanie CY Roy, Ishita Patel, Shagun Sodhani, Andrea Banino, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu. Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Tuned Chat Models. arXiv e-prints, art ar0vv:2307 .99888,98888881 July 2023. doi: 10.48850/arxiv.23077 079888888....
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), 2016.
Ruoyao Wang, Peter Alexander Jansen, Mare-Alexandre Côté, and Prithviraj Ammanabrolu. Sci- enceworld: your agent smarter than a 5th grader? In Conference on Empirical Methods in Natural Language Processing, 2022.
Ziyu Wang, Tom Schaul, Matte Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learming, pp. 2003. PMLR, 2016.
Yunqiu Xu, Ling Chen, Meng Fang, Yang Wang. and Chengqi Zhang. Deep reinforcement learning with transformers for text adventure games. 2020 IEEE Conference on Games (CoG), pp. 65-72, 2020.
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep CALM and explore: Language models for action generation in text-based games. In Proceedings the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pp. 8736-8754, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ htt/:///122000/mppppp main.. 704. URL https::p///o :eatts://acee
704]
Xusen Yin and Jonathan May. Learn how to cook new recipe in new house: Using map familiar- ization, curriculum learning, and bandit feedback learn families of text-based adventure games, 2019. AtRL:ttps: //axivv wwwg/baa..... 908. 04777
Xingdi Yuan, Marc-Alexandre Côté, Alessandro Sordoni, Romain Laroche, Remi Tachet des Combes, Matthew Hausknecht, and Adam Trischler. Counting to explore and generalize text- based games. CoRR, abs/1806. 1525, 2018. URLhttp: http///axivvvi// org abs/ 1806 11525
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 SHRO COORDINATION TO NOVEL PARTNERS
To demonstrate R3D2's robustness, we report self-play and intra-XP performances R3D2, IQL, and OP trained on 3-, 4- 5-player game settings in Figure 7 IQL and OP achieve high self-play scores but perform poorly in cross-play. R3D2 variants, particularly R3D2-S, demonstrate more consistent performance across both metrics, maintaining scores above 15 points in all scenarios despite the general decline performance as player count increases. R3D2-M This suggests that R3D2's training approach leads to more robust and adaptable agents, though at slight cost to self- play performance compared to IQL and OP.
Players setting
Players setting
Players setting
25
SP Intra-X?
20
20
20
1.5
15
15
10
10
5
5-
0-
OP Method
Figure 7: Performance comparison across different settings. The table shows both self-play (SP) and intra-cross-play (Intra-XP) scores for different methods in 3- 4- and 5-player Hanabi settings. While IQL and OP achieve high SP scores but fail in Intra-XP, both R3D2 variants maintain consis- tent performance across both metrics, with R3D2-S showing particularly strong results.
A.2 ABLATION STUDIES ON THE ROLE LANGUAGE MODELING
To better understand the impact of different components in R3D2, we conduct series of abla- tion studies examining the role of language model pre-training, update frequency, and architectural choices. These experiments help isolate the contributions of our key innovations and validate our design decisions.
A.2.1 LM INITIALIZATION AND UPDATE FREQUENCY
25-
11.5
SundeniM
220-
7.-
28
60-
03
20
30
00
L8
20
41
(a) Pretrained vs random LM (b) Frequency of updating the
weights
LM
(c) R3D2 vs R2D2-text R2D2
Figure 8: Impact of pre-trained weights and update frequency on learning efficiency. (a) Perfor- mance difference between R3D2 agents trained with pre-trained language model (LM) weights ver- randomly initialized LM weights, showing significant improvements in sample efficiency with pre-trained weights. (b) The effect of varying the frequency LM updates, highlighting that fre- quent updates are critical for effective learning in the Hanabi environment.
We train two R3D2 agents in a 2-player Hanabi setting: using pre-trained language model (LM) and the other with the same architecture but randomly initialized LM weights. FigureBaalshor that learning from pre- pre-trained weights significantly improves the sample efficiency. Additionally, test updating the LM less frequently with periods of 1, 2, 5, and 10 training steps per LM update examine whether the original pre-trained weights provide sufficient representations for playing
16

---

Published as a conference paper at ICLR 2025
Hanabi or if fine-tuning is necessary. Our results, presented in Figure 8b LM parameters is essential for effective learning.
indicate that updating the
A.2.2 DOES THE R3D2 PERFORMANCE COMES FROM LANGUAGE MODEL OR THE ARCHITECTURE?
As shown in Figure &c while R2D2-text achieves better intra-XP performance than the original R2D2, still falls short of R3D2's capabilities. R3D2 matches R2D2's strong self-play performance while significantly outperforming both baselines in intra-XP scenarios. These results demonstrate that both innovations are crucial: text representation alone provides some benefits for generalization, but the combination with dynamic action space processing is necessary to achieve robust transfer to novel partners.
A.3 R3D2 vs R3D2-M THE FIXED PARTNER
Building upon our previous analysis in Figure 3 where we demonstrated R3D2's zero-shot transfer capabilities with least one specialized agent, we further investigate the generalization capabilities of our multi-task variant, R3D2-M. We conduct comparative analysis by positioning R3D2-M as the fixed partner and evaluating its cross-play performance with partners trained across different game settings. Figure 9 presents the performance comparison between R3D2 and R3D2-M when paired with R3D2-S partners trained on various settings (indicated on the x-axis). The results reveal comparable performance patterns between the two variants, with R3D2 exhibiting superior perfor- mance in certain scenarios (e.g.. 2-player setting) while R3D2-M demonstrates stronger capabilities in others (e.g., 5-player setting). This balanced performance profile suggests that R3D2-M main- tains robust generalization capabilities, achieving performance levels comparable to its single-task counterpart, R3D2-S, despite being trained on multiple settings simultaneously.
25
2p Setting
25
3p Setting
25
4p Setting
25
Setting
R302-M us R3D2-5
20
R302 s R302
15-
15-
15
15
5
2p-sp
ap-sp
4p-xp
5p-a0
2p-ap
3p-ap
4p-sp
sp
3p-20
3p-sp
4p-20
Sp-ap
3p-4p
4p-sp
Sp-ap
Outsider
Figure 9: Policy Transfer Zeroshot setting. Each subplot shows the evaluation setting for n- player game. Each bar combines i n agents trained on different setting, with n players trained on n-player games. Comparing R3D2 and R3D2-M's cross-pla pross-play performance with R3D2-S partners trained on different settings (Figure (.. we observe complementary strengths: R3D2 excels in 2-player settings while R3D2-M performs better in 5-player scenarios. Despite being trained on multiple settings simultaneously. R3D2-M achieves comparable performance to its single-task counterpart, demonstrating robust generalization capabilities.
A.4 R3D2 TRAINING SETUP
Here we provide all the experiment details and hyper-parameterre used to train R3D2 agents as shown in T To train R3D2 agents, we did not run a hyper-parameter tuning and relied on the HPs used by Hu et al. (0221b)
A.5 SOFTWARE DETAILS
The code was implemented using PyTorch, and pre-trained language models were loaded using Huggingface. To gain insights for this paper, we employed Weights & Biases (Biewald (0200 for experiment tracking and visualizations. Lastly, plots are created using the seaborn package, For RL algorithms, used OBL agent (Hu et 2021c) to collect the expert trajectory and forked official instruct-rl codebasd to train the algorithm.
17

---

Published as a conference paper at ICLR 2025
Table 1: ParamePes Parameters R3D2 agents.
Hyper-parameter
Value
replay buffer related
burn in frames
10, 000
soolerrufffe..ee
50, 000
priority.exponet
0.9
priority.weight
0.
max.crajectory.lensthh
80
# optimization
optimizer lr eps
Adam 6.25e-05
1.5e-05
grad.clip
batchsize
64
# Q learning n_step
(R3D2)
discount_factor
0.999
target..ettwrkk.syo..teee
2500
exploration e
co...En: where e, = 00.1+77/(( 11) 80
A.6 PROMPTING DETAILS
System prompt "You are an expert Hanabi player"
System prompt 2: "You are an expert Hanabi player focused on maximizing team coordination and achieving high scores with minimal mistakes. Follow these principles: Efficient Clue-Giving: Provide clues that give maximum information, using finesse and double clues to benefit multiple players. Deduction: Track played/discarded cards and deduce your own cards based on clues and game state. Avoid discarding critical cards. Disciplined Play: Play and discard safely, minimiz- ing risk while optimizing the team's 's progress. Team Coordination: Follow team conventions and subtle cues (timing, actions) communicate intent without verbal clues. Score Maximization: Manage clue tokens and pace the game to ensure enough clues for critical moments. Avoid getting bombed!"
A.7 DATASET DRTAILS
The dataset is acquired through self-play mode, utilizing pre-trained OBL agent in the Hanabi game. Trajectories filtered selectively with gameplay 20. Then, these trajectories are broken down into state- action pairs to suit language model training. During the initial data exploration, we found the action categories are imbalanced as shown in 10 hence the language model overfits to discard 4 based on the confusion matrix for the prediction. To avoid that, we did categorical sampling consisting of 2200 samples per action type, aggregating to 44, 000 instances. Then we checked for duplicate states and dropped them, there were approximately 100 dupli- cates as this could mislead the model's 's learning. After which, 10% of the dataset is reserved for testing by ran- dom sampling. Further, the dataset is split into 90% for train and 10% for validation.
exceeding
Rby
Fouss player
plapar
plapar
Steard
plapar
Revedl
Roved player
8038
Figure 10: Visualizing the number of ac- tions available in the dataset to create diverse dataset of Hanabi gameplay in the form of text.
18

---

Published as a conference paper at ICLR 2025
10-
Valldttion Accursc
0.45
0.40
0.30
DiStBEERT
BERT
2-
Distil
GPTZ-classiiee
0.25
GPT2-genanalive
0-
10
10
Epo
Epochs
Figure 11: Learning graph for (a) Validation accuracy plotted against(b) Game play score, for each epoch for different language model providing insights into the observed trends during training process.
Table GPT-4 and o3-mini performance with different system prompts on text-based Hanabi aver- aged over seeds. including the final scores before getting bombed.
Method
System Prompt
System Prompt 2
GPT-4
0.0 + 0.0 7.66 + 3.13
0.0 0.0
03-mini
12.0 - 0.47
GPT-4*ts Rombed)
3.0 + 0.0 (1009)
2.34 + (127 12.0 0.47(34)
03-mini #cmbed)
9.0 2.05(33.33)
A.7.1 LANGUAGE MODEL SETUP
The model's finetuning process begins with set of train- ing instances, denoted (S, A) drawn from the dataset D where S E {sp, $1..., 8n} and A d1. (ao.a1,..,
Within this set, s and a represent state and its corresponding noisy labelled ac- tion. respectively, and represents the number of examples in the dataset. The training objective of BERT, DistiBBERT, GPT2-Classifier is,
C
LCCE
1 N
ΣΣ
ai log(āu))
Where N is the batch size. C' is the number of classes. a;j is the true probability of class for the i-th example in the batch and 0iy is the predicted probability class for the i-th example in the batch.
The training objective of GPT-2 Generative is to minimize the cross-entropy loss, denoted as C, and do the finetuning of the model. The cross-entropy loss is mathematically defined as follows:
CLEM (E{D,,)) log p(p
Where p(S)A) represents the conditional probability of predicting an action A, given state S. The goal is optimize these parameters, by minimizing the cross-entropy loss. We finetune the model to generate responses that better align with Hanabi game. The learning graph of validation accuracy with the game play score for each epoch is logged understand the trend in the Figure la.b). Mostly the Validation score and game score getting saturated at around 4th epoch.
A.8 How GOOD LLMS ARE IN PLAYING HANABI?
First, we evaluate GPT-4's ability to play Hanabi using our text-based format. While GPT-4 demon- strates basic game understanding by avoiding catastrophic moves, it achieves only rudimentary
19

---

Published as a conference paper at ICLR 2025
scores of 3 points out of 25 (2) highlighting the limitations of pure language models in strategic planning.
To adapt the LLaMA to the gameplay,
Low-Rank Adaptation, LoRA (Hu et al.) 2021a
which learns low-rank decomposition matrices into each layer of the transformer architecture and freezes the pre-trained model weights. Thereby, significantly reducing the trainable parameters. We conducted fine-tuning experiments with LLaMA-7B weights with classifier using varying data sizes [200, 500, 1000] and LoRA ranks [32, 64, 128] for 10 epoch. Despite these parameter variations, the gameplay scores remained suboptimal level of around one as shown in 12 This highlights the challenges in achieving effective gameplay performance for current large languge model on playing hanabi.
L9t5 Bum
Figure 12: Evaluation of Low-Rank Adaptation (LoRA) in LLaMA-7B finetuning, showcasing the impact Validation Accuracy and b) Game Play Score. The experiments involve varying data sizes (200, 500, 1000] and LoRA ranks [32, 64, 128).
A.9 ABLATION STUDIES
A.9.1 THE ROLE OF SCALING THE DATASET AND DIFFERENT MODEL VARIANTS
The dataset size emerges as pivotal factor influencing gameplay scores. As the amount of training data increases there is gradual increase in validation and the gameplay score. When the training percentage is equal to or less than 10% the games scores were poor ranging around out of 25. In contrast, the gameplay score sharply increases when using 25% of the data as shown in 13p. Nevertheless, the performance plateaus at a game play score of approximately 9 for both 75% and 100% indicative of reaching saturation point, affirming the sufficiency of the dataset size for effective model training.
24
6.5
M6
100
Figure 13: Analysis of the impact training data amount on BERT, examining a) BERT Validation Accuracy, b) BERT Game Play Score across different percentages of training data, and c) BERT model variants with varying parameter sizes.
In our experimentation, varied the model parameter sizes- raeeinn from DistilBERT with 66A parameters to 3ERT-base-uncaa with 110M parameters and BEIRTlargg-uncased with 340M pa- rameters. We observed that DistilBERT achieves a competitive gameplay score of approximately 8.7 after 600 game runs 13p. On top of the performance considering the fast inference and low memory usage, DistilBERT was chosen as candidate integration with reinforcement learning through distillation.
20

---

Published as a conference paper at ICLR 2025
A.9.2 THE ROLE OF DISCARD INFORMATION
We examined the impact of incorporating the discard pile into the observation. Surprisingly, we discovered that utilizing the discard pile did not contribute to any improvement in game scores as show in the Figure [4 Rather, it resulted doubling the sequence length of the language model. Given the need for fast inference in the reinforcement learning pipeline, we opted ex- clude discard pile information from the observation dur- ing both language model training and inference. Nonethe- less, there is potential for heuristic-based approaches, to explore the idea of creating derived information from from the discard pile, potentially leading to a more con- cise sequence length and better game score.
Game 4-
DistiBERT
BERT
Model Type
Evaluation of the discard pile 's role in the game is assessed by comparing game scores with the pres- ence and absence of the discard pile in the observation during training.
Figure 14:
21


Reviews:
[{"review_id": "YTmC7Tjkv3", "reviewer": "Reviewer_wJ2W", "summary": "The paper introduces a new approach to utilizing text generation to enable knowledge transfer in more than 2 agents in Hanabi tasks. The proposed method shows performance improvements with different LLMs in ZSC settings. Overall, the paper is well-written and shows some interesting inspirations.", "questions": "Besides the questions from the weakness. I have a few concerns about the text templates used in the paper:\n- if the text is embedded and used by the Q-functions, does it mean the text is better than observation features? Is there any intuition behind this?\n- is there any communication between the agents? In Hanabi tasks, the agents can utilize hint actions to communicate information implicitly. If the text is now used, would this shift to explicit communication and potentially risk leaking private information?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written\n- The idea seems to be promising and novel\n- The experiments are extensive", "weaknesses": "- the text template is not clear enough. What information is included in the text template? Will the text be revealed to other agents? It seems that the template will record all \"previous\" information which is simply the history of the observations. Then why not use history instead of text recording?\n- the author said R2D2 is a foundation of R3D2, but it does not compare to R2D2. Moreover, since R2D2 is not introduced in detail, it is hard to perceive the technical contribution of R3D2\n- what's the intuition behind multiplying the state and action embeddings? Where do you integrate the text information? how do you make sure it can accommodate different numbers of agents and task settings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O7c10lRmH3", "reviewer": "Reviewer_YgHj", "summary": "This paper proposes a new method to train an agent to be able to play the game of Hanabi with the ability to generalize to all game settings and to be able to play with other agents in a zero-shot coordination capability. They do this by reformulating the task as a text task by converting current state and an action and play history into a text representation, embedding it using a language model and then feeding these embeddings into a Deep Recurrent Relevance Q-Network (DRRN) trained in a distributed way with a shared replay buffer. Their architecture called R3D2 uses R2D2 (Recurrent Replay Distributed Deep Q Network) as it’s foundation.\n\nThe biggest claim of this work is that no other work before their work has been able to generalize Hanabi agents across game settings.", "questions": "1. Some parts of the setup were a little confusing. eg: “All baselines use R2D2 as a basis” vs “we utilized OP and OBL checkpoints from the original paper which focused exclusively on the 2-player setting”. Were the original checkpoints foundationally R2D2?\n2. Is a jump of 3-5 points a big jump in the game of Hanabi, regarding Table 5?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is motivated well and written clearly, with a sufficient explanation of the game and the existing literature to bring the reader upto speed\n2.\tThe evaluation and experiments are thorough, ablating and justifying all of the pieces. They start with first demonstrating that LM only is not sufficient to play the game. Then they show comparisons against 3 existing approaches.", "weaknesses": "1. The work is Hanabi-centric. It may be possible to apply this same work to other card games as it's mostly the text representation that would change\n2. Even with its Hanabi focus, I am unable to see the claims being made in the results section. R3D2 scores lower on Self-play and intra-XP but higher on inter-XP. The grid in Fig 5, also does not have massive increase in scores against R3D2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r6hEflIhJo", "reviewer": "Reviewer_uDKb", "summary": "This paper addresses a fundamental challenge in multi-agent reinforcement learning (RL): the difficulty of achieving robust performance across environments beyond those the agents were specifically trained on. To tackle these issues, the authors propose and evaluate their approach in a well-established environment for multi-agent RL, Hanabi, which is commonly used to test cooperation in complex multi-agent settings. The paper makes two primary contributions. First, it reformulates the task using textual representations, motivated by evidence that language can improve transferability. By representing the Hanabi game through text, the authors aim to create agents that are more adaptable and capable of generalizing beyond their training environment. Second, they introduce a novel neural network architecture that combines a language model with the Deep Recurrent Relevance Q-network (DRRN). The results presented indicate that the proposed method achieves high intra-algorithm cross-play scores when trained in a self-play setting, underscoring its capacity for adaptability and teamwork. Additionally, the method performs exceptionally well in inter-algorithm play, highlighting its capability to cooperate with agents utilizing different algorithms.", "questions": "1. Could the authors clarify why the focus is primarily on self-play during the training phase? While it is impressive that the proposed method achieves high cross-play scores after self-play training, it seems limiting not to explore other training strategies, especially when inter-algorithm play is a key evaluation goal.\n\n2. Given the novel combination of language model and DRRN components, was there a reason for not experimenting with a wider variety of neural network architectures? Exploring different architectures might reveal other configurations that could improve performance, especially in self-play or more complex settings. Were alternative architectures considered during development, and if so, what were the outcomes? Additional details about architectural choices and trade-offs would be helpful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. One of the primary strengths of this work is its success in achieving effective transfer learning in a zero-shot setting. The proposed method shows adaptability across varying scenarios without requiring additional retraining, which addresses a significant limitation in existing multi-agent RL models.\n\n2. Through careful evaluation, the authors demonstrate that their approach outperforms existing methods in inter-algorithm play, achieving high scores in cross-play with agents using different algorithms.\n\n3. By representing the Hanabi game through text, this work leverages advances in natural language processing (NLP) and large language models (LLMs), marking a novel direction in multi-agent RL. Their approach allows the potential for more seamless integration with future advancements in language models.", "weaknesses": "1. The reliance on text-based representation, while novel, may face significant challenges in more complex or non-linguistic domains. For instance, domains where numeric data dominates—such as algorithmic trading, where key data points include stock prices and volumes, or robot-taxi applications, where visual and sensory data are crucial—may not naturally fit into a textual framework. An extension or discussion on the applicability limits of textual representation in various domains would provide valuable context and guide future research on adapting the method to broader applications.\n\n2. The self-play performance is relatively poor. Insights into possible contributing factors, such as model structure or training limitations, would be useful. Additionally, outlining potential directions for improving self-play performance, whether through modifications to the architecture, training process, or environment setup, would provide actionable guidance for future work.\n\n3. There is insufficient discussion on the choice of neural network architecture, particularly regarding the component where the element-wise dot product is applied between the embedded observations and actions. Similar ideas have been explored in the literature, such as in https://proceedings.mlr.press/v202/ma23e/ma23e.pdf. The authors may consider incorporating insights from existing explanations in the literature to better articulate the advantages and underlying mechanisms of this architectural choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1cxKEB1jca", "reviewer": "Reviewer_ubVW", "summary": "The paper considers the problem of generalization and coordination in multi-agent reinforcement learning (MARL) within the game of Hanabi, focusing on enabling agents to adapt across different game settings and learn to collaborate with unfamiliar partners. To tackle this, the paper makes two primary contributions: first, it reformulates the Hanabi environment using a text-based representation, which enables use of language model to process state and action space, thereby providing a consistent observation and action space across varying player configurations; second, it extends the distributed training regimen of Recurrent Replay Distributed DQN (R2D2) by combing language model with Deep Recurrent Relevance Q-network,  creating what they call Recurrent Replay Relevance Distributed DQN (R3D2), an agent architecture that can process dynamic observation and action spaces in a multi-agent environment. Experiments assess R3D2’s ability to coordinate with both familiar and unfamiliar agents in zero-shot settings and to transfer strategies across different player configurations, evaluating performance across various player numbers and agent types. The authors propose that this approach could support further exploration of generalization in MARL for complex cooperative games.", "questions": "- Could the authors clarify their rationale for selecting Hanabi as a testbed for studying adaptability to new settings and new co-players? Specifically, in what ways does Hanabi reflect the complexities of real-world collaborative tasks, as mentioned in the introduction?\n\n- How does the proposed approach relate to the existing literature on Unsupervised Environment Design, particularly for multi-agent reinforcement learning? Were there considerations given to this body of work in framing the method, and if so, how does the R3D2 approach build on or diverge from these methods?\n\n- How scalable is the R3D2 approach with respect to increasing the number of players? Given that the current implementation handles a maximum of five players, are there foreseeable limitations if applied to environments with a larger number of agents?\n\n- On line 205, it is mentioned that the text representation “includes the knowledge of the player’s own hand.” Could the authors clarify if this refers to the player’s hand as revealed by others through clues up to the current time point?\n\n- Are the baseline algorithms (such as OP and OBL) trained on the text-based version of the Hanabi environment or on the original bitstring encoding? If not on the text-based version, could the authors address how this may impact the fairness of the comparison?\n\n- Given the different architectures and representations used for each baseline, why were all agents trained with identical hyperparameters (as noted on line 374-375)? Could the authors discuss whether hyperparameter tuning was considered for each model to ensure the most accurate and fair evaluation of baseline performance?\n\n- The explanation for OP’s lower performance on line 410 is somewhat vague. Could the authors provide additional insights or analysis on why OP underperformed relative to R3D2, as well as a more detailed comparison between R3D2 and OBL?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper considers the important problem of enabling agents to collaborate across different player configurations and game settings, a key challenge for advancing multi-agent reinforcement learning. This focus addresses a critical gap in MARL research, as agents must be able to generalize and coordinate flexibly in dynamic, multi-agent environments, an area relevant to both academic research and real-world applications.\n\n- The key contribution of the paper can be seen as creating a text-based version of Hanabi game. Consequently, the exploration in Section 5 of how current language models perform in the Hanabi environment is a valuable addition, as it highlights the capabilities and limitations of large language models (LLMs) in multi-agent coordination tasks. This analysis opens pathways for further investigation into LLM limitations, potentially contributing to future improvements in language model-based reinforcement learning.\n\n- Within the Hanabi environment, the results indicate that the proposed R3D2 approach achieves effective transfer of strategies across different game settings, demonstrating the agent’s adaptability to variable player configurations.", "weaknesses": "- The authors make several broad claims in the introduction about multi-agent reinforcement learning (MARL) and the adaptability of agents, but these claims lack supporting evidence or citations. For example, in the following statements: “Artificial agents should do the same  for successful collaboration of artificial and hybrid systems” (line 34-35), “… players are required to infer the beliefs and intentions of their counterparts through theory of mind reasoning” (line 42-43), and “…resulting in misunderstandings and thus a drop in cooperation …” (line 48-49), the authors do not provide empirical data or references to substantiate these claims, limiting the rigor of their introductory motivation.\n\n- The paper does not sufficiently establish why Hanabi is an appropriate testbed for studying adaptability to new settings and collaborators, nor does it explain how findings in Hanabi might translate to real-world collaborative tasks. The link between Hanabi’s in-game interactions and broader applications remains unclear, and this omission leaves open questions about the broader relevance and applicability of the research.\n\n- While the authors cite work by Hu and Sadigh as justification for their choice of Hanabi and as a baseline for comparison, this prior research does not fully support the tasks explored here. A central focus of Hu and Sadigh’s work was on human-AI coordination, which this paper does not directly address. This gap makes the comparison less relevant, as the primary objectives differ significantly.\n\n- The authors critique Hu and Sadigh's approach for requiring an explicit specification of the expected behavior of the learning agent. However, the proposed R3D2 method also requires specific environment settings, which could present a similar or greater burden. The need to configure a consistent environment specification might, in some ways, be less useful than specifying interaction strategies, as discussed in later sections.\n\n- A significant oversight in this paper is the lack of reference to the extensive body of literature on unsupervised environment design in both single-agent and multi-agent settings [1,2]. This research area directly addresses the problem of adaptability and generalization in agent training, which the authors claim to investigate. For the paper’s contributions to be meaningful within the field, it is essential for the authors to engage with this related work and to include comparative empirical evaluations.\n\n- While creating a text-based version of Hanabi contributes to the approach, it is unclear how this addresses the problem of adaptability that the authors set out to solve. Converting the environment to a text-based form, and leveraging symmetries within Hanabi, simplifies many of the original environment’s complexities. As a result, adaptability may not be fully tested here since the text-based version removes many elements that would typically require generalization, making it a fundamentally easier learning problem.\n- The approach also necessitates adapting the replay buffer to handle a variable number of players, which could be a limiting factor. Although this modification works here because Hanabi caps players at five, it may not scale effectively for environments requiring more flexibility or involving larger numbers of agents.\n\n- The empirical evaluation of collaboration with other agents is limited to two-player settings, as the other agents require training specific to each number of players. This narrow scope restricts the evaluation, as it remains unclear how these methods would perform in settings with more players. Additionally, because the baseline agents are not trained using a text-based representation, this could lead to an unfair comparison between approaches.\n\n- The authors indicate that all agents were trained using identical hyperparameters (line 374-375), despite the agents having different architectures and representations. Given these differences, it is crucial to tune hyperparameters individually for each agent to ensure a fair comparison, as uniform parameters may disadvantage some agents more than others.\n\n- The explanation given for the relatively low performance of OP (line 410) lacks clarity, and the observed performance of OBL seems promising in comparison to R3D2. However, the current experimental setup does not provide sufficient data to substantiate why R3D2 should be chosen over OBL or how these results might generalize across settings.\n\n- The passage on lines 422-425 presents an unclear and convoluted explanation of the comparison between methods, and the reported marginal improvements in R3D2’s performance relative to other settings are not fully justified within the results.\n\n- The insights gained from the results presented in Figure 6 are unclear. Given that R3D2 was trained across all game environments, it is unsurprising that it performs adequately across settings. Without a clearer interpretation, the results lack novelty, as the outcomes align with the fact that R3D2 has been exposed to all environments during training.\n\n[1] Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, Dennis et. al. NeurIPS 2020\n\n[2] MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning, Samvelyan et. al. ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PuDHsmFd0d", "reviewer": "Reviewer_Axwq", "summary": "This paper presents a scalability study of LLMs in the context of tackling Hanabi, an imperfect information cooperative game. The techniques of theory of mind, as well as solving a smaller game robustly then scaling to larger games is used. The work is \"okay\" in its current state but can be improved. See detailed comments below.", "questions": "1: Can the authors modify their presentation somewhat to make their approach more clear? See above.\n\n2: In particular can a better presentation be provided regarding the role of language and Theory of Mind for making the imperfect information portion of Hanabi into perfect information portion.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1: LLMs and theory of mind seem to be the right platform to address the game size of Hanabi.\n\n2: Cooperation with unknown or novel agents seems to be a strong problem that this work addresses, and very much applies to Hanabi.\n\n3: Using robustness to tackle (2) seems to be a reasonable path forward.\n\n4: Using LLM's to \"encode\" the state space of Hanabi does seem to significantly improve the ability to solve the game.\n\n5: Using dot product to create a fixed length embedding for state-action pairs does help in going from solving small 2-player games to 5-player games, for example.\n\n6: The validation is sufficient for the claims that are being made in the work.", "weaknesses": "I'm not opposed to accepting this work in its current state, however I wonder whether we can make the approach more straightforward to understand.\n\n1: The key probelm we're solving is: Hanabi is an imperfect information game where the information is held by other players. This is the unique challenge of Hanabi.\n\n2: Whether you're looking at Information Sets or some sort of statistical model in order to turn partial information games into perfect information games, the standard approach remains the same: reduce the imperfect information game to a perfect information game in some way, then solve it using traditional approaches.\n\n3: If you look at the historical progression of solving games, it often seems to be about scalability. See for example Deep Blue being a showcase of tree search, AlphaGo about the success of using CNNs to understand the game state, then using a combination of techniques (including MCTS) to solve it.\n\n4: So again, it seems that the key theoretical insight here is: how to reduce an imperfect information game into a perfect information game through Theory of Mind.\n\n5: The key shortcoming here is I'm not really sure what Theory of Mind is, nor are many people familiar with the \"inherent\" properties of languages which are undisputed. \n\n6: Given (5), it's difficult for me to view this paper as something much more than a scalability study of LLMs.\n\nA more alternative presentation of the work would be, communication is used to reduce imperfect information to perfect information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kcQcg6sLL1", "reviewer": "Reviewer_w4wV", "summary": "This paper focus on Hanabi benchmark, and devises a MARL algorithm which combines deep recurrent relevance Q-network and language models to build Hanabi agent. The proposed algorithm can deal with resulting dynamic observation and action spaces based on the text-based encoding, and thus can play all game settings on Hanabi. Finally, the experiments ranging from 2-player to the 5-player are performed to validate the effectiveness of the proposed algorithm.", "questions": "1. The authors state that they utilize GPT-4 and LLama-2 to develop action strategies and claim that the results \"struggle with optimal planning\" in Section 5. However, I could not find any related experimental results shown in the paper, including the appendices.\n2. When introducing the results about fine-tuning LLama-7B, the authors only conduct experiments on different data sizes and LoRA rank settings, without comparing the results to their proposed method. Therefore, how was the conclusion that \"the model performs poorly\" derived?\n3. The purpose of Section 5 is unclear. If the authors want to justify choosing BERT instead of other language models, they should replace the BERT structure with other LMs in their approach and show the experimental results in an ablation study, rather than listing results that only depend on LMs.\n4. I do not believe the advantage of generalizing all game settings comes from the dynamic network structure. Rather, it appears to stem from language models' ability to describe different environmental information and encode it without dependence on environment-specific characteristics, correct?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of incorporating all game setting together during trainging is interesting, as many current MARL algorithms need to retrain the policy network when the game setting changes.\n2. The writting is clearly, and the structure is well organized.", "weaknesses": "1. This paper only focuses on the Hanabi game. Whether the proposed algorithm can be scaled to other domains or environments is a critical question.\n2. The claimed first contribution of \"framing Hanabi as a text-based game\" appears weak. Previous work [1] has already devised text-based observations and actions for the Hanabi game. Additionally, the authors need to include comparative experiments.\n3. The explanation of how R3D2 can support environments with varying numbers of players is insufficient.\n4. The open-source code will help other researchers reproduce and build upon this work.\n\n[1] Hu H, Sadigh D. Language instructed reinforcement learning for human-ai coordination[C]//International Conference on Machine Learning. PMLR, 2023: 13584-13598.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e1sxq3FrUL", "reviewer": "Reviewer_YKqN", "summary": "The paper proposes a method based on language model agent for the multi-agent Hanabi game. The motivation lies in transforming Hanabi to a text-based game. The key idea lies in using BERT to provide embeddings of the text information in the game, and to be jointly trained with the policy model using Q-learning. The major target of the proposed method is to obtain a policy which can generalized to various game settings with different number of players and strategies.", "questions": "- How to explain the phenomena in Point 1 and 2 mentioned in the weaknesses?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated for building a general multi-agent game play agent. I agree that the generalization issue is a fundamental one in reinforcement learning. \n\n2. The design of conducting cross-strategy and cross-setting experimental results is interesting to me.", "weaknesses": "1. Even though the proposed method is armed with language models, its performance seems not desirable. From Fig. 5 and Fig. 6, its performance does not outperform classical methods IQL and the more recent one OP. This remains in doubt about its effectiveness.\n\n2. Even though the paper pursues a meaningful goal, the final results provide limited insights. From Fig. 5 and Fig. 6, they show that IQL and OP have better generalization ability than the proposed method. This shows that the proposed method does not provide a sound solution to the generalization problem.\n\n3. In my view, the major promising characters for language models to perform in game playing are knowledge providers and reasoners. While in the proposed method, the language model only plays the role of modeling the text embeddings. This usage is somehow not quite novel and effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ShVcCEpN3S", "reviewer": "Reviewer_5SDa", "summary": "The authors present a new LLM based MARL algorithm, R3D2, capable of playing Hanabi and generalize to a changing number of players by using a text observation. R3D2 is also capable of generalizing to cooperation with unseen players, similar to previous work on Hanabi.", "questions": "Minor typo comments:\n- Some citations should be in parentheses (using \\citep{}), for example in section 3.2.\n- Sections 6.2, 6.3 have a typo in the title (shot --> short)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The paper tackles a complicated MARL problem, where generalization is hard due to the strong reliance on collaboration to achieve high scores. The authors propose a novel algorithm that utilizes the generalization capabilities of language models to boost agent generalization. Using text input allows agents to generalize to different rules (more/less players), a notorious challenge in RL in general. The new algorithm seems on par with previous works on Hanabi when generalizing to new players.", "weaknesses": "The paper's main issue is the clarity of its claims and comparison to prior literature.\n\nThe paper is a bit unclear on the contribution of the algorithm presented, R3D2. Since prior works have already achieved Zero Shot Coordination (ZSC) between training seeds and between AI and human players, it seems like the only novelty in R3D2 is the ability to generalize to new rules, specifically changing the number of players. However, the results on generalizing to novel settings are only briefly discussed at the end of the paper in Section 6.2, and in general are presented more as a minor perk of the model. \n\nReading the Conclusions section gives the impression that the R3D2 is the first algorithm to achieve a high intra- and inter-algorithmic cross-play score, which it is not.\nThe authors claim \"R3D2’s intra-algorithmic cross-play score is on par with its self-play score, a first for Hanabi agents learning through self-play\". While this may be true for self-play agents, the OBL algorithm also achieves equivalent cross- and self-play scores (that are higher than R3D2's scores).\n\nIn contrast, the abstract does a good job of clarifying what the paper's novelty is.\n\nIn general, the paper should be more clear about what R3D2 does not achieve, e.g. SOTA in inter-algorithm play, rather than give the impression that R3D2's main contribution is SOTA performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hRyAooEAJX", "reviewer": "Reviewer_HW8S", "summary": "This work focus on the challenging card game Hanabi and try to solve the current  bottleneck of transferring of learned policy to different settings. Language representation is selected for adapting policies among these settings. A successful zero-shot coordination architecture for MARL has been proposed and achieved sota scores on the Hanabi game, which is among the early solutions for such settings.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is a worthwhile trial to use language modelling to solve the MARL transfer ability issue, which is inspiring. In experiments it also applies to some newly proposed LLM models which shows steady improvement.", "weaknesses": "1. More up-to-date models like LLAMA 3 or GPT-4o is also encouraged to ensure the experiments are more solid.\n2. Is it possible to transfer the learned policy to a different task? How about comparing the proposed approach to existing schemes like curriculum learning or multi-task MARL? Related work should also include such literature.\n3. Quantitative results are given in the paper, it is better to include more qualitative demos which show how the language representation helped in transferring.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tsGLjG2p5x", "reviewer": "Reviewer_2tFM", "summary": "In this paper, the authors focus on playing the game of Hanabi with multi-agent reinforcement learning. They consider the shortcomings of existing MARL systems: unable to perform well on any other setting than the one they have been trained on, and struggle to successfully cooperate with unfamiliar collaborators, which is particularly visible in the Hanabi benchmark. To solve this, they introduce a generalist agent for Hanabi. The reformulate the task using text and then propose a distributed MARL algorithm that copes with the dynamic observation- and action-space. By doing so, they train an agent that can play all game settings concurrently, and can extend strategies learned from one setting to other ones. Also, there agent demonstrates the ability to collaborate with different algorithmic agents.", "questions": "1. Why is the opinion of 'language has been shown to improve transfer' is valid? Can you provide experimental results to support this claim? For example, how would the performance change if natural language inputs and pre-trained language models were replaced with vector inputs and standard neural networks? Alternatively, could you provide more solid literature to support this idea?\n2. What is the relationship among the evaluation metrics SP, Intra-XP, and Inter-XP? Figure 4 shows that R3D2 achieves significantly better Inter-XP compared to the baselines, but the SP metric shows a significant drop, even larger than the improvement in Inter-XP. How should these three metrics be interpreted comprehensively to determine which method performs best?\n3. This paper focuses on the Hanabi game. Could you discuss whether the proposed method can inspire improvements in MARL algorithms for other tasks?\n4. Could you make the code open-source at an appropriate time and provide more detailed information on the experimental setups to help others reproduce the results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors analyze the widespread problems of common MARL algorithms and select Hanabi, a typical scenario reflecting the common problem, for improvement and research.\n2. The authors template the observation and action of the Hanabi game into natural language and use the pre-trained LM as the representation head in the RL value network. They try various pre-trained LMs and finetuning techniques to compare the performance.\n3. The authors conduct extensive experiments to demonstrate the zero-shot capability of their proposed method on 2-player coordination and policy transfer over different settings.", "weaknesses": "1. The general motivation of this work, if I don't understand wrong, stands on the opinion of 'language has been shown to improve transfer'. Therefore, the authors propose to template the observation and action of the Hanabi game into natural language and process them with pre-trained LM. But why this opinion holds? It will be more convincing to provide experimental results to show this. For example, how will the performance change if substituting the natural language inputs and pre-trained LM with vector inputs and normal neural networks? Or at least, the authors should provide more sound literature to support this opinion.\n2. The relationship among the three evaluation metrics, i.e. SP, Intra-XP, Inter-XP, is confusing. The authors claim in Figure 4 that R3D2 achieves significantly better inter-XP compared to the baselines while maintaining a competitive SP and intra-XP. However, the figure indicates significant drop on SP, which is even more significant that the improvement on inter-XP. How should I comprehensively understand these three metrics to determine which method performs best?\n3. This paper focuses on the Hanabi game and conducts an in-depth study, but it is still unclear whether the proposed method can inspire the improvement of MARL algorithms on other different tasks.\n4. The experimental details provided in this paper are limited, making it difficult for readers to directly reproduce the results. It would be great if the authors can make the code open-sourced at an appropriate time and provide more details on the experimental setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "caxAaOnXcC", "reviewer": "Reviewer_k3Nj", "summary": "The paper aims to train a generalist Hanabi agent that can:\n1) Play variants of Hinabi with different number of co-players \n2) Play with novel co-players\n\nTo this end the authors contributions are:\n1) A text based version of hinabi environment - which makes learning across multuple variants easier\n2) A novel-ish algorithm R3D2, which is a recurrent network with a value function generated from a text embedding.", "questions": "1) “it successfully avoids getting bombed for extended periods, “ <- is this a common term in hinabi?\n\n2) The Appendix is missing.\n\n3) The Inter-XP metric is calculated with which other co-player? Figure 4.\n\n4) Could analysis be provided of the compute costs / sample efficiency and limitations of each method? Its really hard to work out why this method is working at all, is this just better domain randomisation, is there some particular underlying set of hinabi skills that generalise well over n players.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The problem is original - multi-task / multi-agent problems have not been tried since the XL land paper or Maestro (both citations which are missing in this work). Although this does truly feel like the limit of how much as a community we can care about hinabi. \n\nI’m a massive fan of the evaluation protocol in this paper - its really nice to see cooperative marl analysis on cross-play and the introduction of intra-xp and inter-xp is a huge improvement. \n\nThe paper is well written and to a high quality.", "weaknesses": "The choice of baselines is really confusing to me, and makes it hard to understand the authors contribution. All baselines presented are trained and evaluated in the existing hinabi environment (the non-text based encoding). This makes it hard to disentangle the importance of the R2D2 style training, the RRN modification or the language based component underneath.  To my understanding, the method isn’t really novel - this is just R2D2 but they used the text embedding for both goals and actions - which is just a trick from the Continuous Control Literature and augmented the Recurrent network to be a recurrent relevant network. Also using a text based version of Hinabi is obvious in 2024, so i don’t see this as a significant contribution. Ablations would be appreciated to understand where the juice is actually coming from. I’d also like to just know if OBL agents in the language based environment generalize better. \n\nThe results lack qualitative analysis and as such, its really hard to understand whats happening here or the significance of the work, Is the hope that the method has learnt via a larger set of environments / tasks to have more robust policies.  If so what do these policies look like? Do they share underlying similar structures or methods. Could we also evaluate methods on OOD tasks (e.g 7 players hinabi or something not in the train?).\nIts hard to interpret the results in Figure 6 - yes it seems R3D2 is robust to co-players changing but because i don’t have an equivalent evaluation with a team of IQL agents or OBL agents its very hard to evaluate if this is comparatively better.\n\nThe paper would be vastly improved if consistent LLMs were used throughout the section. LLMs have quirks, different training data and different methods. It feels hard to compare results in Section 5 with any proposed in Section 6 because the underlying models are very different. I also think that the experiments in Section 5 tell us more about those models (which are rather small number of parameters) and the difficulty of supervised learning here. The conclusions drawn are overreaching.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eegxuoVOJg", "reviewer": "Reviewer_u5D6", "summary": "This paper learns a generalist agent that is able to play any-card variants of Hanabi by casting the game to a text based game and utilize model architectures from language modeling to learn such policy with Q-learning", "questions": "1. \nWhat are the agents on the x-axis of Figure 6? Are they R3D2-k agents trained for each game variant? If so, why is the diagonal not the same as R3D2-SP? The notation and explanation around this figure could be improved. For example, you are using XP to refer to cross-play everywhere else, but use CP here, which could be a bit confusing.\n\n2. \n> In contrast, in our textual representation, symmetrical observations are the same but for a few tokens. Symmetries are thus implicitly tackled by R3D2.\n\nWhy is this the case? For example, considering two hands R1, R2, B5, B3, G4 and B1, B2, R5, R3, Y4 (R, G, B, Y for different colors), these two hands are symmetrical up to a color permutation used in OP, but they are more than a few tokens apart in the language description. The color permutation will also apply on other parts of the observations such as the firework pile so I would expect the language observation to be quite different? Am I missing something? If I did not miss something, I would be curious why the policy does not learn to use arbitrary color coding as a play strategy.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of casting Hanabi as a text game for the purpose of learning a generalist agent is interesting. Language as a common interface for variants of games, or completely different games, is interesting, especially when the game primarily requires reasoning, which is the case in Hanabi.\n\n2. If I understand correctly, the R3D2 agent is able to learn consistent policies, i.e. achieving high intra-AXP score and small gap between intra-AXP and self-play, without any additional assumptions that prior methods use. This is a cool result and probably the benefit of using language as observation and action? Although I am not sure exactly why this model is able to do that. See my question below.", "weaknesses": "1. Although this paper does a good job showing that we can train a single generalist agent for any-player Hanabi, which is an interesting contribution, one downside is that the paper is not able to have convincing results of the benefit of having a “generalist” agent, i.e. the benefit of going from R3D2-S to R3D2-M. Apart from the fact that the generalist agent does better in Figure 6, which it should because the single-variant agents are not trained on other variants of Hanabi, I only found this single comment on the benefit of R3D2-M:\n> Interestingly, R3D2-M learns decent strategies for all settings, despite receiving the same training budget as single-setting algorithms. This shows that knowledge from one setting is useful for other ones, and that the textual representation allows for effective transfer of kenowledge across settings.\n\nHowever, I don’t find this to be convincing. If the generalist agent is given the same amount of data as the sum of the 4 single-variant agents, will it outperform the single agents? I think this is crucial for making such a claim and I do believe that would make this paper stronger to the general audience beyond MARL.\n\n\\\n2. The high inter-AXP results are not super exciting because:\n* IQL is known to produce strange conventions that is hard to coordinate with;\n* OP has significantly higher inter-AXP score with IQL than any other methods, indicating that the reproduced OP is probably as brittle as the IQL agents, which is totally possible as some OP implementation uses tricks such as hiding the last action from the observation to regularize more;\n* OBL coordinates badly with the OP agents, indicating that they are very different;\n\nWith the arguments above, a high inter-AXP score here would just translate to better coordination with  IQL/OP than OBL, which is a useful result but not necessarily an important result because eventually we want policies that coordinates better with human-like policies (OBL) rather than inhuman policies (IQL). Having a human study as in [1], a paper in a comparable conference (ICML) on Hanabi that this paper cited, would greatly address this.\n\n[1] Language Instructed RL for Human-AI Coordination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)