Ad-Hoc Human-AI Coordination Challenge
Authors: Tin Dizdarević
Ravi Hammond
Tobias Gessler
Anisoara Calinescu
Jonathan Cook
Matteo Gallici
Andrei Lupu
Jakob Nicolaus Foerster
Venue: ICML 2025 spotlightposter (2025)
Keywords: multi-agent reinforcement learning
reinforcement learning
multi-agent systems
human-ai coordination
cooperative
challenge paper
Primary area: reinforcement_learning->multiagent

Abstract: Achieving seamless coordination between AI agents and humans is crucial for real-world applications, yet it remains a significant open challenge. Hanabi is a cooperative card game featuring imperfect information, constrained communication, theory of mind requirements, and coordinated action -- making it an ideal testbed for human-AI coordination. However, its use for human-AI interaction has been limited by the challenges of human evaluation. In this work, we introduce the Ad-Hoc Human-AI Coordination Challenge (AH2AC2) to overcome the constraints of costly and difficult-to-reproduce human evaluations. We develop \textit{human proxy agents} on a large-scale human dataset that serve as robust, cheap, and reproducible human-like evaluation partners in AH2AC2. To encourage the development of data-efficient methods, we open-source a dataset of 3,079 games, deliberately limiting the amount of available human gameplay data. We present baseline results for both two- and three- player Hanabi scenarios. To ensure fair evaluation, we host the proxy agents through a controlled evaluation system rather than releasing them publicly. The code is available at \href{https://github.com/FLAIROx/ah2ac2}{https://github.com/FLAIROx/ah2ac2}.

Ad-Hoc Human-AI Coordination Challenge
Tin Dizdarevié' 1
Ravi Hammond
Tobias Gessler'
Anisoara Calineseu 2 Jonathan Cook' Matteo Gallici
Andrei Lupu'
Jakob Nicolaus Foerster
Abstract
making in healthcare (Asan et al., 2020) to shared control in autonomous vehicles (Bansal et al., 2018) and robotics (Haamoja et al., 2018; Ahn et al., 2024) as well as advanced digital assistants (Gemini Team, 2024; Bai et al., 2022). The ultimate goal is to create AI agents that are not limited to solving problems independently but can also work effec- tively with humans to complete tasks in human-compatible ways (Russell, 2019; Carroll al., 2019).
Achieving seamless coordination between Al agents and humans is crucial for real-world ap- plications, yet it remains significant open chal- lenge. Hanabi is a cooperative card game fea- turing imperfect information, constrained com- munication, theory of mind requirements, and coordinated action making it an ideal testhed for human-Al coordination. However, its use for human-AI interaction has been limited by the challenges of human evaluation. In this work, we introduce the Ad-Hoc Human-Al Coordina- tion Challenge (AH2AC2) to overcome the con- straints of costly and difficult-to tofieuul-tt-reroduuce hu- man evaluations. We develop human proxy agents a large-scale human dataset that serve as ro- bust, cheap, and reproducible human-like eval- uation partners in AH2AC2. To encourage the development of data-efficient methods, we open- source a dataset of 3,079 games, deliberately limiting the amount of available human game- play data. We present baseline results for both two- and three- player Hanabi scenarios. To en- sure fair evaluation, we host the proxy agents through controlled evaluation system rather than releasing them publicly. The code is available at https://githu..com/ wwt../ooo///AA///eeeee.e...
Traditional approaches to training Al agents in simulation frequently employ self-play (SP) where agents train under a joint policy that controls the strategies of all players (Samuel, 1959; Tesauro, 1994). While SP has been successful com- petitive games like chess and Go (Silver et al., 2016), in cooperative settings this approach can lead to agents that overfit to specific strategies, limiting their ability to gener- alise to novel partners (Carroll et al., 2019). In human-Al coordination scenarios, forming rigid conventions is partic- ularly problematic, as doing so can pose safety risks where humans are unable to adapt appropriately (Bard et al., 2019).
Despite the growing importance of human-Al coordination, the field lacks standardised benchmarks that accurately re- flect the complexities of interacting with humans in complex, partially observable settings. Existing evaluation methods often rely on closed datasets and proprietary proxy agents (Bakhtin et al., 2022; Jacob et al., 2022), which hinder re- producibility and broad-basee progress. Without accessible and robust benchmarks, it is challenging to measure ad- vancements consistently. Furthermore, the scarcity of open- source human-Al AI coordination datasets limits the ability of researchers to develop and test innovative, data-efficient al- gorithms that are essential for real-world applications where large-scale human data may not be readily available.
1. Introduction
Human-Al interaction is rapidly advancing due to signifi- cant Al progress and its growing integration into daily life (Rawas, 2024). Effective coordination between humans and Al in complex settings becomes crucial as AI agents become more sophisticated, capable, and prevalent. This interaction spans a wide range of domains, from collaborative decision-
Hanabi is an established, fully cooperative benchmark envi- ronment that involves imperfect information, limited com- munication, theory of mind, and the necessity for coordina- tion among different players to achieve shared goal (Bard et al., 2019). These characteristics make Hanabi solid testbed for evaluating human AI coordination. While previ- ous research Hanabi has used held-out sets of human data and human proxy agents to evaluate human-AA Al coordination (Hu al., 2022: 2021: Lupu al., 2021). these datasets and proxy agents have thus far remained closed-soure and un-
FFLAIR. University of Oxford. Oxford, UK Department of Computer Science, University of Oxford, UK Uniierssittt Politècnica de Catalunya, Barcelona, Spain. Correspondence to: Tin Dizdarević <tin @ Stin@robots.ox robots.ox.ac.ukk
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Ad-Hoc Human-AI Coordination Challenge
available to the wider research community. To address these problems, introduce the Ad-Hoc Human-Al Coordina- tion Challenge (AH2AC2) as standardised way to evaluate human-Al coordination in Hanabi. Specifically, we develop human proxy agents through a combination of behavioural cloning (BC) and regularised reinforcement learning (RL). We first train the BC component on a large-scale dataset of human gameplay, comprising 101,096 two-player games and 46,525 three-player games from the hanab.live com- munity. We then refine the BC policy using Independent Proximal Policy Optimisation (IPPO) (de Witt et al., 2020; Schulman et al., 2017) with a regularisation term that encour- ages adherence to human-style play (Bakhtin et al., 2022; Hu et al., 2022: Cornelisse & Vinitsky, 2024). Our empirical evaluation shows that these human proxy agents outperform pure imitation learning while maintaining human-like be- haviour. These human proxy agents serve robust, cheap and reproducible evaluation partners in AH2AC2.
dination; these baselines collectively highlight the signif- icant difficulty current methods face in building human- compatible agents for partially observable environments.
2. Background
Dec-POMDP We consider a decentralised par- tially observable Markov decision process (Dec- POMDP) (Oliehoek, 2012), defined (S, n. {A') Ji-1. { T. R. (=: the finite state space and n is the number of agents. A' and O' are the local action and observation spaces for agent i, and A == xj=1A', O = , are the joint action and observation spaces. The transition function T : S x A x S - [0, 1] defines the probability of transitioning ae (o) ..... 0f) in state St- The agents receive the reward T++1 = R( and agent i receives the local observation 01+1 with probability LI' (s at, o1+1). 7 E [0. 1] is the discount factor, and H is the horizon. i.e. sH is always a terminal state.
9-tuple S is
state Sc+1 when taking joint action
To ensure AH2AC2 evaluation integrity and to prevent over- fitting by the community, we withhold public access to the human proxy agents and their large-scale training dataset, and only open-source a limited dataset for both two- and three-player settings. This also encourages research on methods that are data-efficient with respect to human data. We also provide various baselines: first, zero-shot coordina- tion methods such as Off-Belief Learning (OBL) (Hu et al., 2021), which operate without human data; second, data- dependent approaches such as best response to behavioural cloning policy (BR-BC) (Carroll al., 2019): third, the first evaluation of Fictitious Co-Play (FCP) (Strouse al., 2021) in Hanabi, which serves bascline for population-based methods; fourth, we develop DeepSeek-RI (DeepSeek- Al ct al., 2025) Hanabi agent, to provide insights into the current capabilities of Large Language Models (LLMs) for Auman-Al coordination.
The local cetionnobservaatonn history (AOH) of player i is defined as 9द = (ah of... AOH is defined as .......,-j1 - of-1) of), and the joint (r), ..., r"). Each player i se- lects a local action of according to a local policy n' (ajlr) The joint policy π = (x',..., then selects joint action O with probability ((elTr)) = I 7' (ailrt)) Given a joint policy n, the expected return is defined as [(x) = Ex [H 3rr441-
Zero-shot Coordination and Ad-Hoc Teamplay In many real-world scenarios, Al agents must coordinate with unseen partners, including humans. Traditional cooperative multi-agent RL uses SP training, where agents train together to maximize the expected return (Bard et al., 2019; Carroll et al., 2019). However, SP training often leads to special- ized communication protocols that fail with independently trained agents, including humans. Thus, training in SP is not a solution to the challenges of human-Al coordination (Carroll et al., 2019; Strouse et al., 2021). Zero-shot coordi- nation (ZSC) addresses this by training agents collaborate effectively with new partners which were trained with the same algorithm (Hu et al., 2020; Treutlein et al., 2021).
In summary, our key contributions are:
The first open-source Hanabi human gameplay dataset, containing 1,858 two-player and 1,221 three-player games.
High-performing human proxy agents for both two-player and three- pare-ppayer Hanabi settings, using BC on a closed- source large-scale human play dataset (over 100k games) combined with regularised RL.
Our evaluation protocol, where we host proxy agents behind an API, paired with a public leaderboard to track research progress. API access requires the pre-registration of experiments, gold standard for empirical research.
Ad-hoc teamplay assesses an agent's ability to cooperate with unfamiliar teammates at test time (Stone et al., 2010). Like ZSC, agents are evaluated with partners they have not trained with, but unlike ZSC, teammates may use differ- ent training algorithms. In this work, we focus on ad-hoc teamplay involving human and human-like agents.
A comprehensive set of baselines for both two- and three- player settings, spanning zero-shot coordination, data- dependent, and population-basee methods, alongside the DeepSeek-RI Hanabi agent, which specifically provides an assessment of L.M performance in human-Al coor-
Hanabi Hanabi is a cooperative card game where players can see the cards in each other's hands, but not in their own,
2

---

Ad-Hoc Human-Al Coordination Challenge
Closed Source
3 Human Proxy Agents
Ad-Hoc Teamplay with Human Proxy Agents
Public Leaderboard
Large-Scale Human Play Dataset
Subset of the Large Scale Human Play Dataset
Community Developed Human Compatible Agents
Evaluation API: Grants
| Baseline Agents: Code & Weights
Limited Access to Human for Evaluatiuon
Provided Participants
Provided to Participants
Figure Ad-Hoc Human-AI Coordination Challenge (AH2AC2)
and thus rely on others to give interpretable hints on which cards play or discard. It is designed for 2-5 players, but restrict ourselves to two- and three- player settings in this work due to the availability of human gameplay data for these configurations. The maximum score in the game is 25, but it cannot be achieved for every shuffling of the deck. If the team of players makes three mistakes in total, the game ends with a score of O. For more details on Hanabi we refer Appendix A.I.
specialised form of ad-hoc teamplay, where the set of test policies comprises human strategies, which are inherently valuable due to their real-world relevance.
Ad-hoe human-Al coordination in Hanabi is explored in many previous works, with each presenting a different methodology for acquiring human proxy agents and evalu- ating human-Al coordination capabilities (Hu al., 2021; 2020; Cui et al., 2023). However, there no standard ap- proach for ad-hoc human-Al coordination evaluation in the existing literature. Therefore, work addresses this gap and proposes the AH2AC2.
3. Related Work
Hanabi was introduced as a benchmark encompassing both SP and ad-ho hoc teamplay (Bard et al., 2019). but significant progress has been primarily confined to SP, with methods like SPARTA (Lerer et al., 2019) achieving near-perfect 24.61/25 points on average. However, agents trained in SP often rely on specialised conventions, leading to poor gener- alisation when paired with novel teammates. Consequently, ad-hoc team play, especially with humans, presents a more demanding and unsolved challenge.
Recent works have empirically shown that augmenting imi- tation learning methods with regularised RL creates stronger and more reliable policies. Multiple works have shown that regularised RL leads policies that are more compatible with existing social conventions of the human reference group (Jacob et al., 2022; Bakhtin et al., 2022; Hu et al., 2022). Recently, (Corneliss & Vinitsky, 2024) extended these works to the driving setting, where the authors show that data-driven regularisation leads to human- compatblle policies. Our work builds upon these findings to build strong and human-likk human proxy agents that enable AH2AC2.
central issue when evaluating agent abilities for ad-hoc coordination the selection of test-time partners. One approach to tackle this issue is to ensure the diversity of policies, thereby minimising the possibility of favouring any specific agent. For example, Cui et al. (2023) intro- duce ADVERSITY that aims to produce highly skilled and reasonable policies that play according to diverse conven- tions. Another approach is to focus on a set of policies that hold intrinsic value, with the most natural choice being human policies. Coordinating with humans can be seen
Nekoei al. (2023) urge the multi-agent RL community to address few-shot adaptation alongside zero-shot coor- dination (ZSC). They show that current ZSC algorithms struggle to adapt to new partners. To connect AH2AC2 with few-shot coordination, we introduce data-limited settings. Unlike traditional few-shot scenarios, our approach involves offline, one-sided adaptation: human proxy agents maintain fixed behaviour during testing, and we use small sample of
3

---

Ad-Hoc Human-AI Coordination Challenge
human play data for training. This emphasizes the challenge of coordinating with human-like agents, where only the Al agent adapts due to the constraints of human behaviour (Stone et al., 2010).
4.2. Evaluation Protocol
PART OF AH2AC2: COORDINATION WITH HUMAN PROXIES
We develop four human proxy agents for evaluating ad- hoc human-Al coordination: two for two-player Hanabi and two for three-player. These agents are trained using Daamann-Daa-RRgulaiiiddd IPPO (HDR-IPPO), a procedure combining BC and regularised IPPO (de Witt et al., 2020). First, BC policies are trained on a large-scale dataset of human gameplay 101,096 two-player games and 46,525 three-player games. Because BC alone struggles gener- alise to unseen game states (Carroll et al., 2019; Hu et al., 2022; Bakhtin et al., 2022; Cornelisse & Vinitsky, 2024), we then refine them through regularised SP using IPPO. The regularisation ensures the final policies remain close to human play styles. We provide further details regarding PPO and IPPO in Appendix A.2.
4. Ad-Hoc Human-AI Coordination Challenge (AH2AC2)
This section outlines our proposed challenge, which consists of two key evaluation regimes: (a) evaluation with a set of human proxy agents. This requires participants to develop human-compatible agents based on a small provided dataset of human gameplay, and (b) a human action prediction task on an unseen, closed-source set of games.
4.1. Methodology Overview
We collected a comprehensive dataset from the hanab.live platform, comprising of 101,096 two-player games and 46,525 three-player games. All games adhere to H-group conventions, a set of hand-crafted strategies used by Hanabi players on hinab.live. It is important to note that H-group Conventions are not a single strategy. Instead, it is helpful to think of H-Group Conventions as a collection of different strategies and techniques that players learn and combine. Players often mix and adapt these strategies within a sin- gle game, depending on the players' strength. Therefore, the dataset itself naturally contains a variety of playstyles. Therefore, the use of H-Group Conventions as founda- tion does not overly constrain the strategic variety of our human proxies. Details about data composition and splits are provided in Appendices A.5 and A.4.
When training human proxy agents using HDR-IPPO, we learn parameterised local policy, denoted as T HP. As first step, we train local BC policy T.CC which, given Hanabi's discrete action space, translates into a classification task: the features are local AOHs t/, and the labels are ground truth local actions ai+1 given.
To capture the sequential nature of the actions and observa- tions, model x through LSTMM-aased architecture (Hochreiter & Schmidhuber, 1997). Critically, fixed neural parameters do not imply static behaviour; our proxies con- dition on the history of the game, which includes partner actions. When the proxy agents see an unexpected action or observation, from that action-observation history onward, they will account for the fact that the other agent using a different convention. We train the BC model through super- vised learning, minimising the standard cross-entropy loss between the predicted action distribution and the ground truth human actions.
As part of the AH2AC2, we open source 3,079 games from the large-scale dataset- - 1,858 two-player and 1,221 three- player games. Participants are allowed to use these open- sourced games when tackling the AH2AC2. Key statistics of the open-sourced dataset are summarised in Table 1.
Using the entire dataset we develop human- proxy agents that act as standard and cheap test partners for ad-hoc human-Al coordination evaluation. To prevent overfitting, we host the human proxies behind an API instead of releasing them pub- licly. Participants have to pre-register an evaluation which gives them access to to 1,000 evaluation games with our human proxies. This controlled access ensures consistency across submission and pre-registration of experiments is the gold standard for empirical science. The candidate agent's 's performance is evaluated based on the mean and median scores achieved across 1,000 games with human proxies and will be published on our leaderboard.
At the end of each training epoch, we compute the average SP score of n BC over 5000 games and store the parameters O', that yield the highest average SP score. During each of those SP evaluations, each agent, every timestep, selects the local action with the highest predicted probability ac- cording to the local policy, i.e. of = arg max, TBC (a' Irf).
In the second step of the HDR-IPPO method we leverage the bascline BC policy, BC, to guide the training of more robust policy, क HP First, we initialize the weights of Th to O'. Next, encourage the final policy to remain close to the human-like behaviour exhibited by g& BC, we introduce the KL (Kullback & Leibler, 1951) regularisation term
In the second (optional) part of the challenge, we also assess the agent's ability to predict human actions in an unseen set of human-played games. We evaluate performance using the feacher-foreed cross-entropy loss.
(KL BC (117))1 mg (((7)))
(1)
We build upon previous works (Cornelisse & Vinitsky, 2024: Jacob et al., 2022; Hu et al., 2022; Bakhtin et al., 2022).
4

---

Ad-Hoc Human-Al Coordination Challenge
Table 1. Statistics for open-sourced data. The table presents the minimum, maximum, average, median, and standard deviation for both game scores and game lengths in each setting.
Setting
Metric
Min
Max
Avg
Median
Std
Scores
13
25
23.37
24
1.86
1,858 Two-Player Open-Sourced Games
Game Lengths
52
76
65.45
66
3.35
Scores
14
25
23.25
24
1.91
1,221 Three-Player Open- Open--Surced Games
Game Lengths
45
67
57.86
58
3.38
which demonstrate that introducing KL regularisation in this manner preserves numan-likenes, of the BC policy while being more robust.
Our human proxies serve standardised, robust and cheap partners for human-Al coordination evaluation. In the context of AH2AC2, human proxies play a pivotal role in benchmarking Al performance against human-like be- haviour. This setup not only facilitates the assessment of AI adaptability and robustness but also ensures that evaluations are scalable and reproducible.
This KL term is then added to the standard IPPO (de Witt al., 2020) objective c.IPP (0), with a regularisation weight
[0, 1]:
C HDR IPPO (0) = - A) 2) (0)
Additionally, we present ablation study to examine the im- pact of the HDR-IPPO KL regularisation term in Appendix A.8. This analysis explores the effects of varying regular- isation strength on the learned policies, offering insights into the role of this component. We defer this discussion to the appendix to maintain focus on the properties of the developed human proxy agents within the main text.
+ DKL (-1:(-11)) HP
(+(7)))
(2)
Further details and a complete list of hyperparameters used for training human proxy agents are available in Appendix A.5.
While our approach builds on prior research (Comeelise & Vinitsky, 2024; Jacob et al., 2022; Bakhtin et al., 2022; Hu et al., 2022), we design the following experiments to empirically validate that the generated human proxy agents exhibit human-like behaviour Hanabi:
PART 2 OF AH2AC2: ACTION PREDICTION CHALLENGE
Beyond the primary human-Al ad-hoc coordination chal- lenge, we introduce an action prediction task. Although compa-compatible play does not strictly ensure accurate action prediction, successfully predicting human actions further demonstrates human- like behaviour. The action pre- diction dataset consists of a held-out portion of the human data used to train the human proxies.
Cross-Play between Human Proxies and BC Policies: We test how well the human proxy agents coordinate with baseline BC policies, which closely follow human con- ventions but may lack generalisation. Strong performance in this cross-play setting confirms that our HDR-IPPO agents maintain human compatile conventions while im- proving upon the generalisation limitations of pure BC.
We evaluate performance using the teacher-forced cross- entropy loss since we aim to quantify the difference between the predicted action distribution and the true human actions. Lower cross-entropy loss indicates better alignment with human dwcision-maaeeee Participating agents receive a local action-observation history and must predict the action taken by the human player at each timestep.
Action Prediction Performance of Human Proxies: We compare the performance of the BC policies on the test datasets before HDR- IPPO training procedure with that of the final human proxies the end of HDR-IPPO pro- cedure. This comparison allows us to evaluate whether the agent retains human-like conventions present in the dataset.
4.3. Evaluation API and Leaderboard
To facilitate participation and ensure fair evaluation, we host the human proxy agents and provide access through dedicated evaluation API.
Behaviour Analysis Human Proxies: We use the behavioural metrics Information per Play (IPP) and Com- municativeeess, as proposed by (Canaan al., 2020), to evaluate both the trajectories in the dataset and those gen- erated in SP by the human proxies. These metrics are straightforward to measure and are recognized as strategi- cally important. By making this comparison, we demon- strate that human proxies display behaviours consistent with those in the human play dataset.
We have established a dedicated website for the AH2AC2 challenge at https://ah2ac2..oom process, participants must fill out form to register for ac- cess the evaluation phase. Once registered, we provide participants with a private key, which grants restricted access
To initiate the evaluation
5

---

Ad-Hoc Human-AI Coordination Challenge
our human proxies. This key allows a one-time evaluation run, strictly limited to 1,000 games. Once the evaluation with human proxies is finished, participants get limited ac- cess to the test dataset through the API we provide. Upon completion, results are published on the challenge leader- board. Furthermore, our evaluation API allows interaction with the proxy agents while restricting access to global game state information, enforcing the partial observability inherent to Hanabi.
Moan Scores
Nedian Scores
22
22.78
21.69
21.42
23
23.00
22.00
23.00
10
HP2
22 7B
21.02
21.91
21.75
HP2
23.00
23:00
21.00
23.00
ICT
21.91
19:66
19:27
BC1
22.00
23:00
22.00
23 42
21.75
19:27
18.95
23.00
23.00
22.00
22.00
HP3
187
100
Tamr
Roms
5. Analysing and Validating the Human-Proxies
(a) Two-player setting.
Scores
Modian
IP3
20.90
21.13
10.70
12.
1P5
21.
21.00
19.00
20
This section is organised as follows. First, we evaluate the self-play performance of our human proxy agents, demon- strating significant improvements compared to the BC poli- cies. Second, we validate the -ineenssss of our human proxy (HP) agents. This is done through cross-play experi- ments with BC policies, by an evaluation of the action pre- diction performance of the HP policies on held-out datasets of human gameplay, and with analysis of behavioural metrics.
HP4
23.00
21.20
11.20
13.27
HP4
21.
21.00
18.00
14.00
8C3
17.52
18.20
4.87
6.35
BC3
20.00
21.00
0.00
0.00
50
BC1
18.89
BC4
20,
21:00
0.00
(h) Three-player setting.
Figure 2. Cross-play performance matrices for human proxy agents and corresponding BC policies in two-player and thrce-player set- tings. Strong cross-play performance between two policies indi- cates that policies converged compatible conventions. Each element (i represents the average score achieved by different team configurations, over all possihle permutations of player posi- tions. For the three-player setting team comprises two instances of agent and one instance of agent i. The average score for each team is calculated based on 15,000 games per permutation.
5.1. Self-Play Scores of Human Proxy Agents
Table 2 presents the SP scores of our final human proxy agents and their improvement over the initial BC policies. The performance gains from regularised RL are particularly pronounced in the parce-player setting, where the BC poli- cies trained on limited data frequently lose all their lives, resulting in a large proportion of zero-score games and over- all bad performance. For instance, in three-player SP eval- uation, BC agents scored zero in 70.92% of games. With the help of regularised RL, human proxies score zero points only on 0.27% of the games. This highlights the robustness achieved through regularised RL. Moreover, we observe a larger number of perfect-score games for all agents, com- pared to BC counterparts.
expectation is confirmed in Figures 2a and 2b. By compar- ing median and mean scores, we observe that agents either coordinate exceptionally well or fail completely. These re- sults suggest that HDR-IPPO agents have not only learned to play effectively amongst each other but have also retained strategies employed by BC agents (i.c. agents trained solely on human data).
Furthermore, Figure and 2b illustrate the cross-pla poas-- re- sults for two-player and three-player setting human proxy agents. We observe consistent scores across different pair- ings, suggesting that the agents have converged to compat- ible strategies despite variations in their architectures and regularisation strengths.
In the three-player setting, mean scores are significantly lower than median scores, especially when two BC policies team with one human proxy. This suggests many zero- score games and highlights the brittleness of BC policies in unfamiliar scenarios. Despite this, median scores remain high despite BC policies' poor single-playe performance. When two BC policies are paired with a stronger human proxy, median scores increase substantially, reaching 17 or higher in all configurations.
5.2. Validating Human-Likeneen of Human Proxy Agents
Cross-Play between Human Proxies and BC Policies. Behavioural cloning (BC) policies are closely aligned with human conventions but lack generalization capabilities. However, we anticipate that when BC policies are paired with human proxies, the resulting cross-play scores will be substantially higher than the BC policies' SP scores. This
These cross-play experiments provide evidence that human proxy agents have successfully learned to play the game at a high level while maintaining the ability to interact effectively with agents that utilize strategies derived purely from human demonstrations.
6

---

Hoe Human-AI Coordination Challenge
Table SP evaluation results for the four human proxies. We compare human proxy results with the results of the respective BC policies. For SP evaluation, we report mean + SE over 5,000 games.
Metric
HP R-1
62
HPP
Tga
HP
HP gg
Mean Self-Play Score
22.55
0.03
22.97 = 0.03
20.88
+ 0.03
21.21 + 0.03
Improvement over BC
3.0
4.0
15.7
13.9
Perfect Games (%)
23.86%
29.66%
2.76%
3.88%
Perfect Games BC (%)
16.12%
19.88%
1.34%
1.80%
Zero-Score Games (%)
0.10%
0.04%
0.34%
0.20%
Zero-Scove Games BC (%)
11.42%
17.70%
75.82%
66.02%
Action Prediction Performance of Human Proxies.
sive human play dataset. As shown in Table 4, both metries are nearly identical across human proxies and the human dataset, indicating similar behaviour and strategy.
Next, we evaluate the action prediction performance of hu- man proxies using a test set that was excluded from the training of the BC agents. The results, reported in Table 3, demonstrate that the human proxies achieve similar accuracy and loss metries on the test sets as the BC policies. In many Hanabi game scenarios, multiple human-like actions are possible. We thus report the Top- 10% and Top-20% accura- cies, which represent the probability that the ground-truth action is among the top 10%, 20% of most likely actions under the human proxy policies. This metric corresponds to top-2, top-4, and top-3, top-6 accuracies for two-player and prree-paayers respectively.
Table 4. Comparison of behaviour features between the large-scale dataset and 50,000 trajectories generated by human proxies. IPP (Information Per Played card) measures the average number of known attributes (colour and/or rank) per played card, normalized between O and 1. Communicativeness quantifies the proportion of turns which an agent chooses to give a hint when hint token is available. The metrics are nearly identical across data sources, indicating similar behaviour and strategy.
Setting
Source
IPP
Communicativenes!
Table 3. Loss and accuracy of the four human proxy policies on the test dataset. Compared to respective BC policies, human proxies significantly improve in SP. while still fitting the test sets well. Additionally, the high 10p-10% and Top-20% accuracies suggest that proxies are indeed human-like.
2P Dataset
0.44
0.47
2P
T HP
0.43
0,45
T02
HP
0.44
0.48
3P Dataset
0.42
0.49
Metric
Rg1
HP
622
HP
T8a
H P
g4
H.P
3P
T
HP
RP
0.44
0.44
0.47
0.46
Number of Players
2
2
3
3
Accuracy
0.63
0.63
0.43
0.44
For additional experiments and results, please refer to A.5, A.8 and A.6. We conducted an ablation study by varying the strength of the regularisation term to examine its impact on the final policy. Our findings indicate that initializing training from a BC policy without regularisation leads to weak and/or human- inmmaaiiccompatbblee policies. Finally, we pro- vide further behavioural analysis of our human proxies to illustrate their performance and interactions.
Accuracy Difference to BC
-0.03
-0.08
-0.08
-0.07
Loss
0,53
0.54
0.63
0.60
Loss Difference to BC
+0.05
+0.07
+0.08
+0.06
Top- 10% Accuracy
0.82
0.82
0.71
0.73
Top-20% Accuracy
0.95
0.95
0.87
0.88
Behaviour Analysis of Human Proxies. We assess the behaviour against large-scale human dataset using two metrics from (Canaan et al., 2020): IPP and Communica- tiveness. IPP (Information per Played Card) measures the information an agent has about each card it plays: for ev- ery card played, we track which attributes (colour and/or rank) are known (0, 1, or 2), average these values across all played cards, and normalize by dividing by 2 to obtain score between 0 and 1. Communicativeness measures the proportion of turns where an agent gives a hint when a hint token available, quantifying how often an agent chooses to communicate. These metrics were computed from 50,000 hroxy trajectories in SP and the exten-
6. Results of Baseline Methods on AH2AC2
In this section, we evaluate several baselines in the AH2AC2 challenge, informed by previous research (Hu al., 2021; Carroll al., 2019; Strouse al., 2021). IPPO: Policy trained in SP, without human data. BC: Policy trained only available human data using supervised learning. HDR- IPPO: Policy builds upon the BC reference by incorporat- ing regularised RL. BR-BC (Carroll et al., 2019): Policy is trained by embedding the BC policy in the environment, treating its actions as a part of environment dynamics during
7

---

Ad-Hoc Human-AI Coordination Challenge
SP learning. OBL (Hu et al., 2021): Strong approach for ZSC that does not utilise any human data during training. We use this agent only in the two-player setting since we do not have access to three-player weights. OP (Hu et al., 2020): A ZSC method that prevents agents from learning equivalent but mutually incompatible policies across inde- pendent training runs. OP accomplishes this by enforcing the equivariance of the policies under the symmetries of the Dec-POMDP, which must be provided as an input of the algorithm. FCP (Strouse et al., 2021); A method trained as the best response to a population of self-play agents and their past checkpoints taken throughout training. DeepSeek- R1 (DeepSeek-AI et al., 2025): A strong reasoning L.M.. We benchmark its performance by prompting it to make de- cisions in Hanabi, without any speeifii fine-tuning.
inhibiting its evaluation in this setting). The BC and IPPO baselines perform poorly in both settings, expected.
These results highlight OBL's effectiveees, surpassing BC, BR-BC, and HDR-IPPO without relying on human data. Methods like OP fail to achieve successful human-Al co- ordination, and current approaches that leverage limited human data underperform compared to state-of-the algorithms like OBL. This reveals a research gap; existing methods cannot effectively integrate small human datasets to enhance coordination. Additionally, our evaluation of FCP in Hanabi shows it struggles in complex, partially ob- servable environments, suggesting that population- based methods may not provide robust coordination capabilities. There is thus a need for new techniques that efficiently utilise limited human data to improve human-AI teamwork. Al- though DeepSeek-RI demonstrates foundational capability, significant improvements are still needed. In the two-player setting, even when prompted with H-conventions, it signifi- cantly underperforms compared to OBL. While providing H-conventions substantially improved its score over the basic prompt (5.43 vs 9.91), this approach still falls consid- erably short of OBL. In the three-player setting, shows relatively stronger performance, outperforming all other baselines. Our results suggest that while current LLMs, even without fine-tuning. exhibit some inherent coordina- tion capabilities, they do not yet achieve the desired level of efficacy on AH2AC2. Also, it is crucial to note that, due to resource and time constraints, our evaluation of DeepSeek- RI was conducted on only 100 evaluation games in contrast to the 1000 games used for all other methods, and we leave the action prediction challenge for future work.
ZSC
We adopt the IPPO configuration from Rutherford et al. (2023) and adapt the BR-BC training procedure from Carroll et al. (2019). These baseline approaches utilise feed-forward architectures. Our BC and HDR-IPPO baselines, in contrast, employ an LSTM- based architecture. While based on the same backbone as the human proxy agents, we use different hyperparameters optimised for the data-limited challenge settings. Additionally, we utilise different random seeds than those used for training the human proxies.
For each of the BC, BR-BC, and HDR-IPPO baselines, we train with three different random seeds. The best agent, based on cross-entropy loss on the validation set, is selected for evaluation. Notably, we observe minimal performance variance across different seeds on the validation set (see Appendix A.3). The IPPO baseline, intended showcase the limitations of SP in ad-hoc coordination, is trained with a single seed. Finally, we use pre-trained weights and re- spective hyperparameters for OBL (Hu et al., 2021).
7. Conclusion
The training population for the FCP agent comprises 36 ran- dom seeds. Since Hanabi presents a greater challenge than Overeooked, the environment in which FCP was initially introduced, we employ four checkpoints for each agent. Consequently, the training process for a single FCP agent utilizes 144 checkpoints, Due to the computational demands involved, we train the FCP agent with a single seed.
We introduced the Ad-Hoc Human-Al AI Coordination Chal- lenge (AH2AC2), evaluating human-Al ad-hoc teamplay in the context of the cooperative card game Hanabi. By leverag- ing large-scale human play dataset to generate human- -ike agent policies, we provide a meaningful evaluation frame- work for assessing Al agents' ability to coordinate with human partners. We released a comprehensive set of base- lines, including agents trained with and without human data, to provide a reference point for evaluating novel approaches. Our results highlight the inherent challenge of human-AI coordination. We believe that AH2AC2 represents signifi- cant step forward in the field of Auman-AAl coordination. By providing a standardised evaluation protocol, human proxy agents, and a diverse set of baselines, we aim to foster fur- ther research and development. We invite researchers and practitioners alike to participate, pushing the boundaries of what's possible in human -AI collaboration.
For DeepSeek-R1, we evaluate its capability using two prompting variants: one providing the LLM solely with the current game state in natural language, and another that additionally includes description of H-conventions. Fur- ther details on these prompts are available in Appendix A0
Table 5 shows the initial AH2AC2 leaderboard. OBL (L4) achieves the highest performance without using any human data. In two-player settings, BR-BC achieves the highest score, while HDR-IPPO leads in three-player settings (al- though OBL lacks pre-trained weights for three-players,
Several open challenges and questions remain. We highlight some promising directions for future research:
8

---

Ad-Hoc Human-Al Coordination Challenge
Theoretical Analysis of HDR-IPPO: While experi- ments and previous works provide strong empirical evi- dence for the effectiveness of regularised RL in generating human-like agents, a deeper theoretical understanding of the methodology is crucial.
pabilities for human-Al coordination. Future work should extensive evaluation of LLMs and explore more sophisticated methods, Excitingly, when paired with our human proxies, Hanabi becomes an excellent bench- mark for assessing theory of mind in LLMs and their ability to cooperate with humans in complex, partially observable tasks.
conduct
Generalisation of the benchmark We currently only cover 2 and players as well as "standard" Hanabi, ignor- ing the large set of possible variations of the game that are created e.g. by the "rainbow cards". Extending the benchmark cover those scenarios would be great way measure the generalisation ability of agentic systems. Direct Human-Al Play with Human Proxy Agents: The ultimate validation of our human proxy agents re- quires direct human-AI play. Future work should involve conducting play experiments with human participants, comparing their experiences and performance when play- ing with human proxies versus playing with other humans. Comprehensive Evaluation and Advancement of Agen- tie LLMs: Our preliminary evaluation of DeepSeck-R1. - though limited by resource constraints to fewer games than other baselines, provides initial insights into LLM ca-
Table The AH2AC2 lcaderboard includes Mean and Median scores achieved for human-Al coordination evaluation and Cross- Entropy Loss (CF) for action prediction challenge. OBI. does not make of available human data, yet achieves high scores. There is a pressing need for new techniques that efficiently utilise limited human data improve human-Al coordination.
Players
Method
Mean
Median
CE
OBL (L4)
21.04
22
1.33
BR-BC
19.41
20
10.82
FCP
14.01
16
3.52
2P
OP
13.91
19
7.81
HDR-IPPO
12.76
15
0.96
IPPO
10.16
14
12.60
DeepSeek-RI H-Group
9.91
O
DeepSeek-R1
5.43
BC
2.12
0
0.86
Human Proxies
22.76
23
0.54
BR-BC*t
22.59
23
5.00
DeepSeek-RI H-Group
14.62
18
DeepSeek-R1
14.38
18
HDR-IPPO
14.03
16
0.80
OP
12.87
18
6.40
BR-BC
11.89
12
29.89
FCP
11.55
6
5.97
IPPO
6.34
8.60
BC
3.31
0
0.70
Human Proxies
20.86
21
0.62
BR-
18.80
19
7.53
T Not constrained by game limits, acts as golden stan- dard. BR-BC* is trained with BC policy trained the entire dataset. We report average performance two human proxies.

---

Ad-Hoc Human-AI Coordination Challenge
Acknowledgements
48550/ARXXV.2210005442 org ab////////// 221.22221...555
URL https://arxiv:
We want to thank Johannes Forkel and Darius Muglich for their contributions various aspects of our work. We also thank Ivan Zubak for helping us set up the web- site for AH2AC2. AC acknowledges funding from the UKRI AI World Leading Researcher Fellowship (grant EP/W002949/1). AC and JF also acknowledge support from J.P. Morgan Chase Faculty Research Award. RH gratefully acknowledges support from the Autonomous In- telligent Machines and Systems (AIMS) programme at the University of Oxford, in conjunction with sponsorship from Rosebud ai. AL acknowledges funding from the Fonds de recherche du Quebec.
Bansal, M., Krizhevsky, A., and Ogale, A. ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst. 2018. doi: 10.48550/ARXIVI8812 URL https: http///axivv.// org/ abs hts///bb////1// 1812. 03079.
Bard, N., Foerster, J. N., Chandar, S., Burch, N., Lanc- tot, M., Song, H. F., Parisotto, E., Dumoulin, V., Moitra, S., Hughes, E., Dunning, I,, Mourad, S., Larochelle, H., Bellemare, M. G., and Bowling, M. The Hanabi Chal- lenge: A New Frontier for Al Research. 2019. doi: 10.48550/ARXXI.190000000 org/ abs / 1 000.002.000000
URL https: haapps://arxivv /arxiv.
Impact Statement
Canaan, R., Gao, X., Togelius, J., Nealen, A., and Menzel, S. Generating and adapting to diverse ad-hoc cooperation agents in hanabi. CoRR, abs/2004. 13710, 2020. URL https:/ //t//aavvvv/../////// .rriiioorrg/eeee /000
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Se- shia, S. A., Abbeel, P., and Dragan, A. On the Util- ity of Learning about Humans for Human-Al Coordina- tion. 2019, doi: 10445550ARXXV..11000555 https: http///axivvvv.. wwwp:/iww...oraaaoooooa hbcp://a///// 05789.
References
URL
Ahn, M., Dwibedi, D.,, Finn, C.,, Arenas, M. G., Gopalakr- ishnan, K., Hausman, K., Ichter, B., Irpan, A., Joshi, N., Julian, R., Kirmani, S., Leal, I., Lee, E., Levine, S., Lu, Y., Leal, I., Maddineni, S., Rao, K., Sadigh, D., Sanketi, P., Sermanet, P., Vuong. Q.. Welker, S., Xia, F., Xiao, T., Xu, P. Xu, S., and Xu, Z. Autort: Embodied foundation mod- els for large scale orchestration of robotic agents, 2024. URL https:://// / / /atxiv..vvv..////// orgg abs 2401 12963.
Cho, K. van Merrienboer, B., Gülçehre, Ç., Bougares, F,, Schwenk, H., and Bengio, Y. Learning phrase represen- tations using RNN encoder-decoder for statistical ma- chine translation. CoRR, aws//406. 1078, 2014. URL http:://axxivorg//// 2ttg/abb/1406.1777
Cornelisse, D. and Vinitsky, F. Human-compatible driving partners through data-regularized self-play reinforcement learning. 2024. doi: 10448555/ARRXII22400. 19648. URL https: htt///aaxvvvv/. /ttpp::/arx....ogg/aabba//////// wttp://wwwww...////......
Asan, O., Bayrak, A. E., and Choudhury, A. Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians. Journal of Medical Internet Research, 22(6): e15154, June 2020. ISSN 1439-4456.
doi: 10.2196/
Cui, B., Lupu, A., Sokota, S., Hu, H., Wu, D. J., and Fo- erster, J. N. Adversarial Diversity in Hanabi. In The Eleventh International Conference on Learning Repre- sentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https::t///// /tpp:///ppeeeiie... net
15154. URL ht.pt: wwww ncbi nlm. nih, ..v.ggo//vvv pnc/artticles//MNT33111
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N.. Haafield-Dodds Z., Herman- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, I.., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Training a helpful and harmless assistant with rein- forcement learning from human feedback, 2022. URL https: //axvvv./ org/ org/abs////eeo...ooo.ogoo.bbb 2204 05862.
de Witt, C. S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P. H. S., Sun, M., and Whiteson, S. Is Indepen- dent Learning All You Need in the StarCraft Multi- Agent Challenge? 2020. doi: 00448500ARRXIV22011.0555 URL https:ttp/// htt///axxvvv../ http:::::/aaa...or/aaaa/////s/s/ wtt.s//orrr/bbb//1...e/.
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song. J., Zhang, R., Xu, R., Zhu, Q. Ma. S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F. Luo,
Bakhtin, A., Wu, D. J., Lerer, A., Gray, J., Jacob, A. P., Farina, G., Miller, A. H., and Brown, N. Mastering the Game of No-Press Diplomacy via Gluman-Regulate Reinforcement Learning and Planning. 2022. doi: 10.
10

---

Ad-Hoc Human-AI Coordination Challenge
F. Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Ding, H., Xin, H., Gao, H., Qu, H., Li, H., Guo, J., Li, J.. Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, J. L., Ni, J., Liang, J., Chen, J., Dong. K,, Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, I.., Zhang, L., Zhao, L., Wang. L., Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Li, M., Wang, M., Li, M.. Tian, N., Huang, P., Zhang, P., Wang. Q.. Chen, Q- Du, Q., Ge, R., Zhang. R., Pan, R., Wang. R., Chen, R. J., Jin, R. L., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S., Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S., Wu, S.. Ye, S., Yun, T., Pei, T., Sun, T., Wang, T., Zeng. W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W. L., An, W., Liu, X., Wang, X., Chen, X., Nie, X., Cheng. X., Liu, X., Xie, X., Liu, X., Yang. X., Li, X., Su, X., Lin, X., Li, X. Q. Jin, X., Shen, X., Chen, X., Sun, X., Wang. X., Song, X., Zhou, X., Wang, X., Shan, X., Li, Y. K., Wang. Y. Q. Wei, Y. X., Zhang, Y. Xu, Y., Li, Y., Zhao, Y., Sun, Y. Wang. Y., Yu, Y., Zhang. Y., Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y. Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y. Zhu, Y. Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z. Ma, Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z.,, Zhang, Z. and Zhang. Z. Deepseck-rl: Incentivizing reasoning capability in ilms via reinforcement learning, 2025. URL https: / arxiv. org/ abs / 22010 12948.
Hu, H., Wu, D. J., Lerer, A., Foerster, J., and Brown, N. Human-AI Coordination via Human-Regularized Search and Learning. 2022. doi: 1048550/ARXX..2210005122 URL https:: awwaarxivv..... wwt.://rrgrbbbbb/..oooo
Jacob, A. P. Wu, D. J., Farina, G., Lerer, A., Hu, H., Bakhtin, A., Andreas, J., and Brown, N. Modeling strong and human-like gameplay with KL-regularized search. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9695-9728. PMLR, 17-23 Jul 2022. URL https:// htt.tp:://ppoceeeingge. mlr press// html.
Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization. 2014. doi: 4448550/AXXI.14422..06 URL https: /ttp/vaaavvv//.////////../a..av. org/ aba 1412. 6980.
Kullback, S. and Leibler, R. A. On Information and Suf- ficiency. The Annals of Mathematical Statistics, 22 (1):79866, 1951. ISSN 0003-4851. URL https: //www. www.r.sttr..... wwwp:jsstor.rrgsstteeeeeee 22236673.
Lerer, A., Hu, H., Foerster, J., and Brown, N. Improving Policies via Search in Cooperative Partially Observable Games. 2019. doi: 10:4850/ARXXIV.111200311 URL https: /etp:///axivv... org. 1ttp/11112..2111..
Lewis, P., Perez, E.,, Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-auggmened generation for knowledge- intensive nlp tasks. Advances in neural information pro- cessing systems, 33:9459-994 9974, 2020.
Gemini Team, e. a. Gemini 1.5:
Unlocking multimodal understanding across millions of tokens of context, 2024.
URL https:tp//// / /aaaiaixiivv./..///aaaa/aaaav/at www.iiiorr/roo.... abs 2403. 05530.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft Actor-Critic Algorithms and Applications.
Liu, N. F. Lin, K., Hewitt, J., Paranjape, A., Bevilac- qua, M., Petroni, F., and Liang, Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.034 000000 2023.
2018. doi: 10.45550/ARXXV111110.5505 URL https: //arxiv.orgg abs/ 111./11111.. 05905.
Lupu, A., Hu, H., and Foerster, J. Trajectory Diversity for Zero-Shot Coordination. In Proceedings of the 20th In- ternational Conference on Autonomous Agents and Mul- tiAgent Systems, AAMAS '21, pp. 1593-1595, Richland, SC, May 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 11144883373.
Hochreiter, S. and Schmidhuber, J. Long Short-Term Mem- ory. Neural Computation, 1735-177, November 1997. ISSN 0899-7667, 5530-888X. doi: 10.1162/neco. 1997.9.8. 1735. URL hwtph:tttt..o. /di rect mit edu/ neco/article /ttp////////// 1780//10079
Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J. "Other-Play" for Sero-Shot Coordination. 2020. doi: 10.48550/ARXIV.2000.0 02979. URL hwtptt:/t/aaae.. org/abs/200k 92979.
Nekoei, H., Zhao, X., Rajendran, J., Liu, M., and Chandar, S. Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi. 2023. doi: 10.48550/ARXIV22300.. 10284. Uttpttts:// /arxiv.. org 000.30118.2.......
Hu, H., Lerer, A., Cui, B., Wu, D., Pineda, L., Brown, N., and Foerster, J. Off-Belief Learning. 2021. doi: 10.44550/ARXIV22008000 1000000000 URL https: //rxxiv. org/abs// Frgg/abs/2103.00 22tp..2133.0000 04000.
Oliehoek, F. A. Decentralized POMDPs, pp. 471- 503. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-27645-3, doi: 10.1007/

---

Ad-Hooc Human-Al Coordination Challenge
978-3-642-27645--.15. URL https: /ttp/diooorrrro org/ 10 10.11077/978-33-222277888
Treutlein, Dennis, M., Oesterheld, C., and Foerster, J. N. A new formalism, method and open issues for zero-shot coordination. CoRR, 2206. 06613.
2021.
Rawas, S. Al: the future of humanity. Discover Artificial Intelligence, 4(1):25, March 2024. ISSN 2731-0809. doi:
URL httpst/s// http:////wxwv.........eoo
10.10077/444133444--001111- URL attps://dco 100.:1100/00000 ottp//0..1007/44111------001111-
Wei, J., Wang, X., Schuurmans, D., Bosma, M.. Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-though prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Russell, S. Human compatible: artificial intelligence and the problem of control. Viking, New York?, 2019. ISBN 9780525558613.
Rutherford, A.. Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson, G., Willi, T., Khan, A., de Witt, C. S., Souly, A., Bandyopadhyay, S., Samvelyan, M., Jiang, M., Lange, T., Whiteson, S., Lacerda, B., Hawes, N., Rocktaschel, T., Lu, C., and Foerster, J. N. Jax- MARL: Multi-Agent RL Environments in JAX. 2023. URL https://
doi: 10448550ARRXV22311 10090. arxiv. org/ abs /
Samuel, A. L. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):100-22,, 1959.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algorithms.
2017. doi: 1044850/ARXIV..100.0055 //waxxa...rg///////...... wtwp:i/.ogg/bbb/i//eee 06347.
URL https:
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanetot, M., Dieleman, S., Grewe, D., Nham, J.. Kalchbrenner, N., Sutskever, I., Lilli- crap, T., Leach, M., Kavukeuoglu, K., Graepel, T., and Hassabis, D. Mastering the game of Go with deep neural networks and tree search. Nature, 529 (7587):484448, January 2016. ISSN 1476-4687. doi: 10. 10.10081/ URL https: htmp/atticles/attueeeleee
Stone, P., Kaminka, G. A., Kraus, S., and Rosenschein, J. S.
Ad hoc autonomous agent teams: collaboration without pre-coordination. In Proceedings of the Twenty-Fourds AAAI Conference on Artificial Intelligence, AAAI 10, pp. 1504-1509, Atlanta, Georgia, July 2010. AAAI Press.
Strouse, D., McKee, K. R., Botvinick, M., Hughes, E., and Everett, R. Collaborating with Humans without Human URL
Data. 2021. doi: 11445500/ARXI.221000008
https:/ www.ivivvv. wttp://wwwgg////.......o/
Tesauro, G. Td-gam rddgaammnn self-teaching backgammon pro- gram, achieves master-level play. Neural Computation, (2):215-219, 1994. doi: 10. 10.1162/neco. 1994.6.2.215.
12

---

Ad-Hoc Human-AI Coordination Challenge
A. Appendix
A.I. Hanabi
In Hanabi, players aim to collectively build five ascending stacks of cards, one for each of the five colours. The deck comprises 50 cards, with 10 cards of each colour. These include three cards of rank 1, two each of ranks 2, 3, and and a single card of rank 5. A unique feature of the game is that players can see each other's cards, but not their own. Information about one's own cards is gained through hints from teammates or by interpreting their actions. The team starts with eight information tokens and three lives. On their turn, players can choose one of the following three actions: (1) A player can choose a teammate and provide information about all the cards of a specific colour or rank in that teammate's 's hand. The hint must be complete, meaning all cards of the specified colour or rank must be indicated. Each hint consumes one of the limited information tokens. If no information tokens remain, this action is unavailable. (2) If the team has fewer than eight information tokens, a player can discard a card face-up, revealing it to all players. This action replenishes one information token and the player draws a new card. (3) A player can attempt to play card from their hand onto the corresponding colour stack. If the card's rank is the next in the sequence for that colour, it is successfully played. Successfully playing a rank-five card earns the team additional information token. However, incorrect play results in the loss of one of the team's 's limited lives. Regardless of the outcome, the player draws a new card from the deck. The game concludes when one of the following three conditions is met: (1) the team loses all their lives, (2) all five stacks are completed, or (3) the deck is depleted. The final score reflects the team's 's success in building the stacks. The maximum score is 25, and it is determined by summing up all the played cards. When all lives are lost, the score is zero
A.2. Further Background
INDEPENDENT PPO
Proximal Policy Optimisation (PPO) (Schulman et al., 2017) is a method initially developed for single-agent RL. PPO aims to address performance collapse in policy gradient methods. It does this by bounding the ratio of action probabilities between the old and new policies. PPO optimises the objective function,
7(a 8) A A a), as, clip TTaa| s)
T(a s) 11 1 1 A
a(a 8)
E Egeod goon
min
where clip(t, a, b) is a function that outputs if < a, b b and otherwise. We consider the extension to multi-agent setting by using independent learning (de Witt et al., 2020). Here, each agent treats the others as part of the environment and learns a critic using its local AOH.
OBL: ZERO- SHOT COORDINATION
Most SP methods exhibit a tendency towards brittleness and over-coordination, leading to suboptimal performance when paired with independently trained policies. To address this, OBL (Hu et al., 2021) was proposed as an algorithm that learns optimal grounded policies without relying arbitrary conventions or assumptions about other agents. OBL has demonstrated remarkable success in ZSC scenarios, making it a compelling baseline for ad-hoc teamwork, even though it does not utilise any of the human data provided for the challenge.
A.3. Additional Results
We provide additional results for performance on the test sets in Table 6 and 7.
A.4. Training Details
DATA SPLIT
We begin with a large-scale dataset of game trajectories containing both two-player (2p) and three-player (3p) settings. From this dataset, we set aside validation and test sets. Specifically, for the 2p setting, we select 858 games for validation and 858 for testing; for the 3p setting, we allocate 221 games for validation and 221 for testing.
Once validation and test sets are set aside, the remaining data is used for two purposes: (1) training our human proxy models, and (2) sampling a subset of 1,000 games for each setting (2p and 3p) that we source to the research community. Additionally, we open-source our previously defined validation. The test sets, however, are kept closed-source
13

---

Ad-Hoc Human-Al Coordination Challenge
Table Cross-entropy loss the for BC, BR-BC and HDR-IPPO agents in the two-player setting. For human proxies we report results over two available agents. For the rest, we report results over three different seeds. Even though agents are trained with different seeds, they achieve almost identical results. report loss + SE.
Data Used
Method
Test Set
Loss
Accuracy
BC
0.87 + 0.0
0.46 + 0.0
Open Sourced Games
BR-BC
10.96 + 0.1
0.25 +: 0,0
HDR-IPPO
0.97 + 0.0
0.40 i 0.0
BC
0,48 1 0.0
0.67 1 0.0
Entire Dataset
Human Proxy
0.54 1 0.0
0.63 1 0.0
Table Cross-entropy loss on test set for BC, BR-BC and HDR-IPPO agents in three-player setting. For human proxies we report results over two available agents. For the rest, we report results over three different seeds. Even though agents trained with different seeds, they achieve almost identical results. We report loss + SE.
Data Used
Method
Test Set
Loss
Accuracy
BC
0.70 + 0.0
0,40 + 0.0
Open Sourced Games
BR-BC
32.48 + 1.6
0.07 0.0
HDR-IPPO
0.81 + 0.0
0.31 0.0
BC
0.54 1 0.0
0.51 0.0
Entire Dataset
Human Proxy
0.62 + 0.0
0.44 + 0.0
prevent adaptation overfitting during model development; these serve exclusively for final performance evaluation. The participants of the challenge not required to use the same data split defined here, but encouraged to do so.
BEHAVIOURAL CLONING (BC)
For each sampled mini-batch of games, we extract individual player trajectories, effectively decomposing each game into n trajectories, where n represents the number of players. This yields sets of trajectories, denoted by U. = {up. ),,), with one trajectory per player for each game. Each trajectory, u; = {(o), df) consists of local player observations, of E O², and corresponding actions, af A'' taken at timestep Our BC model takes single player's trajectory as input and predicts action for each timestep, conditioned the AOH. A mini-batch of size b is constructed by concatenating multiple sets of trajectories: Donch Uo || ... || Ub. where || denotes concatenation. To augment the training data and improve generalisation, we randomly shuffle the colour space for both observations and actions within each mini-batch before feeding it to the network, as done in the previous works to enhance performance of our model (Hu et al., 2021).
The loss for each batch is calculated as
BC (0) = ErjEDooto
Σ
Σ l(m
BC
BC ( (4))))))).
(4)
T/EDDatch ==1
where Dratch the batch of trajectories 7; and is the standard cross-entropy loss. The policy, To is conditioned both the hidden state oi, encoding the observation history, and the current local observation of-
BEST RESPONSE TO BEHAVIIUUAL CLONING (BR-BC)
When training with an embedded human model, we initially train in SP and anneal the amount of SP linearly to zero. Then, we continue training with the BC policy, as Carroll et al. (2019) find that this improves agents' performance. Additionally, we consider the three agent setting that was not considered by Carroll et al. (2019). When annealing, based on the annealing factor, we sample agents to decide whether we train in SP, with single BC policy or with two BC policies.
14

---

Ad-Hoc Human-AT Coordination Challenge
IPPO
We follow the same methodology and architecture as presented by (Rutherford al., 2023).
A.4.1. OTHER-PIAY (OP)
We employ feed-forward architecture and follow the methodology presented in the original work (Hu et al., 2020).
FICTITIOUS CO-PLAY (FCP)
FCP population consists of IPPO agents, trained with the same methodology as the IPPO baseline. We train total of 36 agents. For each trained agent, we consider four checkpoints when building a population to train best response. The final FCP partner agent is modelled using GRU (Cho et al., 2014), using the same parameters and methodology presented in (Rutherford al., 2023).
A.5. Human Proxies: Additional Details
DATASET
Dataset used for training human proxies contains 147,621 Hanabi games, comprising 101,096 two-player and 46,525 three-player games. Table summarises key statisties for scores and game lengths across both player configurations.
Table 8. The dataset used for training human proxies comprises total of 147,621 games, including 101,096 two-player and 46,525 three-player games. table presents descriptive statistics for game scores and lengths across both player configurations.
Setting
Metric
Min
Max
Avg
Median
Std
Scores
1
25
23.09
24.00
2.16
Two-Player
Game Lengths
2
88
65.70
66.00
3.63
Scores
2
25
22.94
23.00
2.10
Three-Player
Game Lengths
34
78
58.38
59.00
3.36
ARCHITECTURES
The architectures and hyperparameters used to train human proxy agents are shown in Table 9. Each model includes fully connected layer, multi-layer LSTM block, and a decoder fully connected network that maps the LSTM encodings to a probability distribution over actions. In our implementation, we employ loss masking to ensure that the probability of selecting an illegal action effectively set to zero during sampling from the policy. Optimisation performed using Adam optimiser (Kingma & Ba, 2014) across all configurations. When applying linear learning rate schedule, the learning rate is reduced its minimum allowable value during the final 10% of the training process.
HYPERPARAMETER SEARCH
We run a hyperparameter search for the BC policies used for training human proxies. We performed full grid search across a range of hyperparameters, considering both two-player and three-player game settings. The hyperparameter search space shown in Table 10.
From these configurations, we selected the two pest-performing settings for both two-player and three-player scenarios. These configurations subsequently employed in the second step of the HDR-IPPO procedure. Respective agents what refer to human proxies.
ROLE OF REGULARISATION
Our human proxy policies have demonstrated successful coordination in cross-play with both other human proxies and the original BC policies. Here, we show that arbitrary policies fail to coordinate well with them. We introduce new set of agents trained using the same HDR-IPPO procedure as our human proxies, but with crucial difference, the human data regularisation weight set to zero, =
15

---

Hd-Hoc Human-Al Coordination Challenge
Table Human proxy agent training configurations and architcctures. We showcase both BC and IPPO hyperparameters in single table.
Hyperparameter
HP Ag1
RP n0²
69
HP
HP T
Network Architecture
Num Players
2
Activation
GELU
GELU
GELU
GELU
LSTM Layers
512, 512, 512
512, 512 1024
512, 512 1024 1024
512, 512
Input Embedding
1024
1024
Decoder MLP
512
256
1024
BC
Optimisation
Batch Size
256
256
128
256
Dropout
0.5
LR Schedule
Linear
Linear
Linear
Linear
Initial LR
0.005
0,005
0.005
0.005
Final LR
5.0c-05
5.0c-05
1.0c-05
1.0c-05
Epochs
50
70
70
Training
Permute Colours
Yes
Yes
Yes
Yes
Self-Play Eval Games
5000
5000
5000
5000
IPPO
Optimisation
Learning Rate
0.0003
0.0003
0.0005
0.0003
Gamma Discount
0.99
0,99
0.99
0.99
GAE Lambda
0.95
0.95
0.95
0.95
Clip Epsilon
0.2
0,2
0,2
0,2
Entropy Coefficient
1.0c-05
0.0001
0.0001
0.0001
Value Function Coeff
0.5
0.5
0.5
0.5
Max Gradient Norm
0.5
0.5
0.5
0.5
Update Epochs
4
Num Minibatches
4
4
Critic Network
Critic MLP
512
512
512
512
Training
BC Policy KL Weight
0.3
0.2
0.1
0.15
Total Timesteps
5e9
5e9
5e9
5e9
Environments
Num Steps
90
Num Train Envs
1024
1024
1024
1024
Num Eval Envs
512
512
512
512
Table 10. Hyperparameter search space for BC when developing human proxies.
Hyperparameter
Values
Dropout
0.3, 0.5
Batch Size
64, 128, 256
LSTM Layers
(512, 512), (512, 512), (1024, 1024), (1024, 1024, 1024)
Input Embedding
512. 1024, (512, 512)
Decoder MLP
512, 1024
These new policies start from the same BC policy weights as our human proxies and utilise identical hyperparameters, except the KL regularisation term, which eliminated. We focus on two specific agents, one for the two-player setting (based on T BC) and one for the three-player setting (based on n BC). We use the weights obtained the final training timestep checkpoints for these model weights.
Firstly, we observed rapid deterioration in SP performance for the non-regularised policies. The agents quickly lost the ability to perform well, even in SP. This degradation is evident in Figure 3 and Figure 4. This sharp decline suggests divergence from the strategies initially learned during the BC phase. Without the regularisation term guiding the policy towards human-like behaviour, the agents appear to explore alternative strategies that may lead to suboptimal performance in the context of the game.
16

---

Ad-Hoc Human-Al Coordination Challenge
After the initial performance drop, the scores of the non-regularised policies gradually improve during training. However, with the current set of hyperparameters, this progress is slow and do not observe convergence. The agents unable to efficiently re-learn different, yet still effective, policy. It is important to note that this behaviour is not observed with different hyperparameter settings, but for the purpose of direct comparison in this analysis, we maintained the same configuration as the human proxy agents.
When paired with human proxies and BC policies, the non-regularised agents exhibit significantly lower coordination and overall performance compared to the pairings between human proxies and BC policies exclusively. This discrepancy highlights the erucial role of the human data regularisation term in ensuring that the learned strategies remain aligned with human-like play, facilitating successful coordination.
Means Scores
Median Scores
HP2
23.04
4.03
21.77
HP2
23.00
0.00
23.00
Candidates
Candidates
3m0-
9.44
113
A=O
0.02
30.00
0.00
10
BC2-
18.89
BC2
HP2
BCZ
H92
Tearss
Tearns
HP) )) its corresponding BC policy Figure 3. Cross-play performance matrix for two-player agents, comparing a human proxy agent (T (BBC), ) and non-regularised HDR-IPPO agent initialised from the same BC policy, aga but where 0. Each matrix element represents the average score achieved by a team, averaged over both player orderings. Averages are calculated based on 15,000 games per team permutation.
BC
Consider the three-player cross-play matrix depicted in Figure The BC policy, while achieving a low SP score of 7.19, significantly improves 18.92 when paired with two human proxies. This substantial boost underscores the BC policy's inherent understanding of human-like strategies, even if it struggles execute them independently. In contrast, the non-regularised policy, with SP score of 3.99, only sees a marginal improvement to 6.93 in cross-play with human proxies. Furthermore, pairing the BC policy (the stronger player in SP) with two non-regularised policies actually decreases the performance compared to the SP performance of non-regularised policies. This degradation suggests fundamental incompatibility between the strategies learned by the non-regularised policy and the human-like conventions given by the BC policy.
A.6. Behaviour Analysis
The dataset we use this work comes from the human policies that adhere to H-Group Conventions. This distinguishes our work from previous human-Al coordination studies, these strategies can be explicitly described natural language and documented.
We also include behavioural metrics proposed by (Canaan et al., 2020), for both the trajectories found in the dataset and the ones created by the human proxies. Results are shown in Table 11.
Information per Play (IPP): This metric assesses how much information the agent possesses about each card it plays. For every card played, we determine the number of known attributes (0, 1, or 2, representing colour and rank). These values are averaged across all played cards. The final score is then normalized by dividing by 2, resulting in range between 0 and 1. An agent with an IPP of I exclusively plays cards for which it knows both the colour and rank. Conversely, an agent with an IPP only plays cards about which it has no prior knowledge.
Communicativenes:: This metric quantifies how often an agent chooses to communicate information. calculated as
17

---

Ho- Human-AT Coordination Challenge
Means Scores
Median Scores
HP4
21.28
2.00
13.35
HP4
21.00
3.00
13.00
Candidates
Candidates
A=o
6.93
3.99
2.15
A=
8.00
4.00
0.00
BC4
18,92
2.57
T.19
BC4
21.00
HP4
BCB
HP4
Teams
Teams
Figure 4, Cross-play performance matrix for three-player setting, with scores for a human proxy agent (x H ( and a non-regularised HDR-IPPO agent initialised from the same BC policy, &C, but with X Each matrix element (f,j) represents the average score achieved by team composed of two instances of one agent the z-axis and one instance of agent the y-axis, averaged over all possible permutations of player positions. Averages are calculated based on 15,000 games per team permutation.
), its corresponding BC policy
the proportion of turns in which the agent opts to give hint, given that hint token is available at the beginning of the turn. A fully communicative agent, scoring I, would always provide a hint when possible. An agent that never gives hints, scoring O, would be considered completely non-communicative.
Table 11. We compare behaviour features available in the dataset and once present in trajectories generated by our human proxies. Precisely, we compute Information per Play (IPP) and Communicativeness. We show metrics are almost identical both cases.
Trajectories
Proxy
IPP
Communicativeness
2P Dataset
0.44
0.47
Two-Player
HP T gl
0.43
0.45
HP π g2
0.44
0.48
3P Dataset
0.42
0.49
Three-Player
HP T ga
0.44
0.47
H π g
0.44
0.46
Furthermore, we provide qualitative assessment by analysing games played by our human proxies. Our findings show that our agents exhibit highly human-like behaviour and in most instances adhere to H-convention strategies. Sample renders can be found the anonymous repository we provide and here we analyse the first game provided. We analyse each of the moves below, marking each move as success or failure, depending on whether the convention followed correctly. We also find that the mistakes made when following conventions are very human-like mistakes. For example, Turn the agent violates the good touch principle, but this move also looks lot like 2 save. Hence, it violates convention because it tried to follow different one, which was not applicable in this case. Most of the failure cases we found come from confusing conventions that should be played in that particular instance, which is a human-lik- mistake.
1. Turn 0: Success (Hint blue, play clue on the blue 1)
2. Turn 1: Success (Play clue both the 1's)
3. Turn 2: Failure (Looks like save, but doubles the yellow 2's, violating good touch principle)
18

---

Ad-Hoc Human-AI Coordination Challenge
4. Turn 3: Success (Actor plays card slot knowing it's B1, which follows the "play clue" convention)
5. Turn 4: Success (Actor 0 gives a 5 save when Red 5 is on the chop)
6. Turn 5: Success (Actor plays a known playable card)
7. Turn 6: Failure (Actor 0 should have 2-saved the Red 2 on chop)
8. Turn 7: Success (Actor discards Red 2, the chop card)
9. Turn 8: Success (Actor 0 plays slot 0, known playable 1)
10. Turn 9: Success (Actor discards chop on slot 3)
11. Turn 10: Success (Fix clued the duplicate cards)
12. Turn 11: Success (Plays known playable card)
13. Turn 12: Success (Gives play clue to the Red 1)
14. Turn 13: Success (Plays the Red 1)
15. Turn 14: Success (Discards chop)
16. Turn 15: Success (Gives play clue to the Yellow 3)
17. Turn 16: Success (Plays Yellow 3)
18. Turn 17: Success (Discards known trash)
19. Turn 18: Success (Discards chop)
20. Turn 19: Failure (Doesn't t understand chop-focus, giving play clue to wrong card, and violates good touch principle by
clueing Red and Red 3 twice)
21. Turn 20: Failure (The focus of the last clue was the chop, so it should have played the Red 3, but played Red 2 instead)
22. Turn 21: Success (Fix clue on the duplicated Red 3's)
23. Turn 22: Success (Plays known playable Red 3)
24. Turn 23: Success (Discards chop, position 1)
25. Turn 24: Success (Gives play clue to White 2)
26. Turn 25: Success (Plays White 2)
27. Turn 26: Success (Play clue the Red
28. Turn 27: Success (Plays the Red 4)
29. Turn 28: Success (Play clue Blue and filling in White 3)
30. Turn 29: Success (Plays White 3)
31. Turn 30: Success (Discards known trash)
32. Turn 31: Success (Play clue on White 4)
33. Turn 32: Success (Plays White 4)
34. Turn 33: Success (Plays known playable Red 5)
35. Turn 34: Success (Discards known trash Red 1)
19

---

Ad-Hoc Human-Al Coordination Challenge
36. Turn 35: Failure (Should have played its Blue 3 because of the play clue, instead gave 5 hint off chop, which is illegal
after the late game)
37. Turn 36: Success (Discards chop)
38. Turn 37: Failure (Should have played its Blue 3, instead gave a hint which is illegal)
39. Turn 38: Success (Discards chop)
40. Turn 39: Failure (Should have played its Blue 3, instead discarded chop)
41. Turn 40: Success (Play clue on Blue 4, filling in Blue 3)
42. Turn 41: Success (Plays Blue 3)
43. Turn 42: Success (Discards chop)
44. Turn 43: Success (Plays Blue 4)
45. Turn 44: Success (Discards chop)
46. Turn 45: Success (Hint Blue, filling in Blue 5)
47. Turn 46: Success (Plays Blue 5)
48. Turn 47: Success (Play clue Green 1)
49. Turn 48: Success (Plays Green 1)
50. Turn 49: Success (Discards chop)
51. Turn 50: Success (Play clue Yellow 4)
52. Turn 51: Success (Plays Yellow 4)
53. Turn 52: Success (Plays Green 2)
54. Turn 53: Success (Discards chop)
55. Turn 54: Success (Reveals Green 4 identity)
56. Turn 55: Success (Discards chop)
57. Turn 56: Success (Plays Yellow 5)
58. Turn 57: Success (Discards chop)
59. Turn 58: Success (5 save on Green 5)
60. Turn 59: Success (Stalling, hinting 1s)
61. Turn 60: Failure (Hinting Green is seen as play clue on Green 1, which illegal)
62. Turn 61: Success (Plays Green 1, which it thought was Green 3 because of convention)
63. Tur 62: Success (Play clue Green
64. Turn 63: Success (Plays Green 3)
65. Turn 64: Success (Hints White 5)
66. Turn 65: Success (Plays Green 4)
In summary, in this game, the human proxy followed H-group conventions for 88% of the moves and used various strategies while playing the game.
20

---

Ad-Hoc Human-AT Coordination Challenge
Successful H-Group conventions played:
Giving play clue (14x)
Responding to play clue (23x)
2 Save (1x)
5 Save (2x)
Discard chop (13x)
Fix clue (3x)
Discard known trash (3x)
Stall clue (1x)
H-Group Conventions violated:
Violating Good touch Principle (1x)
Failure to 2 save (1x)
Incorrectly gives chop focus clue (Ix)
Incorrectly responding chop focus clue (1x)
Failure to play play clue (3x)
Incorrectly give play clue (1x)
A.7. AH2AC2: Challenge Implementation Details
We have established a dedicated website for the AH2AC2 challenge at https//abhac2..com/ Participants can request to join the challenge through this website, where they will also find a leaderboard displaying existing results.
Upon submitting participation request, candidates will be notified when the challenge becomes public and will receive an API key for testing their implementations and for official evaluation. The results of candidate agents will automatically appear on the leaderboard once all games are completed. Participation in the action prediction challenge is currently optional but encouraged.
For the evaluation, we assess the performance of each candidate agent over a total of 2,000 games played with our human proxy agents: 1,000 games in the two-player setting and 1,000 games in the three-player setting. We consider all seating configurations and all combinations of agents and seating positions, including scenarios where candidates control multiple agents (except for self-play situations). The evaluation is conducted uniformly across different seating configurations to ensure fair and comprehensive assessment. This setup follows the procedure introduced in the seminal work on ad hoc teamwork evaluation by (Stone et al., 2010). Participants obtain evaluation results and information for their agents any time through our dedicated API.
CODE AND DATA AVAILABILITY
We provide our code in an anonymous repository: https: / /a////nnnnnoouuooe// wwwpe..science/raaaaaa.....
At the time of submission, the open-sourced codebase includes:
Data, including splits, used to train AH2AC baselines.
Code for training the baselines.
Pre-trained weights for all provided baselines.
An API for accessing human proxies.
Implementations for Human-AI Coordination evaluations for baselines except FCP, as the FCP implementation is in its final testing phase.
A.8. HDR-IPPO: Ablation Study
We conduct an ablation study to investigate the impact of the human data regularisation term on the performance and behaviour of agents trained with the HDR-IPPO. By systematically varying the strength of the regularisation, we aim to gain insights into role in guiding policy learning towards strategies learned during BC (in this case, human-like strategies).
21

---

Ad-Hoc Human-Al Coordination Challenge
METHODOLOGY
We focus the two-player setting, where BC has demonstrated strong performance as a baseline. Due the computational and time constraints associated with training HDR-IPPO agents, it was not feasible extend this study to the three-player setting. To systematically analyse the effect of the KL regularisation term introduced in the HDR-IPPO algorithm, adopt the following methodology:
1. We train new BC policy that serves both starting point for subsequent HDR-IPPO training and baseline for comparison in this ablation study.
2. From the baseline BC policy, we train multiple HDR-IPPO agents. These agents share identical architectures, hyperparameters, and training procedures, with the sole exception of the human data regularisation weight, A. We vary the weight of regularisation term across range of values, from no regularisation (A 0.00) very high regularisation (A = 0.70) to comprehensively investigate the effects of this term during both training and evaluation. The final policy weights for each agent are stored for subsequent analysis.
3. We evaluate, analyse and compare the trained HDR-IPPO agents and the baseline BC policy. Precisely, we:
(a) Assess the SP performance of each agent understand how the strength of KL. regularsaationn influences its ability play Hanabi effectively on its
(b) Evaluate the cross-play performance each HDR-IPPO agent when paired with the baseline BC policy. Higher scores this setting indicate that the HDR-IPPO agent has learned strategies compatible with the human-like conventions exhibited by the BC agent.
(c) Conduct rross-play evaluations among different HDR-IPPO agents to identify potential coordination patterns.
(d) Evaluate each agent on the held-out validation and test sets of human data. Good performance on this data indicates that the agent has effectively retained the conventions observed in the human demonstrations.
(e) Plot the KI. divergence between the action distributions of each HDR-IPPO agent and the baseline BC policy throughout the training process. This visualisation allows us to identify potential convergence or divergence patterns for the KL divergence term.
EXPERIMENTAL SETUP
The hyperparameters used for training the agents in this ablation study are listed in Table 12.
22

---

Ad-Hoc Human-Al Coordination Challenge
Table 12. Hyperparameters used for training agents in the ahlation study. We showcase both BC and IPPO hyperparameters single table.
Hyperparameter
Value
Network
Num Players
2
Activation
GELU
LSTM Layers
512
Input Embedding
1024
Decoder ML.
Dropout
0.5
BC
Optimisation
Optimiser
Adam (Kingma & Ba, 2014)
Batch Size
128
LR Schedule
Linear
Initial LR
0.005
Final LR
1.0e-05
Epochs
70
Training
Permute Colours
Yes
Self-Play Eval Games
5000
IPPO
Optimization
Learning Rate
0.0005
Gamma Discount
0.99
GAE Lambda
0.95
Clip Epsilon
0.2
Entropy Coefficient
0.01
Value Function Coeff
0.5
Max Gradient Norm
0.5
Update Epochs
4
Num Minibatches
4
Critic Network
Critic ML.P
512
Training
Total Timesteps
2e10
Environments
Num Env Steps
128
Num Train Envs
1024
Num Eval Envs
512
Num Test Actors
1024
Num Test Envs
512
In addition the listed parameters, consider setups with the following human data KL regularisation weights: 0.00, 0.01, 0.08, 0.13, 0.20, 0.30, 0.50, and 0.70. This range allows us explore the effects of regularisation from complete absence to very strong influence on the learning process.
RESULTS AND DISCUSSION
We start by presenting the cross-play matrix in Figure
From the cross-play matrix, we observe consistently high SP scores across all agents, indicating that all variations of HDR-IPPO lead to significant improvements over the baseline BC policy's mean SP score of 19.53. The HDR-IPPO agents with the lowest SP scores those at the extremes of the regularisation spectrum: the non-regularised policy (A 0.00) and the one with the highest regularisation (A 0.70). The agent showing the best SP performance, with mean score of 23.42, is the one trained with X = 0.13.
This result provides initial empirical evidence supporting the effectiveness of the regularisation employed in HDR-IPPO. We hypothesise that excessively high regularisation weights might prevent the policy to generalise, while the complete absence of regularisation could lead to the adoption of strategies that deviate significantly from human- conventions.
23

---

Ad-Hoc Human-Al Coordination Challenge
Means Scores
Median Scores
A-0.0
22.33
80
20.03
20.76
20.45
13.30
27.00
13.09
12.33
A=0.0
23.00
22.00
3..00
21.00
21.00
21.00
23.00
20.02
10.00
A-0.01
2:00
22
22.39
22.26
21.00
23.44
21.33
13.30
15.00
X-0.01
22.00
23.00
00
23.00
22.00
22.40
22.00
21.02
21.00
20
>=0.08-
22.30
23.34
2328
22.80
22.85
X=0.08
21.02
25.00
24.00
23.00
21.05
Candidates
Dm0.13
15
Candidates
15
k=0.20
20.43
23.29
25.23
23/22
22:83
22:31
A=0.20
20,00
24.00
24.00
24:00
23:00
25.02
0=0330
19.39
24
23.19
23.24
25.21
25.88
22.43
22.7
21.99
x=
21,00
22,00
01
25.10
24.00
23.00
23:00
23.05
23.00
000.5
23.01
22.39
21:71
0=0.5
20.00
23.03
5
5
0=7.7
22.83
22.57
21
0=0.0
23.0
BC
1251
16
22
22.37
22:11
11.66
21.31
21.32
14
BC
19/0
m
2m
7000
23.00
21.00
71.00
21.02
20
Aw0.0
Aw
Aw
Aw
Aw 20
Aw
Aw 50
Ams, 70
Aw0.0
Aw0..01
Aw0.13
Aw
.g
Aw 50
Aw0.70
Teams
Teams
Figure 5. Cross-pla performance matrix. Each element (i,j) represents the average score achieved by team comprising of Agent j and Agent i, averaged over possible permutations of player positions. The average score each team calculated based on 25,000 games per permutation.
Next, we examine the distribution of perfect and zero-score games. As shown in the Table 13 below, with an appropriate level of regularisation, with score of zero. In contrast, the non-regularised policy (A 0.00) yields only 143 perfect games, and its minimum score is comparable to that of the policy with À = 0.13, which boasts 1599 perfect games. This discrepancy further supports our hypothesis that these agents employ fundamentally different strategies.
acquire policy that achieves a high number of perfect games while completely eliminating games
Table 13. Perfect and zero-score games count out of 5,000 SP games. We also show the minimum score in the set of 5,000 games.
Perfect
Zero
Min
À 0.00
143
13
À = 0.01
247
0
14
À = 0.08
1351
14
0.13
1599
13
À 0.20
1821
0
15
À 0.30
1564
3
0
À 0.50
1280
21
0
A 0.70
1166
113
O
Let us now analyse the cross-play results from the perspective of the BC policy. Examining the BC policy column in Figure 5, we observe that tross-play scores generally increase compared to the initial BC SP score when paired with most HDR-IPPO agents. However, notable exception occurs for pairings with agents trained using À = 0.00 and À = 0.01. Despite these two policies having significantly higher SP scores than the BC policy, their coordination with the BC agent is poor, resulting in a substantial drop in cross-play performance. This widening gap between SP and cross-play scores indicator of poor coordination. Additionally, the policy with the highest regularisation weight (A 0.70) does not exhibit this coordination breakdown, even though it performs worse in SP compared to the non-regularised agent (A = 0.00). This observation provides empirical evidence that training with insufficient regularisation can lead to divergence from the strategies encountered in the dataset used for BC, even when the final policy develops strong strategies and performs well in isolation.
recognised
Furthermore, let's examine the cross play results from the perspective the non-ragularised policy (A = 0.00). A clear trend emerges where cross-play performance deteriorates as the regularisation weight of the partner policy increases. This
24

---

Ad-Hoc Human-AI Coordination Challenge
suggests that the non-regularised policy, having diverged from human-like conventions, struggles coordinate effectively with agents that adhere more closely to those conventions. In contrast, for policies trained with higher regularisation weights, we observe the opposite trend. The gap between SP and cross-play scores diminishes, and in some cases, cross-play even yields higher scores than the weaker policy's SP performance. This indicates that these policies, guided by the KL regularisation term, converge towards similar strategies, enabling them to coordinate exceptionally well, even surpassing individual SP scores in certain instances. The same holds true when pairing these agents with the baseline BC agent, further underscoring their compatibility with human-like conventions. This analysis provides strong empirical evidence that policies trained with higher regularisation weights tend to converge shared set of strategies, and that those strategies align closely with ones learned during the BC procedure.
We now shift our focus to evaluating the agents on the held-out validation and test sets, as presented in Table 14. Increasing the regularisation weight generally leads to improved performance on the held-out data, suggesting better adherence to the conventions present in the training set. Notably, the increase from À = 0.00 to = 0.08 results a substantial improvement of over 20% accuracy on both the validation and test sets. Importantly, subsequent increases yield only marginal gains.
Importantly, agents that perform well in cross-play tend have a lot stronger performance on the held-out datasets. Hence, we again show that they converge similar conventions, but we also show that these conventions are very close ones learned during training.
Table 14. Performance on hold out sets for all agents. We show the best result in bold.
Val Loss
Val Ace
Test Loss
Test Acc
BC
0.466
0.676
0.468
0.674
X = 0.00
7.368
0.330
7.385
0.327
A = 0.01
1.354
0.435
1.369
0.433
À 0.08
0.716
0.574
0.725
0.571
À = 0.13
0.618
0.607
0.628
0.605
À 0.20
0.540
0.636
0.548
0.634
À 0.30
0.488
0.659
0.495
0.656
À 0.50
0.475
0.665
0.481
0.663
À 0.70
0.464
0.671
0.469
0.668
Finally, we turn our attention to the evolution of the KL divergence term throughout the training process. Figure 6 illustrates the KL divergence for all trained HDR-IPPO policies. It is immediately apparent that the KL terms for A 0.00 and A = 0.01 are significantly higher than the others, rendering the remaining curves barely visible in the plot. This observation aligns with our previous findings, further reinforcing the notion that insufficient regularisation can lead to substantial divergence from the human-like strategies learned through BC.
To gain clearer understanding of the KL divergence dynamics for policies with higher regularisation weights, we present Figure 7, which excludes the policies with À = 0.00 and X = 0.01. While policies with lower regularisation weights show increasing KI. divergence during training, those with higher weights demonstrate a decreasing trend. This suggests that stronger regularisation effectively prevents the policy from deviating too far from the human-like strategies captured by the BC policy.
We do observe an initial increase in KI. divergence for all policies during the first few update steps. This likely attributable to the dominance of the IPPO loss over the KL divergence term in the early stages of training, particularly as the value function is being learned from scratch.
In conclusion, our ablation study provides compelling evidence that the KL regularisation term in HDR-IPPO effectively prevents divergence from the human-like strategies leamed through BC. However, it is crucial to set the regularisation weight, X, to a sufficiently large value to ensure sustained adherence to these conventions throughout the training process, particularly in the context of extended training duration.
A.9. Baselines: Hyperparameters
For OBL, we use open-sourced weights and hyperparameters (Hu et al., 2021).
25

---

Ad-Hoc Human-AT Coordination Challenge
KL Regularisation Term (Unweighted)
KL KL LOSS
4
3
2
20k
40k
60k
80k
100k
120k
140k
A=0.70 - 2=0.50
- 2=0.30
- 2=0.20
- >=0.13 - 2=0.08
2=0.01
- 2=0.00
Figure 6. KL divergence throughout training for trained HDR-IPPO policies.
KL Regularisation Term (Unweighted)
KL LOSs
0.3
0.25
0.2
0.15
0.1
0.05
Step
0
20k
40k
60k
80k
100k
120k
140k
>=00770
2=0.50
2=00330
2=0.20
2=0.11
2=0.08
Figure KL divergence throughout training where À 2 0.08.
26

---

Ad-Hoc Human-AI Coordination Challenge
Table 15. Hyperparameten used for training BC, HDR-IPPO baselines 1,000-game data limit challenge. BC policies trained as baselines are used for starting points HDR-IPPO later for BR-BC.
Hyperparameter
Two-Player Setting
Three-Player Setting
Network Architecture
Num Players
3
Activation
GELU
GELU
LSTM Layers
512
512
Input Embedding
1024
512
Decoder MLP
256
256
BC
Optimisation
Batch Size
32
128
Dropout
0.5
0.5
LR Schedule
Linear
Linear
Initial L.R
0.005
0.005
Final LR
0,0001
0.0001
Epochs
70
50
Training
Permute Colours
Yes
Yes
Self-Play Eval Games
5000
5000
IPPO during HDR-IPPO
Optimisation
Learning Rate
0,0005
0.0005
Linear Schedule
True
True
Gamma Discount
0.99
0.99
GAF Lambda
0.95
0.95
Clip Epsilon
0.2
0.2
Entropy Coefficient
0.001
0.001
Value Function Coeff
0.5
Max Gradient Norm
0.5
0.5
Update Epochs
Num Minibatches
4
Critic Network
Critic MLP
512
512
Training
BC Policy KL Weight
0.25
0.25
Total Timesteps
le10
le10
Environments
Num Env Steps
128
128
Num Train Envs
1024
1024
Num Eval Envs
512
512
27

---

Ad-Hoc Human-AI Coordination Challenge
Table 16. Hyperparameten used for training BC, HDR-IPPO baselines 5,000-game data limit challenge. BC policies trained as baselines are used for starting points HDR-IPPO later for BR-BC.
Hyperparameter
Two-Player Setting
Three-Player Setting
Network Architecture
Num Players
3
Activation
GELU
GELU
LSTM Layers
512
512
Input Embedding
1024
512
Decoder MLP
256
256
BC
Optimisation
Batch Size
128
256
Dropout
0.5
0.5
LR Schedule
Linear
Linear
Initial L.R
0.005
0.005
Final LR
0.0001
0.0001
Epochs
50
50
Training
Permute Colours
Yes
Yes
Self-Play Eval Games
5000
5000
IPPO during HDR-IPPO
Optimisation
Learning Rate
0,0005
0.0005
Linear Schedule
True
True
Gamma Discount
0.99
0.99
GAF Lambda
0.95
0.95
Clip Epsilon
0.2
0.2
Entropy Coefficient
0.001
0.001
Value Function Coeff
0.5
Max Gradient Norm
0.5
0.5
Update Epochs
Num Minibatches
4
Critic Network
Critic MLP
512
512
Training
BC Policy KL Weight
0.25
0.25
Total Timesteps
le10
le10
Environments
Num Env Steps
128
128
Num Train Envs
1024
1024
Num Eval Envs
512
512
28

---

Ad-Hoc Human-AI Coordination Challenge
Table 17. Hyperparameters used for training all IPPO and BR-BC bascline agents. Here, use feed-forward architecture. BC agents used for BR-BC are the same shown in Table 15 and 16, depending on the challenge variety.
Hyperparameter
Two-Player Setting
Three-Player Setting
Network Architecture Num Players
2
Activation
RELU
RELU (512, 512)
MLP
(512, 512)
IPPO
Optimisation
Learning Rate
0.0005
0.0005
Gamma Discount
0.99
0.99
GAE Lambda
0.95
0.95
Clip Epsilon
0.2
0,2
Entropy Coefficient
0.01
0.01
Value Function Coeff
0.5
0.5
Max Gradient Norm
0.5
0.5
Update Epochs
Num Minibatches
4
Critic Network
Critic MLP
512
512
Training
Total Timesteps
1e10
le10
Environments
Num Env Steps
128
128
Num Train Envs
1024
1024
BR-BC
BC Anneal Start
le9
le9
BC Anneal End
6e9
699
29

---

Ad-Hoc Human-AT Coordination Challenge
A.10. LLM Prompts: Additional Details
To benchmark off-the-shelf large language models (LLMs) in cooperative play, transform every Hanabi observation into natural-language prompt and perform zero-shot inference, i.e., without parameter updates or retrieval augmenta- tion (Lewis et al., 2020). Modern work on chain-of-thooee ofouthhht prompting demonstrates that LLMs reason more reliably when instructions are explicit and structured (Wei et al., 2022). Conversely, long uncurated contexts increase the risk of eall the "lost-i-tthemmidlle" error in which models ignore mid-prompt evidence (Liu et al., 2023). We therefore send the model only (i) Hanabi rules prompt segment, (ii) the H-Group conventions, and (ii) the observation for the current game state. These prompt segments are populated from parameterised templates so that any variant (player count, token budget, deck composition) can be generated dynamically.
A.10 HANABI RULES PROMPT
This prompt segment embeds a complete description of the Hanabi rules. It specifies deck size, token mechanics, legal actions, hand indexing, and end-game triggers. All imperative verbs match the action names used later in the "Valid Actions" list, so parser can easily identify the LLM's choice:
You are playing the card game Hanabi, the three player variant. Every turn I will give you the state of the game, the actions that your teanmates just took, your teammates hands, and your hand. You will then need to choose an action to take based on the state of the game.
Hanabi Game Rules
## Overview and Objective:
Hanabi is cooperative card game in which players take turns working together to build firework card sequences in five colors.
Each sequence must begin with a rank card and continue increasing order up rank 5.
The goal is complete many fireworks stacks as possible; a perfect game scores 25 when every color stack is completed.
## Game Components:
Deck: The deck consists of cards in five colors (Red, Yellow, Green, White, Blue). For each color there are three copies of rank 1, two copies of rank 2, two copies rank 3, two copies of rank 4, and only one copy of the rank card.
Clue Tokens: There are 3 clue tokens available, These
spent when giving clues and can be regained by discarding card or playing a 5.
Life Tokens: There are 8 life tokens available. Each misplayed card causes the loss of one life token; if all three are lost, the game ends immediately, and all players lose.
Firework Stacks: There is one stack for each color. Cards are added these stacks in ascending order.
Discard Pile: A common area where discarded misplayed cards are placed that all players can see.
Player Hands: Each player's hand is arranged so that other players can see the cards, but the owner cannot see their own cards. so you can't see the identies of your own cards, but you know the identies of your teannates cards.
Deck Draw Pile: The remaining deck from which new cards are drawn.
# Game Turn and Actions:
on each turn, player must take one of the three following actions (Give clue, Discard Card, or Play a Card).
30

---

Ad-Hooc Human-AI Coordination Challenge
The other teammates in the game are Teanmate 0 and Teammate 1. After your turn, Teammate will take their turn, and then Teammate will take their turn after Teamnate 0.
After Teamnate 1 takes their turn, you will take your turn again, and the game continues in this way until the game ends.
## Give a Clue:
Provide information about either all cards of a specific color or all cards of specific rank in one of your teammates hands. The clue will identify every card in the teamnates hands that matches the given clue information (color or rank)
-Tiis action consumes 1 clue token.
## Discard a Card: Choose one card from your hand to discard.
This action will regain clue token (up a maximum 8).
This action will draw
card from the deck if one is available.
# Play a Card: Choose one card from your hand and attempt to play it on the corresponding firework stack.
If the card the next rank number in sequence (or rank card for an empty stack), the play is successful and the card is added the stack. If the card played is of rank 5, 1 clue token is regained for the team (if not already at the maximum)..
If the card does not match the required sequence, it is misplay: The team loses one life token and the card is discarded. The game ends immediately with a score of if all three life tokens are lost (bad).
This action will draw a new card from the deck if one is available.
## Card Positions:
Slots: Your hand comprises cards with slots 0, 1, 2, 3, 4, with slot 4 as the leftmost card and slot 0 as the rightmost. The order of all the slots are: slot 4: leftmost, slot 3: second leftmost, slot 2: middle card, slot 1:
second rightmost, slot 0: rightmost.
After play/discard: When player plays or discards card from slot their hand, the card is removed from the hand, and all cards in higher-numbered slots are shifted down (to the right) one slot.
New cards: Any drawn card enters into slot 4 (the leftmost position).
## Game End Conditions:
When the deck is empty, each player gets one final turn, including the player that drew the last card.
The game ends inmediately if all three life tokens are lost, and the players get a score of 0 (very bad).
After the final round (once the deck is exhausted and all players have taken their last turn), the game ends.
A perfect game occurs if all five firework stacks are completed up to card rank
5 (score 25).
# Scoring:
If all three life tokens lost, the game ends with score of 0 (very bad)
31

---

Ad-Hooc Human-AT Coordination Challenge
Otherwise, the final score is the sum of the highest played rank card on each rirwork pile.
The maximum possible score is 25, achieved when every fireworks stack is completed.
## Possible Actions:
when you take an action, here are all of the possible actions you may take:
Discard card in slot 0 from your hand
Discard card in slot 1 from your hand Discard card in slot 2 from your hand Di scardr card in slot 3 from your hand
Discard card in slot 4 from your hand
Play card in slot 0 from your hand
Play card in slot 1 from your hand
Play card in slot 2 from your hand
Play card in slot 3 from your hand
Play card in slot 4 from your hand
Clue Red to Teammate 0
Clue Yellow to Teammate 0
Clue Green to Teammate
Clue White to Teammate
Clue Blue Teammate
Clue Red Teammate
Clue Yellow to Teammate
Clue Green to Teammate 1
Clue White to Teammate 1
Clue Blue to Teammate
Clue Rank to Teammate
Clue Rank to Teammate
Clue Rank
Teammate
Clue Rank 4 Teammate
Clue Rank 5 to Teammate
Clue Rank to Teammate
Clue Rank to Teammate
Clue Rank to Teammate
Clue Rank to Teammate
Clue Rank Teammate
A. 10.2. H-GROUP CONVENTIONS PROMPT
For experiments with convention-awar agents-and and to mirror the play of our human-proxy agent-we include the Level 1 H-Group 'onventions', a small, community-agreed protocol that teaches players to share just enough information to coordinate safely. These rules tell an agent how to signal "this card is safe to play next" or "don't discard this card," and they precisely define which card in your hand each hint refers to. By using these rules, every agent (LLM or human-proxy. speaks the same simple clue vocabulary, letting measure exactly how much these rules improve teamwork.
When you choose an action to take, always the following convention rules as the shared protocol for giving and interpreting clues---so you and your teammate can coordinate discards, saves, and plays unambiguously. The following rules are written with respect to you, but the same rules apply to your teammates. The rules are written in the first person, but they apply to all players.
hhtpps:/hanabi.giithb githhb.io/
32

---

Ad-Hoc Human-AT Coordination Challenge
Conventions Rules
# Chop:
Definition: Your chop is the rightmost card (smallest slot value) in your hand that has not received any clues.
Instructions: When forced to discard, always discard your chop. If a card in your hand has been clueed as useless (and you can clearly tell it won't help because it's already been played), you may discard that card instead the chop.
Reminder: If the rightmost slots of your hand (0, 1, etc) have received clues, then those slots are NOT your chop, your shop is the rightmost card that has not received any clues.
## Clue Interpretation:
(Card slot position reminder): Your hand comprises cards with 5 different slots, with slot 4 as the leftmost card and slot 0 as the rightmost. The order of all the slots are:
slot 4: leftmost, slot 3: second leftmost, slot 2: middle card, slot 1:
second rightmost, slot 0: rightmost.
Single Card Focus: When a clue touches two or moze cards, it conveys information about only one specific card/the focused card; non-focused cards receive no actionable instruction. The focus card must always be either Save Clue or Play Clue.
New Cards: Cards are \new" if they had clues them prior to this clue. Instructions for determining which card is the focus when a clue touches
multiple cards:
1.
One New Card: If exactly one card is newly clued (had no prior clues), that card is the focus of the clue.
2. Multiple New Cards (chop focus) : If more than one card is newly clued and the chop is included in the clue, the chop card is the focus.
3. Multiple New Cards (not including chop): If more than one card is newly
clued and the chop is not included in the clue, the leftmost new card is the focus.
4. No New Cards: If the clue only touches cards that already had clues, the leftmost re-clued card is the focus.
Clue Type:
If the chop is included the clue, the chop card is the focus and it is either playable card (if it's Play Clue) or critical card (if it's Save Clue).
If the chop card is not touched by the clue, the focus card (leftmost newly clued card) is a Play Clue or a Delayed Play Clue.
## Play Clues:
Play Clue (Direct):
Definition: clue given signal that the focused card is immediately playable right now (it is the next needed card its firework stack).
rnstructioss: If the chop card is not touched by clue, then the focus card is always the leftmost card, and it is always a Play Clue. When giving a Play Clue, ensure that the focused card will fit directly onto its firework. The Play Clue tells your teammate that their focused card is playable right now. Instructions: If the chop card is touched by a clue, then the focus card can either be Play Clue or Save Clue, it's up to the player to figure out which type of clue it is based on what cards are critical. All Play Clues are interpreted as potential Delayed Play Clues.
33

---

Ad-Hoc Human-AT Coordination Challenge
Delayed Play Clue:
Definition: clue given card that is net immediately playable because an earlier card that has received Play Clue has not been played yet; however, once that missing card is played, the clued card will become playable.
Instructions: When giving a Delayed Play Clue, make sure that the missing card will be played soon by either yourself or your teammate. Interpret any such clue as promise that, after the necessary preceding card is played, the clued card will be safe to play.
# Critical Cards:
Definition: A Critical Card is the last copy of a card of color and rank combination that hast not been discarded yet, where discarding this critical card makes inpossible to achieve a perfect score.
Examples: (only one copy in each color), a unique rank card (if both other copies are have been discarded), or any rank 2, 3, or 4 card whose other copy has been discarded.
Instructions: Always treat critical cards high priority for saving. If critical card becomes the chop card, it must receive Save Clue to ensure it isn't discarded.
Save Clues:
Definition: Clues used to protect critical cards from being discarded. Save Clues can only be given to cards on the chop.
Instructions:
Save Clues can only be given to cards on the chop. If a clue touches the chp card, it is either a Save Clue or a (Delayed) Play Clue. It's up to you to figure out which type of clue it is based on what cards are critical.
Use Save Clues to safeguard cards that are vital for perfect score 25. Critical Save: A clue given to a critical card the chop to save it from being discarded. You can give a Critical Save clue with eithr a color or number clue.
5 Save: when saving a 5 card on the chop, you must always give a rank 5 clue to indicate that the clue is a 5 Save, not a color clue because a color clue could be interpreted as play clue.
2 Save: All rank 2 cards on the chop should be saved with a 2 Save clue if it's the only copy of that card visible in any players hand, even if 2 are not critical. When saving rank 2 card the chop, always give rank 2 clue to indicate that the card is 2 Save, a because color clue could be interpreted as a Play Clue.
If a clue touches any card that is not on the chop, interpret it as a (Delayed) Play Clue, not a Save Clue.
When receiving a Save Clue on your chop, do not discard that kerd it safe until it becomes playable.
All Save Clues, Critical Save Clues, Saves, and Saves can only be given to cards on the chop.
## The Three Main Principles:
1. Good Touch Principle: Only give clues to cards that are have not been played yet; avoid clueing cards that have already been played.
34

---

Ad-Hoc Human-Al Coordination Challenge
2. Save Principle: Including cards that are saved with Save Clues, do not allow other players to discard playable cards. All cards that are playable need to be "protected" by giving them a Play Clue. The following cards must not be discarded: All rank 5 cards, Unique rank 2 cards with only one copy visible, Critical cards with only one copy left undiscarded, and Unique playable cards that are the only copy currently visible.
3. Minimum Clue Value Principle: Every clue must either make one or more cards critical card. If a clue does not make card playable prevent the discard of a critical card, you should discard instead of wasting clue.
safely playable or prevent the discard
# The Early Game:
Definition: The phase before any player has ever discarded their chop card.
Instructions:
During this phase, use every available Play Clues and Save Clues to protect critical cards on chop (including 5 and Saves)
Only discard if there no valid Play Save Clue available.
Discarding for the first time ends the Early Game and begins the Mid-Game.
# General Strateqy:
Check Teammate Chops: Always review the rightmost unclued cards in every player's s hand to identify which ones need protection.
Clue Selection: Prefer giving Play Clues when possible over Save Clue, as they not only protect critical cards by giving the player something else to do (playing the card) but also facilitate immediate or near-future plays. Use Save Clues only when a critical card on the chop is at risk of being discarded (the player has nothing else to do on their turn).
Clue Type: Prefer to use color clues over rank clues for precise information about the card's identity unless a rnak clue can secure additional plays or prevent vital discard.
Safe Discards: Discard only cards that clearly non-critical. Protect any card that might be needed for perfect score.
Identify Meaning of Clues: When it's your turn, and you' re trying to interpret clue, always follow the follwing algorithm: Identify which card slot in your hand is the focus of the clue, and then decide on if the clue focus card is a Play Clue or a Save Clue.
Ever clue you give always needs to be valid Save clue when the card you want Play/Delayed Play Clue following the rules above.
to save is the chop,
## Prompts:
Definition: A Prompt is clue on a (currently) unplayable card, but it directs a player to immediately play a clued card (which is the card that can be played before the clued card) that they would not normally play right now because they previously didn't have enough information about the cards identity. Instructions:
The player receiving the prompt to play the connecting card before the clued card can either be the player receiving the Play Clue, or any other player in the team.
If more than one card could be intezpreted as the card that is being prompted to play, the player being prompted should play the lefrmost eligible clued card.
when you give a Prompt, the intended action is for the recipient play the prompted card as soon possible.
35

---

Ad-Hooc Human-Al Coordination Challenge
Use Prompts sparingly and only when you are confident that the focused card is safe to play without further clues.
# Finesse:
Definition: A move where clue to one player impl ies that another player must bland-play the connecting lower card (one rank below) from their Finesse Position (leftmost unclued slot) even though it has no clues.
Finesse Position: The card slot containing player's leftmost unclued card; this position slides right whenever leftmost cards receive clues
Trigger: You see an unplayable card clued as Play Clue. If the required lower-rank card is not visible anywhere with clues (the clue is prompt), assume the player immediately before the clued card holds it in their Finesse Position and must blind-play it.
Priority: Prompts override Finesses. If you must choose between playing a prompted card and blind playing a card in finesse position, play the prompted card first.
Urgency: Blind-play into Finesse inmediately to resynchronise team knowledge; delaying risks desynchronised information. Instructions (for the finessed player that needs to blind-ple ppay their finessed card)
1. Identify your current Finesse Position (leftmost unclued slot).
2. Blind-play that card at once, assuming is the connecting card rank below the clued card. Instructions (for the clue-giver):
You can only give a finesse clue to Teammate 1, which would trigger Teammate 0 to blind-play their card in their Finesse Position before Teamamte 1's turn.
Ensure the clued higher card is unplayable and that the card 1 rank below should exist in Teammate 0 Pinesse Position.
Give the higher card clue (usually its color or rank) that unmistakably Play Clue.
marks it
A.10.3. HANABI STATE PROMPT
The final prompt section serialises the simulator observation into an I.MM-reddalle text block: game state counters, public firework stacks, the discard pile, teammates' hands, and information about your own hand. At the end of the prompt we require the LLM provide its response using simple JSON response schema. We ask it to fill four fields helds-game state, action.options, best_action and rationale so that understand why it chose the action.
Hanabi Game State
## General Info:
Game Turn: 39
Score: 16
Life Tokens: 3
Clue Tokens:
Deck Size: 12
## Fireworks:
(Last card played on each stack, 0 means card has been played that stack
yet) x
Current Stacks: Red: 3, Yellow: 2, Green: 2, White: 5, Blue: 4
## Discard Pile:
cards: Red 1, Red 4, Yellow 2, Yellow 3, Yellow 4, Green 4, Blue
36

---

Ad-Hoc Human-AT Coordination Challenge
## Teammate 8 Last Action Two Turns Ago: Action Type: Clue Rank, Clue: 2, Clue Recipient: Teamnate 1, Affected card slots in teammate I's hand: [4]
## Teamnate I's Last Action on The Prevous Turn: Action Type: Play, Slot: 4, Card: Yellow 2, Successful Play: Yes
## Your Hand:
(Card identities unknown to you)
Slot 0: Clues (Color: None, Rank: 4), Possible Colors: [Red, Yellow, Green,
White], Fossible Ranks: [4]
Slot 1: Clues (Color: None, Rank: 5), Possible Colors: [Red, Yellow, Creen],
Possible Ranks: [5]
Slot Clues (Color: None, Rank: None), Possible Colors: [Red, Yellow, Green,
Whitel, Possible Ranks: [1, 2, 3, 4, 5]
Slot 3: Clues (Color: None, Rank: None), Possible Colors: [Red, Yellow, Green,
white], Possible Ranks: [1, 2, 3, 4, 5]
Slot 4: Clues (Color: None, Rank: None), Possible Colors: [Red, Yellow, Green, White, Blue], Possible Ranks: [1, 2, 3, 4, 5]
Teammate 0 Hand:
(Card identities known you)
Slot 0: Card: *Blue 3x, Clues (Color: None, Rank: None), Possible Colors: [Red,
Yellow, Green, Blue], Possible Ranks: [1, 3, 4]
Slot 1: Card: +Yellow 4+, Clues (Color: None, Rank: None), Possible Colors:
[Red, Yellow, Green, Blue], Possible Ranks: [1, 3, 4]
Slot 2: Card: *Red 1+, Clues (Color: None, Rank: None), Possible Colors: [Red,
Yellow, Green, White, Blue], Possible Ranks: [1, 2, 3, 4, 5]
Slot Card: *Green 5x, Clues (Color: None, Rank: None), Possible Colors: [Red,
Yellow, Green, White, Bluel, Possible Ranks: [1, 2, 3, 4, 5]
Slot 4: Card: <Witte 4x, Clues (Color: None, Rank: None), Possible Colors: [Red,
Yellow, Green, White, Blue], Possible Ranks: [1, 2, 3, 4, 5]
## Teammate l's Hand:
(Card identities known to you)
Slot 0: Card: *Red 4t, Clues (Color: Red, Rank: None), Possible Colors: [Red],
Possible Ranks: [1, 4, 5]
Slot 1: Card: xYellow 3x, Clues (Color: None, Rank: 3), Possible Colors:
[Yellow, Green, White], Possible Ranks: [3]
Slot 2: Card: *Blue 5*, Clues (Color: Blue, Rank: None), Possible Colors:
[Blue], Possible Ranks: [1, 4, 5]
Slot 3: Card: *hite 3x, Clues (Color: None, Rank: None), Possible Colors: [Red,
Yellow, Green, White, Blue], Possible Ranks: [1, 3, 4, 5]
Slot 4: Card: *Yellow 1x, Clues (Color: None, Rank: None), Possible Colors:
[Red, Yellow, Green, White, Blue], Possible Ranks: [1, 2, 3, 4, 5]
## Valid Actions:
(Choose one action exactly as written)+ Discard card in slot 0 from your hand
Discard card slot from your hand
Discard card in slot 2 from your hand
Discard card in slot 3 from your hand
Discard card in slot 4 from your hand
37

---

Ad-Hoc Human-AT Coordination Challenge
Play card in slot 0 from your hand
Play card in slot 1 from your hand
Play card in slot 2 from your hand
Play card in slot 3 from your hand
Play card in slot 4 from your hand
Your Instruction:
Please consider all of the Rules and the current Game State, and decide on the best action to take from the "Valid Actions" list.
You cannot play discard cards from your teammate's hand. Only play card from your hand if you are certain it is playable based on hints or deduction based on the rules. Check fireworks status before playing. When hints are available for your cards (e.g., 'Colour Hint: Blue' or 'Number Hint: 31), use them. Evaluate possibilities like 'Colours: [Blue]' or 'Ranks: [3]'. When only one 1ife remains, extremely cautious about playing cards.
Response Format
You MUST output responses strictly in JSON format. Adhere EXACTLY to the JSON schema and content requirements provided below.
## JSON Schema:
.. json
"game_state" "<str>",
"action_options": "<str>",
"best_action" "<str>", "rationale": "<str>"
## Content Requirements:
"game_state"; Provide summary of the current game state. Include teammate actions, your previous actions, and incorporate any outstanding key reminders from earlier actions, if available, and any important things to taken into account when choosing an action to take. How do you interpret the meaning of the other players previous actions?
"action_options"" List the most promising action options from the Valid Actions list, along with their pros and cons, and why each of them are promising actions. Say the actions exactly as they are written in the Valid Actions list. "best_action" Select the best action to play right now. Say the action exactly as it is written in the Valid Actions list.
"rationale": Provide an explanation for why the selected action is the best choice compared to the other options.
## Instructions:
1. Generate ONLY single JSON object matching the schema.
2. Do NOT include any text outside the JSON object (e.g., explanations, markdown formatting like ""ijsn, apologies, or status messages).
3. Ensure all required fields from the schema are present in the JSON.
38


Reviews:
[{"review_id": "YSwzc58UO4", "reviewer": "Reviewer_SVSM", "summary": "The authors present a new test-bench for Human-AI collaborative RL using Hanabi.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your feedback. We appreciate the opportunity to address your concerns and clarify the contributions of our work.\n\n## Adaptivity of Human-Proxy Agents\n\nYou raised an important concern about whether using human proxies, trained using fixed parameters, sufficiently captures the dynamism necessary for Hanabi, where players adjust strategies based on their partners’ behavior. Critically, fixed neural parameters do not imply static behavior; our proxies condition on the history of the game, which includes partner actions. Specifically, when the proxy agents see an unexpected action or observation, from that action-observation history onward, they will account for the fact that the other agent is using a different convention. The proxies are trained using SOTA methods and on a diverse dataset of human gameplay data.\n\nSome concrete evidence that the proxies are capable of adaptivity includes:\n\n-  **Action Prediction Results (Tables 3 & 4):** The proxies accurately predict human actions on unseen human gameplay data, which comes from players of various skill levels, who follow different levels of H-group conventions. Such accuracy requires the ability to infer from a given action-observation history what sort of conventions the human is following.\n-  **Successful Coordination with Brittle Partners (Figure 2):** The proxies achieve high performance even when paired with behavioral cloning agents, which are known to be prone to suboptimal play. The proxies’ success even in such scenarios provides strong evidence that they make adaptations during gameplay to ensure effective coordination.\n\nWe have further clarified this in the paper and hope this addresses how the proxies reflect dynamic adaptivity similar to human play.\n\n## Real-World Relevance of AH2AC2\n\nSuccess in human-AI collaboration requires the ability to robustly generalize cooperative behaviors from limited human data. As well, the literature requires a standardized testing protocol to rigorously measure algorithmic progress. AH2AC2 directly addresses both of these fronts: we release an experimental suite consisting of carefully validated human-proxy agents, trained on extensive human gameplay data, which serve as robust and reproducible evaluation partners. As well, we open-source a deliberately limited dataset of human games to encourage research specifically on data-efficient coordination algorithms.\n\n## Novelty and Suitability for ICML\n\nWe appreciate the opportunity to clarify our decision to submit this work to ICML. Benchmark creation is explicitly recognized by ICML as a meaningful and rigorous research contribution. In particular, our benchmark required:\n\n-   **An Extensive Original Effort:** Large-scale human data collection, meticulous data cleaning, validation, and rigorous modeling using SOTA methods.\n-  **Novel Evaluation Platform:** Developing reproducible human proxy models, carefully validating them, and establishing a standardized evaluation protocol for human-AI coordination in partially observable, cooperative environments.\n\nTo the best of our knowledge, AH2AC2 is the first benchmark to explicitly assess human-AI coordination in complex, partially observable settings using realistic human-proxy agents.\n\n## Conclusion\nIn light of your review, we have modified the paper to make the broader significance of our work more clear. We believe a benchmark such as AH2AC2 is highly necessary for progress in human-AI collaboration, and is to date decidedly lacking in the literature.\n\nPlease let us know how we can further improve the paper. Otherwise, if you believe these clarifications have benefited the work, we would appreciate a corresponding update in the score.\n\nThank you for your continued engagement with the review process.", "claims_and_evidence": "The main claim is the creation of a benchmark test which is present but with only one team's submissions. Thus the claims that the benchmark will impact the community or improve RL in general aren't supported.", "methods_and_evaluation_criteria": "I'm not convinced the benchmark will work as described. The use of human proxies means that the learning objective is static, Hanabi was originally picked as a target for RL as it requires dynamic planning. Optimal Hanabi agents can update their strategies as they observer the other players. A static (even if complicated) player model doesn't have this property, ditto the human games. How does success on the leader-board show improvements in Human-AI collaboration?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See Methods", "relation_to_broader_scientific_literature": "This paper claims to be building support for the literature to grow, but does not display new techniques or methods itself. I'm not sure ICML is the correct venue as a result", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The model training and evaluation looks good, but I think explaining why this paper is relevant to people outside Hanabi researchers would greatly aid the paper. The methods used to train the models do not appear to be novel, and releasing a testing procedure along with new models is also common. So, if the authors can explain the \"original and rigorous research of significant interest to the machine learning community\" component then this would be good paper, but I feel like I'm missing that.", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VcOfZGNtkT", "reviewer": "Reviewer_CEGr", "summary": "This paper proposes a new ad-hoc human AI co-ordination challenge using the game of Hanabi. The authors have trained a human proxy agent and have created a controlled benchmark environment for researchers to test their new ad-hoc coordination algorithms.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thorough review and constructive feedback. We have carefully considered your questions and provided our responses below.\n\n**Q1. Benchmark Fairness**\n\nWe thank the reviewer for raising this point and agree that it is theoretically possible for researchers to attempt scraping game data or collecting new human data independently, and believe this is a valid concern.\n\nTo clarify, the core goal of AH2AC2 is to measure how well methods perform when trained *only* on a limited dataset we provide, and using scraped data would not be fair or allowed. Having in mind that scraping is theoretically possible, as correctly noted by the reviewer, we rely on community transparency. We will strongly encourage participants to make their training code publicly available and reproducible. Also, submissions on the leaderboard will indicate whether reproducible code has been provided (and link to the codebase will be added in this case). We will update our submission system to account for this! We hope this helps reduce reviewers' concerns. Finally, while possible, scraping game data is non-trivial, and crucially, there is currently no open-source dataset of human play available.\n\nWe should also note that past benchmarks (such as Overcooked AI) relied solely on trust. We believe our approach offers a more rigorous and fair evaluation protocol – we aimed to reduce the chance of overfitting, but we can't exclude the possibility of unethical practices completely.\n\nWe hope this addresses your concerns regarding data usage and the fairness of the benchmark.\n\n---\n\n**Q2. Human Proxies and Benchmark Goals**\n\nWe thank the reviewer for raising this important point. We will update the paper to clarify both the specifics of our agents and the goals of AH2AC2. In particular: \n\n1.  **Number of Agents:** We developed *four* human proxy agents (two for the 2-player setting and two for the 3-player setting), not a single one. We will make this clearer in the paper.\n2.  **Diversity of Playstyles:** You are right that real-world human play varies significantly. Our agents were trained on a large dataset of games acquired from the `hanab.live` community. Players on this platform generally follow H-group conventions, but this does not mean they follow a single strategy. Instead, H-group conventions are a collection of diverse strategies and techniques. Players mix and match these based on their skill and the game context, leading to significant variation in play style within the dataset our agents learned from. We agree this is not clear in our original paper, and we will update the paper to provide more details.\n\nFinally, we would like to clarify our goal for the benchmark. Our approach is pragmatic and mirrors a real-world use case:\n*   There is a given (inherently meaningful) human population that an algorithm is supposed to cooperate with.\n*   We assume it is possible to collect a small dataset from this population (as is often the case).\n*   The task is then to develop a method that can do well with the source population while having access to only this small dataset.\n\nSuccess on our benchmark indicates an algorithm's ability to effectively learn cooperative strategies from limited, representative human data, which we believe is a key step towards building human-compatible agents/systems.\n\nWe hope this explanation clarifies the nature of our human proxy agents and the practical relevance of our benchmark.\n\n---\n\n**Q3. 3P OBL Weights**\n\nWe thank the reviewer for reading our paper in detail and raising this point. We contacted several authors of the original OBL paper, and they informed us that they unfortunately no longer have access to the 3P OBL weights and the weights were not released with their original work. Re-implementing OBL in JAX is a complex task, and it is beyond the scope of our current work.\n\n---\n\n**Q4. Multi-task Learning Agent**\n\nWe definitely agree that lifelong learning and multi-task learning algorithms could be highly competitive in AH2AC2. However, given the substantial complexity involved in fully implementing and tuning such an agent, we consider this outside the immediate scope of our benchmark. That said, we encourage and welcome the broader research community to benchmark such methods within AH2AC2, as these promising techniques may significantly enhance human-AI coordination.\n\n---\n\n**Q5. Seeds**\n\nThis is correct.\n\n---\nRegarding <https://openreview.net/forum?id=pCj2sLNoJq>: We thank the reviewer for highlighting this very relevant ICLR paper. We agree that this recently published method directly addresses one of our proposed future works and represents a valuable addition to our study. We plan to add this approach to our benchmark as soon as possible.\n\n---\n\nWe hope we have addressed the reviewer's comments and concerns. We are happy to discuss any of these points further. If our response has resolved your main concerns, we would be grateful if you would consider updating your score.", "claims_and_evidence": "This is a benchmark paper. The main claim is that the human proxy agent approximates human game play. Authors have done several analysis to justify this claim.", "methods_and_evaluation_criteria": "Check Q2 for my concern about the proposed evaluation criteria to measure the progress in Human - AI coordination.", "theoretical_claims": "There is no theory or proof in the paper.", "experimental_designs_or_analyses": "Yes. No complaints.", "relation_to_broader_scientific_literature": "The paper is well-positioned in the literature.", "essential_references_not_discussed": "* It is worth benchmarking multi-task learning agent from Nekoei et al, ICML 2021. (https://arxiv.org/abs/2103.03216)\n* It is worth noting that the final future work mentioned in the paper has already been  explored in this recent ICLR paper: https://openreview.net/forum?id=pCj2sLNoJq (of course this was after the ICML submission deadline)", "other_strengths_and_weaknesses": "Strength:\n\n* Great first step towards designing algorithms for better human-AI coordination.\n\nWeakness:\n\n* Having only one human proxy agent is a weakness.", "other_comments_or_suggestions": "* In page 1, what is \"broad-based progress\"?\n\n* Setting up API access for evaluation is a great idea to avoid overfitting!", "questions_for_authors": "Q1. The main premise of this paper is the collection of a large data set of two-player and three-player games. The authors have decided not to release the dataset and only release a small subset for the researchers to finetune with. However, it is not clear what will stop other researchers from scraping the dataset themselves or just collecting more episodes of human play and training an agent using that. Can you clarify whether this is allowed to participate in the leaderboard?\n\nQ2. The fact that there is only one human proxy agent is very limiting. In practice, the agent might have to deal with multiple humans and each human will have their own style and strategy. Why is it that making progress in this single proxy benchmark would lead to better human AI coordination? \n\nQ3. Why dont you have access to OBL weights for 3-player games? Did you try contacting the authors?\n\nQ4. Authors are testing OBL, OP, and FCP. Another relevant agent that can do better zero-shot coordination that is missing here is the multi-task learning agent from Nekoei et al, ICML 2021. It can also be considered as population based and I would like to see this agent benchmarked here as well.\n\nQ5. Is it right that you train 3 seeds for BC, BR-BC, and HDR-IPPO and then pick the best seed based on the validation set?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0LI8j1c6mk", "reviewer": "Reviewer_FDiM", "summary": "This paper trains a human proxy model from human gameplay records on the Hanabi game and proposes that the proxy model can be used as a cheaper evaluation for algorithms developed for human-AI coordination. They also open-sourced a smaller human dataset on Hanabi.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your positive feedback and detailed review of our paper. We especially appreciate you taking the time to read the appendices. We hope the following answers clarify the points you raised.\n\n### Question 1\n\nWe thank the reviewer for raising this point, and we agree with the point that human play is diverse. Regarding the **H-Group Conventions** mentioned (line 812), we want to clarify what this means in practice and how it relates to strategic variety.\n\nWhile our dataset mainly features games where **H-Group Conventions** are used, this term doesn't refer to a single strategy. Instead, it is helpful to think of **H-Group Conventions** as a collection of different strategies and techniques that players learn and combine. In fact, conventions are sorted into levels. Each level uses a different set of strategies that are used. Mastering even beginner H-Group levels requires learning several of these. Players often mix and adapt these strategies within a single game – depending on the player's strength. Therefore, the dataset itself naturally contains a variety of playstyles. Our action prediction results (Table 3 & Table 4) show that the trained proxies achieve good accuracy and low cross-entropy loss on unseen human data. This suggests that they have successfully learned to represent multiple strategies and patterns present in the human dataset.\n\nWe hope this explanation clarifies that the use of **H-Group Conventions** as a foundation does not overly constrain the strategic variety present in our data, and consequently, in what our proxy models represent. Finally, as a similar concern was raised by multiple reviewers, we will clarify these points in our camera-ready submission.\n\n### Question 2\n\nAdaptation is a crucial aspect to consider when developing human proxies, and we thank the reviewer for raising this point. To clarify how our proxies address adaptation, we believe it is helpful to consider two adaptation types:\n\n*   **Adaptation within a game:** This refers to adapting actions based on the current game state, partner's moves, and available information within a single game. Our human proxies exhibit this type of adaptation. While the parameters of the neural network policy are fixed after training, behaviour adapts dynamically, much like a human player reacts to changing circumstances. When the proxy agents see an unexpected action or observation, then from that action-observation history onward, they will account for the fact that the other agent uses a different convention. Thus a fixed policy does not imply fixed behaviour.\n*   **Adaptation between games/learning new strategies over time:** This means changing fundamental strategies or learning entirely new conventions. Our proxy models are not designed to do this. We want them to consistently represent the playstyle of the population found in our dataset, providing reproducible and consistent partners for evaluation.\n\nFurthermore, our experiments provide evidence for the proxy's within-game adaptability. As shown in Figure 2, BC agents (which can be quite rigid and brittle) perform better when paired with our human proxies compared to SP. This suggests that our proxies are flexible enough within a game to coordinate effectively even with simpler partners and can adjust to mistakes made by these less sophisticated policies.\n\nWe hope this clarifies the specific type of adaptation our human proxy models capture and why they are designed this way for consistent evaluation.\n\n### Question 3\n\nWe are happy to emphasize why we choose our metrics and how we intend them to be interpreted. \n- **IPP (Information per Play)** is normalised to a scale of 0 to 1. A value of 0.44 means players, on average, know slightly less than one “information” (color or rank) about the cards they play. \n- **Communicativeness** is defined as a percentage of turns where a player gives a hint, when it's possible to give a hint.\n\nThe fact that the values for these behavioural metrics are close between different models suggests that they exhibit similar tendencies regarding information usage, hint efficiency, and communication frequency. We aimed for these metrics to provide a clear comparison.\n\nPerhaps we misunderstood your concern regarding normalisation? If you could elaborate on what aspect feels unnormalized or difficult to interpret, we would be happy to provide further clarification.\n\n---\n\nWe hope these explanations address your questions. Thank you again for your valuable insights. If our responses have resolved your concerns, we would appreciate it if you would consider increasing the support for our work.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "This work does not include proofs.", "experimental_designs_or_analyses": "Yes, I reviewed the experiments in Sec 5.", "relation_to_broader_scientific_literature": "In human-AI coordination, a good human proxy model is important to evaluate the performance of trained agents. This work provides human proxy models trained from a large human dataset, open-sourced a human dataset, and built a public evaluation platform, which are helpful for future studies on human-AI coordination problems.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: This paper provides crucial components for human-AI coordination evaluation. The analysis of the human proxy model is comprehensive. The benchmark is a good evaluation for previous human-AI coordination algorithms. \nWeaknesses: Since the main purpose is to evaluate human-AI coordination, the most important aspect should be whether the model can reflect real human evaluation results. However, this paper dose not have human study results to show whether the performance against real humans aligns with the performance against human proxy models.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Humans are known to be diverse. How do you model different strategies with only two proxy models? It is mentioned in line 812 that the dataset  ``adhere to H-Group Conventions’’. Does that constrain the strategy coverage of the dataset?\n2. Compared to neural policies, humans are usually much more adaptive. Does that apply to the human dataset in Hanabi? If so, is adaptation captured in the human proxy model?\n3. The numbers in table 4 are close, but (since it is not ``normalized’’) it is hard for me to directly see why it shows the behaviors are close.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d0j7iBWX6H", "reviewer": "Reviewer_QVEs", "summary": "The paper introduces the AH2AC2 to evaluate human-AI teamwork in Hanabi, a cooperative card game. Key contributions include:\n- AH2AC2 Framework: A standardized benchmark using human proxy agents (trained via behavioral cloning + RL) as evaluation partners, hosted via a controlled API.\n- Open-Source Dataset\n- Human Proxy Agents: Combining BC with KL-regularized RL, achieving robust performance while retaining human-like behavior.\n- Baselines: Evaluation of zero-shot (e.g., OBL) and data-driven methods (e.g., BR-BC), revealing gaps in integrating limited human data.\n\nMain results show OBL outperforms human-data-dependent methods, highlighting challenges in leveraging small datasets. Proxies exhibit human-like behavior via metrics (e.g., IPP, Communicativeness) but lack direct human validation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for the detailed and constructive feedback. We appreciate the opportunity to address your points and clarify aspects of our work. Please find our answers and improvements below.\n\n## Addressing Reviewer Questions\n\n### Q1. Human Validation\n\nWe agree that testing human proxies with real humans would be ideal for a human-AI benchmark, but large-scale evaluations are impractical due to logistical challenges and resource constraints. Therefore, we created these proxies using a method studied extensively in prior research, and our experiments and tests confirm they exhibit **human-like qualities** and behave as expected. Overall, our validation shows that the proxies capture key aspects of human play, making them suitable and robust partners for the benchmark.\n\n### Q2. Three-Player Data\n\nWe thank the reviewer for this excellent question! We collected data for two-, three-, four-, and five-player games; our analysis showed that proxies for four- and five-player settings gave less convincing results due to data sparsity. In contrast, the **three-player setting, with 46k games**, showed strong performance (comparable to the two-player setting), and tests on data subsets revealed diminishing returns with more data. This evidence suggests that the 46k games available for the three-player setting are sufficient to train robust, human-representative proxies, making it a valuable and worthwhile addition to the benchmark.\n\n### Q3. LLM Integration\n\nThank you for your suggestion. To strengthen our benchmark and given the popularity of LLMs, we have implemented an LLM-based agent using **o3-mini** and are currently benchmarking it with **AH2AC2**. So far, we're getting scores ranging from **14 to 20 out of 25** (along with some games where all lives are lost), and we'll include these results in the final paper copy, along with additional experiments and results. We believe this is a strong addition to our paper and hope the reviewer will recognise the effort and resources we invested into benchmarking LLMs.\n\n### Q4. Benchmark Maintenance\n\nWe are committed to the long-term success of the AH2AC2 benchmark. We will maintain and update the evaluation website, submission API, and leaderboard, offer ongoing support, and expand the benchmark, potentially adding four and five-player settings and other Hanabi variants as we collect more data and different techniques emerge.\n\n## Addressing Other Comments\n\n**Fixed proxies and dynamic behavior:**\nThank you for raising this point. Although our agents’ parameters are fixed after training, their in-game behavior is dynamic – _fixed parameters do not imply fixed behaviour_. Due to space constraints, please see our response to reviewers CEGr and FDiM. Action prediction results (Tables 3 & 4) confirm high accuracy/low loss on unseen human data, indicating they capture diverse strategies and adapt in-game.\n\n**Challenge utility:**\nWhile we agree that human-AI play is ideal and we acknowledge this limitation, large-scale human testing is often impractical and hard to reproduce. Our approach offers a practical alternative: we create robust, reproducible proxy agents trained on extensive real human gameplay data using _SOTA methods_ for developing human-like policies.\n\n**API-hosted proxies:**\nThank you for this comment. We believe AH2AC2 improves on previous methods that used completely inaccessible proxies and closed datasets. Our API provides _open_, but _controlled, pre-registered access_ to human proxies, ensuring fair, reproducible evaluation.\n\n**Behavioral metrics:**\nThank you for your suggestions regarding the breadth of our behavioural evaluation. Our experiments show that BC agents trained on the entire dataset achieve better coordination when paired with our human proxies (*Figure 2*), proving that our proxies adapt and recover from errors even with fixed BC agents. Low **IPP** means hints are effective (they carry implicit info that players must interpret), and our action prediction results (*Tables 3 & 4*) confirm the proxies learn a wide range of implicit conventions.\n\nWe couldn’t find standardised definitions for the extra metrics like *hint utility* or *convention compliance* in prior Hanabi works, so we'd be grateful if you could point us to any. We believe our current analysis provides substantial evidence of human-like behavior, and we're open to expanding our evaluation with suitable metrics.\n\n**Ablation study:**\nWe appreciate the time you invested in reading the appendix. We completely agree that it should be present in the main text. We will integrate key findings into the main text in the camera-ready version, using an extra page allowance.\n\n---\n\nWe hope these clarifications and planned revisions address your concerns and strengthen our paper. Thank you for your valuable feedback. if you feel our responses have sufficiently resolved your concerns, we kindly ask you to consider updating your score accordingly.", "claims_and_evidence": "Claims are generally supported, however there are some concerns:\n- Proxy human-likeness is validated via cross-play with BC, action prediction, and behavioral metrics. However, direct comparison with real human players is absent, weakening claims about human compatibility. Also, overtime, the human behavior may change while the proxy agents will be fixed.\n- Superiority of regularized RL over BC is evidenced by improved self-play scores (Table 2).\n- Challenge utility is demonstrated via baseline evaluations (Table 5), though reliance on synthetic proxies (not humans) limits real-world applicability claims.", "methods_and_evaluation_criteria": "- Hanabi is appropriate for testing coordination under partial observability.\n- KL-regularized RL is a sensible approach to balance human imitation and robustness.\n- API-hosted proxies prevent overfitting, but the limited open dataset (1k games) may restrict research scope, thus the same concern as closed existing evaluation methods would apply to the proposed benchmark.\n- Three-player evaluation is underexplained given data scarcity (46k games vs. 101k for two-player).", "theoretical_claims": "No new theoretical claims. KL regularization builds on prior work. Theoretical analysis of HDR-IPPO is deferred to future work.", "experimental_designs_or_analyses": "- Self/cross-play and action prediction tests are sound but rely on proxy-human comparisons which could be limited in terms of generalization similar to human players. It would be worthwhile to evaluate with human players as well and compare the results.\n- Behavioral metrics (IPP, Communicativeness) are validated against a human dataset (Table 4), but metrics are narrow. The paper validates human proxy agents using IPP and Communicativeness, which measure strategic aspects of gameplay (e.g., hint frequency and card knowledge). However, these metrics narrowly focus on specific behaviors, omitting critical dimensions of human-like coordination. For instance, they don't assess risk management (e.g., balancing safe plays versus strategic gambles), temporal adaptation (adjusting strategies as the game progresses), or error recovery (recovering from misplays through ad-hoc coordination)—key facets of human adaptability. Additionally, while prior Hanabi research incorporates broader metrics like hint utility (hint effectiveness) or convention compliance (adherence to implicit rules), the current analysis lacks these, limiting claims of comprehensive human-likeness. Furthermore, the reliance on quantitative metrics overlooks qualitative aspects such as trust or intent alignment, which could be captured through human evaluations (e.g., surveys or A/B testing). Expanding the evaluation to include these dimensions would strengthen assertions about behavioral fidelity and better reflect the complexity of human teamwork.\n- Ablation study (Appendix A.8) on regularization strength is critical but not detailed in the main text. I believe it's worth discussing this in more detail in the main paper.", "relation_to_broader_scientific_literature": "Connects well to Hanabi research (e.g., SPARTA), zero-shot coordination (OBL), and ad-hoc teamwork. Missing recent LLM-based coordination studies (e.g., theory of mind in LLMs), though mentioned in future work.", "essential_references_not_discussed": "No critical omissions noted.", "other_strengths_and_weaknesses": "- Strengths: Practical framework, reproducible evaluation, actionable insights (e.g., data-efficient methods needed).\n- Weaknesses: Limited human validation, sparse three-player data, incremental methodology (BC+RL), limited to only one environment.", "other_comments_or_suggestions": "In Section 2, discount factor is said to be in range $[0,1]$. How extreme different values affect the results, eg $\\gamma = 0$ vs $\\gamma =1$?\nMinor: in line 163, authors should use `citet`. Same should be applied in several places in the Appendix.", "questions_for_authors": "- Human Validation: How do proxies perform when paired with real humans? If untested, how confident are you in their human-likeness? And how can this be evaluated?\n- Three-Player Data: Why prioritize three-player evaluation given limited data? Does data scarcity bias proxy behavior?\n- LLM Integration: The conclusion mentions LLMs. What are the plans to benchmark LLM-based agents in AH2AC2?\n- Since this is a benchmark paper, it's of utmost importance to keep it updated and maintaining the projects. What are the promises and plans by the authors to do so?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a human-like agent to test human-AI coordination, based in the cooperative game Hanabi. The proposed agent is synthetic rather than based directly on human data. The authors provide analyses to validate it is human like. This is a dataset contribution, which will be accessible to the community. 

The reviewers raise several questions, but ultimately, it is clear that the contribution, while possessing some limitations, has the potential to be a significant contribution to the community and the work to construct and test the agent performed is substantial.
Decision: Accept (spotlight poster)