Learning to Cooperate with Humans using Generative Agents
Authors: Yancheng Liang
Daphne Chen
Abhishek Gupta
Simon Shaolei Du
Natasha Jaques
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-agent reinforcement learning
human-AI cooperation
Primary area: reinforcement_learning
TL;DR: We use generative model to sample partner agents to train a coordinator agent. These agents cooperate well with real human players.

Abstract: Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL). Current algorithms focus on training simulated human partner policies which are then used to train a Cooperator agent. The simulated human is produced either through behavior cloning over a dataset of human cooperation behavior, or by using MARL to create a population of simulated agents. However, these approaches often struggle to produce a Cooperator that can coordinate well with real humans, since the simulated humans fail to cover the diverse strategies and styles employed by people in the real world.  We show \emph{learning a generative model of human partners} can effectively address this issue. Our model learns a latent variable representation of the human that can be regarded as encoding the human's unique strategy, intention, experience, or style. This generative model can be flexibly trained from any (human or neural policy) agent interaction data. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents. We evaluate our method---Generative Agent Modeling for Multi-agent Adaptation (GAMMA)---on Overcooked, a challenging cooperative cooking game that has become a standard benchmark for zero-shot coordination. We conduct an evaluation with real human teammates, and the results show that GAMMA consistently improves performance, whether the generative model is trained on simulated populations or human datasets. Further, we propose a method for posterior sampling from the generative model that is biased towards the human data, enabling us to efficiently improve performance with only a small amount of expensive human interaction data.

Learning to Cooperate with Humans using Generative Agents
Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S. Du", Natasha Jaques* University of Washington (yancheng, daphc, abhgupta, ssdu, nj]@cs. wwshingtohnnnnnn edu
Abstract
Training agents that can coordinate zero-shot with humans key mission multi-agent reinforcement learning (MARL). Current algorithms focus training simulated human partner policies which are then used to train a Cooperator agent. The simulated human is produced either through behavior cloning over a dataset of human cooperation behavior, or by using MARL to create a population of simulated agents. However, these approaches often struggle to produce Cooperator that can coordinate well with real humans, since the simulated humans fail to cover the diverse strategies and styles employed by people in the real world. We show learning generative model of human partners can effectively address this issue. Our model learns latent variable representation the human that can be regarded as encoding the human's 's unique strategy, intention, experience, or style. This generative model can be flexibly trained from any (human or neural policy) agent interaction data. By sampling from the latent space, we can use the generative model produce different partners train Cooperator agents. We evaluate our method Generative Agent Modeling for Multi-agent Adaptation (GAMMA) on Overcooked, a challenging cooperative cooking game that has become standard benchmark for zero-shot coordination. We conduct an evaluation with real human teammates, and the results show that GAMMA consistently improves performance, whether the generative model is trained simulated populations or human datasets. Further, we propose method for posterior sampling from the generative model that is biased towards the human data, enabling to efficiently improve performance with only a small amount of expensive human interaction data.
Introduction
Producing agents that can cooperate well with unseen partners such as humans (28) is an important problem for variety of multi-agent systems across domains like robotics software agents. While being cooperative is notable characteristic of human intelligence [2], training an artificial agent that cooperates well with humans poses significant challenge. Human behaviors uncertain and diverse, encompassing a wide range of preferences, abilities, and intentions. While humans can rapidly adapt different partners, Al agents particularly poor at generalizing to working with novel human partner. Solving this issue of distribution shift between the human player's strategy and those seen during the training of Al agent is crucial achieving human-Al cooperation.
is tempting to tackle this problem of learning to cooperate with humans using only data of human- human interactions M. While this may certainly provide some leverage, in many situations the amount of human- humann data available is far less than the amount of data required by data-hungry sequential decision making algorithms. This suggests that we need way generate synthetic data at
See our website for human-Al study videos and an interactive demo. The training code is also available.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

scale, while still providing relevance to the problem of coordinating with new, real human teammates on deployment. On the opposite end is the paradigm of self-play, that generates entirely synthetic data by iteratively training agents against copies of themselves. Self-play, while initially envisioned as a paradigm for learning how to compete with novel human players [27] 33,, has been applied to cooperative multi-agent reinforcement learning to find joint strategies for a team of agents when all agents are trained together [10] 23]. Self-play is an appealing approach for learning cooperative strategies, since it can be done in simulation, and does not require the expensive and time-consuming process of collecting and training on human cooperation data M. However, agents trained with self may fail to coordinate effectively with novel humans [I 29]. During self-play, the agent learns to form a convention with a copy of itself and relies on that convention to achieve seamless cooperation [6]. When human adopts different strategy at test time, the agent fails to adapt, and merely continues to follow the convention formed in training. Such scenarios are fairly common, as self-play often generates "non-human" conventions [1. This points to the dual problems of 1) human-only data being expensive, and 2) synthetic-only data lacking coverage over human behaviors.
To tackle these challenges, a popular approach to solving the distribution shift problem in human-AI cooperation has become training a Cooperator agent against a population of simulated agents rather than just itself 15 18
37 40). The goal of this work is often to create a diverse enough population of agents to cover the strategy space used by humans [2 24 37]. However, the bottom line remains that truly covering the space of strategies with discrete samples of strategies can quickly become untenable. As we move towards more complex real-world tasks, with a myriad of possible strategies, representing every strategy by individual agent in the population and coordinating with them becomes computationally difficult. Indeed, we empirically compare the two lines of work on using simulated versus human data for zero-shot coordination, and find that contrary past results, using human data provides better performance, especially for more complex environments.
19
24 29
The key insight we will make in this work is that generative models offer solution to the dual problems mentioned above. By training a generative model that can simulate cooperation partners, we can easily incorporate both human and simulated data. The resulting model can generate a diverse range of partner strategies, by interpolating between the policies it has been trained on, as well as composing strategies to create novel partners. Therefore, propose to train a generative model of partner behavior using a variational autoencoder (VAE) [12] on a collection of coordination trajectories, either human or synthetically generated. The inference model of the VAE can be used to infer the latent variable : of a partner from its interaction data, encoding information about that partner's unique style or skill levels. The decoder can be used to generate the corresponding actions taken by the partner. Because the generative model can be trained on any combination of synthetic or human data, it can overcome the challenges of data scale and coverage train adaptive Cooperators that much better cover the space of human behavior (see Figure
Given the ability to generate this diverse distribution of partner policies, a single adaptive Cooperator can be trained to adapt to range of partners, by sampling a novel partner strategy from the generative model at each episode during training. We call our full method GAMMA: Generative Agent Modeling for Multi-agent Adaptation. While the interpolation properties of the generative model enable GAMMA to train more robust Cooperator agents, the availability of controllable latent encoder also allows our desired adaptation behavior to be targeted to real human behavior. We propose novel, ecnonomical way of incorporating a small amount of human-provided data into the sampling procedure of the generative model. This Human-Adaptive sampling method enables training Cooperators that are more targeted to partner with real human coordinators.
We test our method using Overcooked M. a collaborative multiplayer cooking game requiring close coordination between two partners to successfully prepare recipes. In evaluation in which trained agents play against novel humans in real-time, we find that GAMMA improves the performance of -ftte-a coordination methods which rely simulated populations of agents 24 29 or on human data We summarize our contributions below:
We present GAMMA, a novel approach for learning generative model of partner strategies which can be trained on data from humans or a simulated population of agents. We sample from the generative model to simulate novel partners that are used to train robust Cooperator agent. See Section 4. I. and Section 4.2
We propose data-efficient and auman-adeptive sampling technique to steer the model to generate more human-like partners, even with limited amounts of human data. See Section 4.3
2

---

Generallon Gererciti (4kman
Datavior clining Generazie age
Euamn players
Simuuaated aQUITS
Simrotoo agarts
(a) Simulated data.
(b) Human data.
(c) Adaptive sampling.
We show the latent space covered by different methods For either simulated data or human data, the generative agents produced by GAMMA can cover a larger strategy space. Generative models can provide novel agents by interpolating the agents in the simulated population (a). On human data (b), the human proxy model only covers subset of all human player behavior patterns, while the generative model can capture the diversity in the data. We can also control the latent space sampling (c) to model target population of agents (e.g.. human coordinators).
Figure 1:
We conduct an empirical evaluation via a user study with real human players. We bring together and directly compare two lines of prior work on zero-shot coordination with humans: training on simulated population data and training on human data. We find that contrary to past published results, training on human data is highly effective in our human evaluation, especially for the new, more complex layout we propose. However, regardless of the data source, GAMMA consistently provides significant performance improvements over multiple lines of past work when evaluated with real humans. See Section 5
2 Related Work
Training against simulated populations. Building cooperative agents that generalize well to humans and novel partners is a long-standing problem of AI, and known as ad-hoe team-play (28). or
zero-shot coordination [1I. Recently, FCP (29) adopted the idea of using a population of policies to train a strong zero-shot Cooperator agent, as a diverse population effectively prevents the Cooperator from exploiting one specific convention. Following this framework, many techniques 37 have been proposed to improve the diversity of the population, such as Maximum Entropy Population MMP) Reward shaping 30 37 and quality diversity (22] BJ 351 utilize domain knowledge to create agents with diverse behaviors. Statistical diversity based on trajectory distributions [19 population entropy [40] straightforward optimization objectives. However, indicated in 2]. agents with different behavior distributions do not necessarily use different high-level strategies. To address this, the LIPO algorithm (2) instead tries minimize the cross-play reward between different agents in the population to encourage them to form incompatible conventions. However,
propose strategies for preventing this, including Cross-play Optimized, Mixed-play Enforced Diversity (CoMeDi) 24]. While this extensive literature presents different population creation methods, in contrast we propose novel approach to modeling this population with a generative model, and thus benefit the training process through the ability flexibly sample unlimited new partners. Our experiments directly compare to FCP 29 CoMeDi (24), and MEP 40). and show that GAMMA can enhance performance above each of these iopulationngeneraane
because LIPO agents can potentially sabotage the game, follow-up works 3 24
techniques, even using the same underlying population of simulated agents, while also allowing for the incorporation of human data.
Training with human data. Another line of work has focused collecting and training on human cooperation data 1] Learning from human data is inherently challenging due to limited scale, arbitrary human behavior, subjective preferences, and relative speed of human actions compared to agents, Another challenge of coordination comes from the heterogeneity, uncertainty and suboptimal- ity [8] 21) of humans. The agent might fail to coordinate well when does not properly infer the preferences or intentions of humans. Our work takes novel approach of combining these two lines
PPoints are the latent vectors. Taking the population of simulated agents (c.g., the FCP population with FCP agents) as an example, a point is generated by: 1) sampling an agent in the FCP population; 2) using the agent generate an episode with self elfy: 3) using the VAE encoder to encode the episode into a latent vector and mapping the latent vector to the 2D space using t-S.E.
3

---

Learn generative mocel
Train Cooperator
simulattd
agents
decoler
Gennnlive ogonts
Cooperator
generolive model
Figure 2: Overview of the method for GAMMA. The generative model learns latent distribution over partner strategies from either simulated or human data. Sampling partners from the generative model enables training robust Cooperator that can coordinate with variety of different humans.
of work by training a generative model of partner strategies on simulated data, then using a limited amount of human data fine-tune the model and sample from it more effectively.
Inferring the partner type. Our work is also related to the many works on multi-agent learning which attempt to learn latent partner embedding to aid coordination. Some works use Bayesian inference [26] 36] or weight update (39] to decide the partner type from historical experience. Their methods require
predefined set of all possible partner policies. The idea of embedding-condee conditioned agent modeling is used in imitation learning [14] to learn more interpretable policies from human demonstrations. Grover et al. [5] train latent variable model to learn agent representations from interaction data of different agents, and then infer a cooperation partner's s latent vector and use it to condition multi-agent RL policy. Papoudakis al. [20] learn to predict the partner's s representation from only the agent's own observation, avoiding accessing the global states during execution time. The idea of partner modeling has also been extended to offline RL [9] to train a zero-shot RL agent from a dataset. It is also known that generative models can learn diverse cooperative strategies from human-human cooperation demonstrations (34 While our work also infers hidden context about partner type, our aims are different; we use a generative model to simulate novel partners during training time way to train Cooperator be robust for zero-shot coordination with real humans.
Problem Formulation
A multi-agent system typically modeled as Markov game 17 In this work, we focus on two-player Markov games, but the formulation and our methods can be easily extended to more agents. The state space S. At each step, two agents execute their actions a, A, E B from their policies π : S + ((A), μ S + ABB). They then receive shared reward S x A x B - R. The next state determined by a transition function T S x A -> A(S). The value function ((T,A) μ) of two policies is defined as the expectation of discounted cumulative reward E 7'ri. Under this formulation of Markov games, the learning objective for the fully cooperative joint policy max=,a V(T, 11) can be perfectly defined. However, it is unclear how to describe the learning objective of human-Al cooperation under the framework of Markov games.
Inspired by the latent Markov Decision Process [3], we model the problem of cooperating with humans MDP with latent variable a. We assume the human policy is conditioned latent variable 2 E 2. This latent variable can be regarded as a representation of the partner's unique style, skill level, reaction speed, etc. Then the human population
be defined distribution A(B) conditioning on the latent Eees/e (b s)T(s' s, a, b) and the reward function (s a) EoEB Ms s)r(s, a, b) can both be defined. Now with the value function for MDP M, [T., 1 is Vi(IT) = V(T,, Mz), the learning objective can be defined as T E arg max, E2-D(2) [T)).
D(Z) over the latent space. Given the human policy ;: S
variable 2, the conditional transition function Tg(s' s,
4 GAMMA: Generative Agent Modeling for Multi-agent Adaptation
We will leverage generative models as our key tool for strategy diversification, to overcome the dual challenges of 1) lack of real human data and 2) the difficulty of synthetically covering the large strategy space of human partners. In being able to generate diverse strategy profiles beyond the data,

---

these generative models can be used to train Cooperator that can be deployed to coordinate with novel users, each with their own diverse styles, preferences and capabilities. Here, we first describe our procedure for learning a generative model from training data of discrete agents (or human-human data), and then describe how this can be used for targeted coordination with real human partners. We call our approach Generative Agent Modeling for Multi-agent Adaptation (GAMMA).
4.1 Learning Generative Models of Partner Behavior
The first step of GAMMA is learning the generative model from pre-existing coordination data. We assume access to a dataset of trajectories 'D Decordination = {n. [- where each trajectory 7 = {(tt, Gt. be) )e is a sequence of multi-agent coordination behaviors. This dataset can be derived from human playing records. Alternatively, if there is a population of simulated agents available, this dataset can be collected by pairing them together to generate joint trajectories. For instance, in our experimental evaluation, we use agents generated through techniques like fictitious co-play (FCP) [29] to generate this dataset. The purpose of generative modeling here is to be able to model the multimodal marginal distribution over various strategy profiles in zovedimationn challenging distribution to model with standard maximum likelihood methods. Notably, the generative model is able to sample a landscape of agents that can be used to train Cooperator, going beyond the quantity and diversity of the training data.
We develop a variant of the Variational Autoencoder (VAE) 12 to model the diverse strategies underlying the training data D. As is typical in variational inference frameworks, we propose to learn an encoder-decoder generative model with approximate posterior (encoder) g(= oi that identifies the agent style from the trajectory. The decoder in this model, p(a, ž, To..t.1000) uses the agent's is own past experience T (so, Ci...., .... 8t- 1, at-11 1, 8t) and the latent variable = to predict the agent's 's next action. This structure of the variational inference model can be thought of as multi-modal behavior cloning, where the encoder history provides the latent variable required to model the distribution of generated actions. The encoder-decoder architecture described above, and shown in Figure can can be trained using evidence lower bound (ELBO) loss:
T
log p(a; 1o...-1:))) + SKL(g(> TT, [T) n)
(( = ET-D (1.3)
Σ
(1)
Importantly, this generative model is able to generate behaviors that go far beyond the training data, both in quantity and diversity, as is shown in [i We show in [T while simulated behavior may not cover the space by itself, the generative model has significantly greater coverage over the strategy space. Importantly, we hypothesize that human behavior is more likely to lie within the span of strategies generated by the generative model. In this way, the interpolation and generalization of the generative model (even if it is not perfect) provides the expansion of the data required to effectively coordinate with humans.
4.2 Training a Cooperator with Generative Coordination Models
Once generative model has been obtained, it can be used train robust Cooperator agent. This is accomplished by treating the generative model as a partner generator, using it to simulate partner behavior that covers the space of real human behaviors, and training the Cooperator to optimize performance with these partners in simulation. In particular, the generative agent model 1; 0) can now be used
p(at 2, To..t-1;
generator of partner policies μl to train our Cooperator agent. As shown in Figure 2 for every episode we can sample latent variables from the prior a ~ p(2). and use the conditioned action distribution p(a, 2, To..-11 8) as a partner μıg for that episode. We aim to leverage these sampled partners from the generative model to train a single Cooperator πC. treating the sampled partners μı as part of the environment. At each iteration, batch of agents (Mz, J are generated using latent variable samples ~ D2. Then a batch of MDPs (M,, J.M can be derived from these training partner policies learn our Cooperator policy л. Then we use PPO
22 to optimize मं over these training MDPs:
(TT) = Ez-D, [V=(T)] = Ee-p(a) [V]T,
Importantly, the Cooperator मe is not trained with imitation learning, but rather trained with reinforcement learning to learn task-specific coordination behavior. The generative model from
5

---

Section 4.1 is simply used to provide the quantity and diversity of agents necessary for meaningful
generalization to real human behavior.
4.3 Targeted GAMMA using Human-Adaptive Sampling and Fine-tuning
While the Cooperator described in Section 4.2 is trained by sampling partner agents from the generative model as ~ p(z), ae < p(a, 3, 1o 0), this does not make use of human specific data if available, instead simply treating any human and synthetic data as equivalent. However, when coordinating with real human partners, it is useful target model adaptation be human-specific rather than to span the entire space of potentially irrelevant synthetic strategies (as shown in Figure l}. The key insight we will leverage to do so that the latent space afforded by our generative model provides the ability to do controllable sampling from any latent distribution. This distribution can be chosen to incorporate additional information about the desired population. In particular, when coordinating with real human partners, the latent prior distribution for sampling p(=) can be replaced with a human-centrie one pı. (2), which is more focused on the part of the latent space relevant to latent distribution pa (z) can be quickly inferred by encoding the human data and estimating the latent Gaussian distribution that best explains encoded human data. The latent Gaussian mean is given by:
human behavior. This suggests that given a small amount of human data Dr, a
= ED, [Es
Given this human-centered latent distribution pn (=) (2,, f), a targeted Cooperator that is meant for particular target population can be trained by maximizing J(TC) = Ez~pa (2) (T,
Human Human-adaptive adaptive sampling makes the model focus less adaptation to irrelevant synthetic partners and more on "human-centric" partners sampled from the generative model. As our experiments will show, this approach outperforms training partner simulator using only human data, since with limited human data it is easy for the model to go out of distribution during generation and it thus be brittle. In contrast, GAMMA is able to train robust partner simulator using large amounts of synthetic data. Adding human-adaptive sampling to GAMMA can provide considerable generalization benefits even with modest amounts of data. Note that better capture human data, we do not just target the latent space, but also perform some fine-tuning on the encoder and decoder of the VAE using human data.
5 Experiments
We evaluate GAMMA using the Overcooked environment [ as a popular benchmark for prior work on human-AL AI cooperation [ 24 29 36 37]. In Overcooked, two players need cooperate to divide the work of cooking and avoid getting in each other's ' way. This involves anticipating the intended goal and actions of the other player, and inferring which task would most usefully assist them. The layouts proposed in previous work are shown as the first five environments in Figure 3 The Counter Circuit layout poses a hard coordination challenge with multiple strategies such as around
passing items across the countertop the partner moving Nockkwsse/cuutterclockk the ring. Additionally, we include a custom layout based on Counter Circuit, termed Multi-strategy Counter. This new layout involves recipes with multiple ingredients. significantly increasing the complexity of the strategy space. Failing to infer the intentions of the partners and adding the wrong ingredients to the pot could ruin the entire dish, which increases the importance of coordination.
Our experiments investigate the following questions:
H1: Using simulated agents. Will training a Cooperator against generative model of partner strategies trained on a population of simulated agents outperform training the Cooperator against the simulated agents directly?
H2: Using real human data. Can the generative model be used to effectively leverage (small amounts of) human data, and also combine it with simulated agents?
H3: State-of-the-art. Can we obtain better performance than competitive baselines using both simulated and human data?
Learning from a Simulated Agent Population. We first investigate H1, and test whether generative agents can improve the performance of the Cooperator agent when training with simulated agent partners. We include three state-of-thee the approaches for creating a population of simulated agents.

---

Figure 3: The first five layouts Cramped Room, Asymmetric Advantages, Coordination Ring, Forced Coordination, Counter Circuit are originally proposed in Carroll et al. [[]. We create an additional Multi- Counter layout. In this new layout, humans additionally choose between making onion vs. tomato soup, which makes coordination significantly more challenging.
Fictitious Co-Play (FCP) uses multiple self-play agents initialized across different random seeds with checkpoints sampled throughout training. which simulates a diverse population with varied skill levels. Maximum Entropy Population-basee Training (MEP) [40] enhances FCP by further diversifying the population through a population entropy learning objective. Cross-plas Optimized, Mixed-play Enforced Diversity (CoMeDi [24] generates a diverse set of coordination policies by minimizing the cross-plas play reward and prevents self-sabotage using mixed-play regularization. For each method, we use the simulated agent population to create a dataset to learn the generative model.
Learning from Human Data. To investigate H2, we test whether the generative model can be used to model human behaviors from data produced by real human players. In this work, we assume the human dataset contains 20 50 trajectories, reflecting the popular open-source dataset provided by Carroll et al. M. We use PPO-BC [ as the baseline where a BC model over human data is used as the partner during training. We test whether replacing the BC agent in this framework with our generative model trained on the human data (PPO-BC-GAMMA) provides better performance. Since we assume the amount of human data limited, we also fine-tune a generative model pretrained from the simulated agent population with human data with both decoder-only (DFT) and full fine-tuning (DFT), as described in Section 4.3 We apply our Human-Adaptive (HA) sampling method the fine-tuned generative model to sample from the human-centered latent distribution ps. (2).
Simulated Agent Evaluation. Following prior work (. we create automatic evaluation mecha- nism by using held-out human data to train a behavior cloning policy as a human proxy agent, and report the performance the Cooperator when it paired with this test human proxy agent. However, we note that methods which are explicitly trained against human- proxy agent (PPO-BC) can easily exploit this metric in a way that is not indicative of actual performance with real humans, so we do not use this automatic evaluation to assess those methods, Instead, we conduct evaluations with real human players as the gold standard evaluation technique.
Human Evaluation. We run a user study with real human players in order to determine which method can most effectively coordinate with humans, and which method is rated as subjectively better by humans (H3). We conducted a study with 80 users recruited via online crowdsoureing from Prolific. Our study follows guidelines set by UW IRB protocol. During the study, each user is instructed to play multiple rounds of Overcooked with a partner via a web interface, where in each round the partner an agent following one of the policies, in randomized order. We trained 5 random seeds per agent, and used different candomly-selectedd seed for each of the 9PP game rounds. For the human study, we focus on the most complex layouts, Counter Circuit and Multi-strategs Counter. Each game lasts for 60 seconds. After each round, the user answers Likert scale survey questions [6] to rate their experience playing with the agent. At the end of the rounds, humans also answer qualitative questions about the performance of the agents. Users are required to pass attention check to ensure quality of their data. Using these answers, we conduct a qualitative analysis to understand which factors most heavily influence overall performance and users' preferences when playing with real humans.
Results
6.1
Evaluation in Simulation
In this section, we present the evaluation results against a human proxy (behavior cloning) agent.
H1: Using Simulated Data. We train the generative models FCP+GAMMA, CoMeDi+GAMMA and MEP+GAMMA on the simulated agent population of FCP, CoMeDi and MEP, respectively. Figure 4 shows the learning curves of each method averaged on all layouts, and the final performance
7

---

FCP
USF
PCP
FOR
Cille's
Use
GAMPA
Caleri- GANMA
MGP
GAMMA
FOP
GAMMA
Average Performance Acrose All Layouts
Performance Across Layouts
Performance Across Layouls
Normakzed Reenard
Reward
8:00
22
de devitaaloo diny
Conelaate
Maa-magg cacter
Coort
Derto
Adungigh
Hiw Makmaagg cacter
Clock
Step
Layouns
Layous
(a) Learning curves of different (h) Normalized reward across differ-(e) Original reward across different methods.
ent layouts.
layouts.
Figure 4: Evaluation of different methods using human proxy model. Rewards are normalized by the highest reward achieved on each layout. The learning curves in (a) show the average normalized reward across all environments, indicating that GAMMA helps the Cooperator converge to a higher reward. This improvement is also consistent across individual layouts, as illustrated in (b) and (c). We observe the largest performance gap on the 'Counter Circuit' and 'Multi Surtt--Srrateyy Counter' layouts, which are the most complex in terms of the number of valid cooperation strategies.
of different methods for each of the different layouts. The Cooperator agent evaluated by the zero-shot cooperation performance with human proxy BC model. GAMMA consistently improves performance over the baselines, across layouts. This demonstrates that GAMMA provides a more ef- ficient way utilize the simulated agents by providing a landscape of partners to train the Cooperator. We also find the improvement gap increases as the layouts become more complex.
6.2 Evaluation with Real, Novel Human Partners
As described in Section we conduct an evaluation with real human players, recruiting new participants that were not in the training data. We evaluate all agents against the novel human players, and plot the cooperative scores achieved by each agent-human team in Table
6.2.1 H1: Training with Simulated Data.
We find that GAMMA offers significant improvements over prior techniques for training against simulated populations (FCP, CoMeDi, MEP). Although the trend of these results is consistent with the previous results for HI, we find that here GAMMA provides significantly enhanced performance improvements when tested with real humans, reaching the new state-of-the- performance for both Counter Circuit and Multi- Suutii-Srrateg Counter.
On our newly proposed, more complex layout, Multi-Strateen Counter, we find that the CoMeDi baseline performs so poorly that it cannot discover strategies which make use of the new tomato ingredient. The reason is because playing the onion-soup strategy and the tomato-soup strategy together can recreate an unrecoverable game state, which is not favored by the CoMeDi algorithm to include both types of agents in the population. Therefore, when we use GAMMA train generative partner model using data generated from the CoMeDi population, also fails to learn any strategies involving tomatoes, and both models generalize poorly to playing with humans, although CoMeDi+GAMMA. still offers a performance benefit over CoMeDi. This points to the fact that GAMMA. which is based training generative model on cooperation data, can fail to perform well if the cooperation data is not sufficiently diverse. This problem can be aptly described by the well-known adage "garbage in, garbage out".
However, when GAMMA is trained using the high-quality population found by MEP, we see that performs extremely well, reaching a score of 88 on Multi-Strategy COunter, while MEP, the most competitive simulated population baseline, only achieved 64. This represents a 38% improvement.
6.2.2 H2: Training with Human Data.
Comparing methods that make use of human data reveals some interesting findings. First, our results directly compare two lines of prior research: training on simulated populations vs. training against a BC model trained with limited amount of human data (PPO-BC). While previous works show that using only simulated data can exceed PPO-BC and reach state-of-the-ar performance (2)] we find that the more complex layout, PPO-BC is actually the best performing baseline. Modeling

---

(a) Counter circuit layout
(b) Multi stratiy Counter
Figure 5: Performance of different agents when played with real humans. Error bars of the Mean (SE) for statistical significance (p < 0.05). Methods trained on human data are shown green. Whether training with simulated or human data, GAMMA shows consistent, statistically significant advantages over the baselines. HAMMA-HA is able to efficiently use the real human dataset to learn a better sampling of its latent space, achieving the best performance when cooperating with real humans.
the Standard Error
the human data with the generative model (PPO+BC+GAMMA only provides performance im- provements half the time. However, combining simulated and human data with Human Adaptive GAMMA provides significantly higher performance in both layouts, surpassing state-of- ofat-oof-the-art zero-shot coordination techniques.
6.3 H3: Can we obtain better performance than competitive baselines?
As revealed in Figure 5 GAMMA HA achieves the highest performance in both layouts, surpassing the most competitive baselines by 60% and 43% in Counter Circuit and Multi-Strategy Counter, respectively (See TjbleB).. We conducted a statistical analysis of these results using Holm-Bonferroni correction, which can be found in Table 4
6.3.1 Subjective Human Ratings
As demonstrated in prior work [I], humans can adapt to deficiencies in the policies of Al agents and narrow the apparent performance gap between different agents by simply completing the task themselves. This means cooperation performance alone cannot measure whether a particular agent is frustrating or cumbersome for the human to cooperate with. Therefore, we also collect subjective ratings of the agents from the human participants. The results for the question about overall coopera- tion are plotted in Figure 6 additional results are available in the Appendix, which pertain agent's ability to adapt (Fig. 9.. and whether it was human-like like (Fig. 10 or frustrating (Fig. 1TI.
For the overall ratings in Fig subjective ratings mirror the cooperative performance scores: when GAMMA is trained with the same data available to a baseline technique it improves perfor- mance in terms of the human ratings, and GAMMA HA gives consistently good performance on both layouts, in the sense that it receives a higher proportion of positive human ratings. We note that while PPO+BC+GAME GAMMA did not show performance benefit over PPO+BC on Multi-strategy Counter, human ratings of PPO+BC+GAMMA trends is that CoMeDi obtains high subjective ratings in Counter Circuit, although performs poorly in strattyy Counter. From the additional figures in the appendix, we can observe that GAMMA methods are rated more adaptive, human-lin like, and less frustrating than other techniques.
were more positive. One exception to the previous
6.3.2 Qualitative Findings
Analyzing the qualitative results reported by participants in the human study, we find that adaptation to the human partner was core theme distinguishing agents that humans liked. Participants reported that the FCP + GAMMA agent demonstrated an ability to learn from the user's 's actions, such as mimicking the user's strategy of placing onions on the table to save time. This adaptive behavior was positively received by study participant: "I noticed that once started to put back onions on the table that did the same as wanted to save time rather than going back for onions times for soup. thought it was interesting that it learned about my behavior." Because GAMMA models have been trained over more diverse range of partners, they consequently exhibit the ability to better adapt to real humans during gameplay.

---

heiig for Ajarel Aa to Coordinate in courter dreur Logor
Baing for Agerasl Acklt Ceordrate in deens counter drout Loyout
Ziit
3P8
2PA
Sth
205
CCANAA
27%
HEP CANNA
35
FCP CANMA
251
2N
FCP GANMA
175
5h
24
116
18
26
56
18
10 Ir
1P
118
26
25
216
256
24
16h
GANNA
26
100
346
136
28
(a) Counter circuit layout
(b) Multi-strategy Counter
Figure 6: Human ratings for different agents. Individuals were asked to respond to the following question: "Overall, I felt that the agent's ability to coordinate with me was: Veery poor, Poor, Neutral, Good, Very good]". FCP GAMMA, PPO BC + GAMMA, and GAMMA- consistently receive higher ratings for ability to coordinate compared to their respective baselines.
Another common theme that emerged from the qualitative analysis was consistency and predictabil- ity. Erratic behavior was common observation regarding the baseline methods. Users observed the Al performing random actions, such as moving onions around without an evident purpose or failing to complete necessary steps in the cooking process. In contrast, users report that agents using GAMMA behave in a logical, consistent manner. For example, for FCP GAMMA, participants provided the following feedback that the agent was "more deliberate in its actions" " and "its actions were logical". In contrast users reported that the baseline FCP agent, "didn't behave logically" and "the agent this time was inconsistent and did not help me with any of my orders at all." We hypothesize the reason for this "Inconsistency" that occurred in baseline methods may be due to the human's 's behavior going nut-of-distributt (OOD) of their training data, causing the resulting policy to make errors and behave in an unpredictable way. This points the importance of obtaining better coverage of the human data distribution, as provided by GAMMA (see Figure I).
7 Conclusion
In this work, we propose GAMMA, novel approach to training coordinator agent by using generative models to produce training partner agents. We conduct a comprehensive analysis using data from study with real human cooperation partners, and show GAMMA outperforms baselines over both subjective human ratings and quantitative measurements of cooperation performance. We also provide new perspective to compare different populations under the latent space of a generative model, showing how the simulated populations may not provide sufficient coverage of the range of human players.
Limitations. As shown by the performance of GAMMA+CoMeDi on Multi-Strutegy Counter, obtaining good performance with our approach depends on having reasonably diverse amount of cooperation data to train the model. If the quality of the simulated population data is too low, the approach can fail to provide significant benefits.
In this work, our human studies recruit participants from Prolific, which may not be representative of broader populations. Additionally, our human dataset is limited, which could reduce the diversity of strategies and force participants to adapt to strategies that the Cooperators are already familiar with.
We focus on the two-player setting in this study following prior work 29 37 40
because it is a first step toward enabling an AI assistant that could help a human with a particular task. Scaling up to more agents would exponentially increase the dataset size with our current techniques. Therefore, better sampling techniques are needed address this issue.
Future work. Several potential directions are interesting for future work: 1) In this work, the amount of human data is limited, which restricts the performance of the generative model that learns human data from scratch. 2) An orthogonal direction is to condition the policy of the Cooperator on the embedding of the partner policy. We provide some preliminary results in Figure 15]
Social impact. Our work focuses on how to train Al agents that can effectively cooperate with diverse humans to assist them with tasks. We believe this is critical component of eventually enabling assistive robots that could operate human environments to assist the elderly or disabled to live more comfortably, reduce the burden of domestic labour for all people.
10

---

Acknowledgment
SSD acknowledges the support NSF IIS 2110170, NSF DMS 2134106, NSF CCF 2212261, NSF IIS 21 2143493, NSF CCF 201 9844, and NSF IIS 2229881.
References
[1] M. Carroll, R. Shah, M. K. Ho. T. L. Griffiths, S. A. Seshia, P. Abbeel, and A. D. Dragan. On the utility of learning about humans for human-ai coordination. CoRR, abs/1910.0578, 2019. hRtp::///arxii org/ wwt.a//.........0 05789
[2] R. Charakorn, P. Manoonpong, and N. Dilokthanakul. Generating diverse cooperative agents by learning incompatible policies. In The Eleventh International Conference Learning Representations, 2022.
[3] B. Cui, A. Lupu, S. Sokota, H. Hu, D. J. Wu, and J. N. Foerster. Adversarial diversity in hanabi. In The Eleventh International Conference on Learning Representations, 2022.
[4] G. Cumming, F. Fidler, and D. L. Vaux. Error bars in experimental biology. The Journal of cell biology, 177(1):7-11, 2007.
[5] A. Grover, M. Al-Shedivat, J. Gupta, Y. Burda, and H. Edwards. Learning policy representations in multiagent systems. In International conference machine learning, pages 1802-1811. PMLR, 2018.
[6] N. Grupen, N. Jaques, B. Kim, and S. Omidshafici. Concept-based understanding of emergent multi-agent behavior. Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
[7] J. Henrich. The secret of our success: How culture is driving human evolution, domesticating our species, and making us smarter. In The secret of our success. princeton University press, 2015.
[8] J. Hoch and G. Loewenstein. Time-inconsistent preferences and consumer self-control. Journal of consumer research, 17(4):492-5) 507, 1991.
[9] Hong, A. Dragan, and S. Levine. Learning influence human behavior with offline rein- forcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[10] H. Hu and J. N. Foerster. Simplified action decoder for deep multi-agent reinforcement learning. In International Conference Learning Representations, 2019.
[11] H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster. "othee--pay" for zero-shot coordination. In International Conference Machine Learning, pages 4399-4410. PMLR, 2020.
[12] D. Kingma and M. Welling. Auto-encoding variational bayes. In Bengio and Y. LeCun. editors, 2nd International Conference on Learning Representations, ICLR 20/4, Banff. AB, Canada, April 14- 16, 2014, Conference Track Proceedings, 2014. URL httt:://arxil org/
abs/1312. 1112..112..
[13] Kwon, Y. Efroni, C. Caramanis, and Mannor, RI for latent mdps: Regret guarantees and lower bound. Advances in Neural Information Processing Systems, 34:24523-24534, 2021.
[14] Y. Li, J. Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. Advances in neural information processing systems, 30, 2017.
[15] Y. Li. S. Zhang, J. Sun, Y. Du, Y. Wen, X. Wang. and W. Pan. Cooperative open-ended learning framework for zero-shot coordination. In International Conference on Machine Learning, pages 20470-20484. PMLR, 2023.
[16] R. Likert. technique for the measurement of attitudes. Archives of Psychology, 140:1-55. 1932.

---

[17] M. L. Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[18] X. Lou, J. Guo, J. Zhang, J. Wang, K. Huang, and Y. Du. Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 679-688, 2023.
[19] A. Lupu. B. Cui, H. Hu, and J. Foerster. Trajectory diversity for zero-shot coordination. In International conference on machine learning, pages 7204-7213, PMLR, 2021.
[20] G. Papoudakis, F. Christianos, and S. Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances Neural Information Processing Systems, 34: 19210-19222, 2021.
[21] W. Pratt. Risk aversion in the small and in the large. In Uncertainty in economics, pages 59-79. Elsevier, 1978.
[22] J. K. Pugh, L. B. Soros, and K. O. Stanley. Quality diversity: A frontier for evolutionary computation. Frontiers in Robotics and Al, 3:40, 2016.
[23] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234-724, 2020.
[24] B. Sarkar, Shih, and Sadigh. Diverse conventions for human-ai collaboration. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
[25] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: prrXiv11077 aeei::007707444 2017.
[26] M. Shum, M. Kleiman- -leiman-Weiner. M. L. Littman, and J. B. Tenenbaum. Theory of minds: Understanding behavior in groups through inverse planning. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 6163-6170, 2019.
[27] D. Silver, J. Schritwieser, Simonyan, I. Antonoglou, Huang, A. Guez, T. Hubert, Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature, 550 (7676):354-33) 2017.
[28] P. Stone, G. Kaminka, S. Kraus, and J. Rosenschein. Ad hoc autonomous agent teams: Col- laboration without pre-coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1504-1509, 2010.
[29] D. Strouse, K. McKee, M. Botvinick, F. Hughes, and R. Everett. Collaborating with humans without human data. Advances in Newral Information Processing Systems, 34:14502-14515, 2021.
[30] Z. Tang, C. Yu, B. Chen, H. Xu, X. Wang, F. Fang. S. S. Du, Y. Wang, and Wu. Discovering diverse multi-agent strategic behavior via reward randomization. In International Conference on Learning Representations, 2020.
[31] B. Tjanaka, M. C. Fontaine, J. Togelius, and S. Nikolaidis. Approximating gradients for differentiable quality diversity in reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1 100-11111, 2022.
[32] M. Tomasello. Why we cooperate. MIT press, 2009.
[33] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3500-354 2019.
[34] C. Wang, C. Pérez-D' Arpino, D. Xu, L. Fei-Fei, K. Liu, and S. Savarese. Co-gail: Learning diverse strategies for human-robot collaboration. Conference on Robot Learning, pages 1279-1290. PMLR, 2022.
12

---

[35] S. Wu. J. Yao, H. Fu, Y. Tian, C. Qian, Y. Yang, Q. Fu, and Y. Wei. Quality-similar diversity via population based reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[36] S. A. Wu, R. E. Wang. J. A. Evans, J. B. Tenenbaum, D. C. Parkes, and M. Kleiman-Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cognitive Science. 13(2):414 432, 2021.
[37] C. Yu, J. Gao, W. Liu, B. Xu, H. Tang, J. Yang, Y. Wang, and Y. Wu. Learning zero-shot coop- eration with humans, assuming humans are biased. In The Eleventh International Conference on Learning Representations, 2022.
[38] C. Yu, Velu, Vinitsky, Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 1 1-24624, 2022.
[39] M. Zhao, R. Simmons, and H. Admoni. Coordination with humans via strategy matching. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9116-9123. IEEE, 2022.
[40] R. Zhao, J. Song, Y. Yuan, H. Hu, Y. Gao, Wu, Z. Sun, and W. Yang. Maximum entropy population- poased training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6145-6153, 2023.
13

---

A
Reproducibility
Our demo website is http://siisssoggleee.e.. wwwwwwwwwwwwwaaan....... -aapuuuaaaaaaaaa---aaaaaaa and contains the code and more experiment results. We also provide information about the implementation details and hyperparameters used in our experiments D to help reproduce our results.
B Implementation details
Generative models. The dataset used to train the VAE model contains the joint trajectories of two players. For the simulated agent population ( wyJ. we create this dataset by evenly sampling T; x generate the trajectories. A simulated dataset contains 100K joint trajectories. To train VAE on it, the dataset is split into a training dataset with 70% data and a validation dataset with the truncated to length 100 for better
rest of 30% data. To compute the ELBO loss the trajectories optimization for the recurrent module. A linear scheduling of the KI penalty coefficient / adopted control target value for the KL. divergence of the posterior distribution. The target KL value is set to 7 for the layout Forced Coordination over the CoMeDi population. All other VAB models are chosen by target KL value of 32.
Train Cooperator agents. On Overcooked, the Cooperator trained by PPO (25]. We based our implementations on HSP 337. Reward shaping for dish and soup pick-up is used for the first 100M steps to encourage exploration. All results are reported with averaged episode reward and the standard error of at least 5 seeds.
Simulated agent populations. We use MAPPO [8] create the FCP 22 agent populations. Eight self-play agents are trained and three checkpoints for each agent are added to the population, making
the population size 3. For the CoMeDi agent population (24), we download the population proposed by the authors for the original five layouts [.. The population size is 8 for CoMeDi. For our custom layout Multi-strategy Counter, we use CoMeDi's 's official implementation and keep the
population size the same.
C Human Dataset
For the human dataset in the original Overcooked paper (I), their open-sourced dataset contains "16 joint human-human trajectories for Cramped Room environment, 17 for Asymmetric Advantages, 16 for Coordination Ring, 12 for Foreed Coordination, and 15 for Counter Circuit.." with length of T x 1200. In Multi-strategy Counter, we collect 38 trajectories with length T 400 which is closer the actual episode length during training.
D Hyperparameters
We use MAPPO to train our Cooperator agent. The architectures and hyperparamers fixed throughout all layouts. All policy networks follow the same structure where an RNN (we use GRU) followed by CNN.
The generative model follows similar architecture to the policy model. An encoder head and a decoder head are used produce variational posterior and action reconstruction predictions from the representations.
Computational Resources
We conducted our main experiments on clusters of AMD EPYC 64-Core Processor and NVIDIA A40/L40.. It takes about day train one Cooperator agent. The main experiments takes about 3600 GPU hours. We do some preliminary experiments to search for the best hyperparameters and training frameworks.
14

---

hyperparameter
value
CNN kernels
[3, 31. [3, 31. [3, 3]
CNN channels
[32, 64, 32]
hidden layer size
[64]
recurrent layer size
64
activation function weight decay environment steps parallel environments
ReI.U
100M (simulated data) or 150M (human data)
200
episode length PPO batch size
400
x 200 x 400
PPO epoch
15
PPO learning rate
0.0005
Generalized Advantage Estimator (GAE) X
0.95
discounting factor y
0.99
Table 1: Hyperparameters for policy models
hyperparameter
value
CNN kernels
[3, 31, [3, 31, [3, 3] [32, 64, 32]
CNN channels
hidden layer size
[256]
recurrent layer size
256
activation function
ReLU
weight decay parallel environments
0.0001
200
episode length epoch chunk length
400
500
100
learning rate
0.0005
KL penalty coefficient 3
0 + 1
latent variable dimension
Table 2: Hyperparameterss for VAE models
F
Human Evaluation
Since diversity of humans is key component in our approach, in this section report the demo- graphics of our participants. The study demographics include a population of 54% female and 46% male participants. The user demographies skewed towards younger ages, with 39% of participants
between ages 18 26, 44% of participants between 27 37, 11 of participants between 38 48, and 6% of participants ages 49 and above. The game experiences the participants are shown in Figure
21.1%
Agree
24.2%
51.9%
Strongly olssgree
5.3%
Disagree
Figure 7: Percent of participants who agree with the statement "I have experience playing the game Overcooked".
15

---

Performance vs Number of Gameplay Thaas
70
50
Figure 8: Human performance improves with the number of trials, indicating that the humans learn, change, and adapt their gameplay during the course of the evaluation.
F1 Humans are adapting during evaluation
As shown by Fig human performance improves with the number of trials, indicating that the humans learn, change, and adapt their gameplay during the course of the evaluation. In evaluation with real humans, each user can change their strategy at any point in the game. significant proportion of our users self-identify as novice players, both for video games and for Overcooked (17.9% and 49.5%, respectively, Figure & thus often exhibiting improved performance over the course of an increased number of trials. Figure [ provides evidence of this pattern, demonstrating that humans change their gameplay style over the course of the evaluation.
G Additional Human Study Results
G.1
Human-Al team scores
Agent
Training data source
Counter circuit
Multi-strategy Counter
FCP
26.24 + 0.96
34.35 + 1.47
FCP-generated population
FCP + GAMMA
57.87 1.35
65.36 + 1.24
CoMcDi
56.87 = 1.43
27.11 + 1.46
CoMeDi-generated population
CoMeDi GAMMA
72.17 = 1.61
34.72 + 1.65
MEP
76.19
1.89
64.02 1 2.07
MEP-generated population
MEP+ + GAMMA
81.10 + 2.03
88.30 +: 2.51
PPO + BC
53.51
1.37
85.26 + 2.28
human data
PPO + BC + GAMMA
59.67 + 1.35
77.53 + 2.00
GAMMA HA
MEP pop. + human data
91.11
2.96
93.09 + 3.19
Table 3: Human evaluation results. Our methods (GAMMA) show significant improvements.
conducted statistical significance tests and computed the p-value using the folm-Bonferroni correction. See Table 4
Hypothesis
Counter circuit (p-value)
Multi-(p-value)
FCP GAMMA FCP
1.27 x 10 10-69
1.43 x 10 50
CoMeDi + GAMMA > CoMeDi
7.31 x 10-12
3.25 10 10-3
MEP + GAMMA > MEP
0.639
2.04 x 10-12
PPO BC + GAMMA PPO BC
9.80 10-3
7.48 10
GAMMA HA > FCP
4.21 10-113
1.10 10
GAMMA HA > CoMeDi
1.55 x 10
3.43 x 10 77
GAMMA HA > MEP
1.43 x 10-4
2.92 x 10-13
GAMMA HA >> PPO BC
3.14 x 10-32
0.426
Table 4: Statistical significance (p < 0.05) is achieved for the majority of the results.
16

---

G.2 Qualitative analysis
We include additional analyses of users self-reported responses for qualitative questions from our user study. Figure shows the results for users' response to the agents' ability to adapt. The results indicate that two of our methods. FCP GAMMA and GAMMA--A-DDES receive higher ratings indicating better ability to adapt than their respective baseline agents, across both layouts. Figure T displays users' ratings for whether the agents demonstrated human-like behavior. The responses show that FCP + GAMMA, CoMeDi + GAMMA, and GAMMA-HH. -A--DFF exhibit the most human-like gameplay both layouts. Figure [T includes responses for whether the agents' behavior was frustrating. Individuals consistently reported that FCP GAMMA and GAMMA-HA-DE' demonstrated the least frustrating behavior.
consistently
Furthermore, we provide additional participant feedback for our agents from the user study as follows, starting with feedback for agents using our methods:
Feedback for FCP + GAMMA:
"It was very coordinated."
"This agent figured things out the fastest and worked with me well."
"When it was my way, moved of the way instead of blocking the path, which was issue with other agents."
"Much better cooperation by comparison." "
Feedback for CoMeDi GAMMA:
"He the best teammate."
"This one did well, it moved around me enough to complete tasks and we were relatively efficient passing items around."
"The agent figured out the next step would have done."
Feedback for PPO BC + GAMMA:
"It was very efficient and placed things down so I could grab it on the other side."
"This agent was very collaborative. All did was put onions out and he did the rest of the work super smooth."
"It behaved smoothly and intentionally. responded to my moves and helped rather than getting in the way."
In contrast, the feedback for the baseline agents is overall more negative, including themes such as frustration, inefficiency. and lack of cooperation.
Feedback for FCP (baseline):
"Frustrating""
"He got in the way and didn't help all."
"Just seemed to lack coordination and felt less intelligent."
Feedback for CoMeDi (baseline):
"The agent was helpful, but still did not fully cooperate as well as thought should have."
"The agent executing some random move actions before actually cooperating."
Feedback PPO BC (baseline):
"The agent was not helping much, it was doing its own thing, which meant we were not coordinating well."
"Agent did everything by himself; didn't 't see the onions the counter that I had placed; got in my way a bunch." "
"Very inefficient and lacking intelligence." "
17

---

Euiapp Agenial Abrlly seatse shout Lavet
S15
FCP CANNA
21b
14h
96
25
215
203
Galdd CamnA
035
Nestral
Strgrooge
(a) Counter circuit layout
(b) Multi-strategy Counter
Figure 9: Human ratings for different agents. Individuals were asked to respond to the following question: "The agent adapted to me when making decisions: (Strongyy Disagree, Disagree, Neutral, Agree, Strongly Agree]". FCP GAMMA and GAMMA HA-DFT consistently receive higher ratings for ability to adapt compared their respective baselines.
Aatinge for Whether Agerti Bebwior Noc H. Ee couster croat Layout
Peling Wtetter Agenti Echevtr We Iummn Uie dvers counter chestt
2Bh
X4
28
RS
38
255
15
NO
265
206
201
re: N" cauna
75
105
140.0C caneaa
Storgy deagie
Desgge
Raura
Agre
Seorgy og
Srang Ssages
Dkligne
Ages
Srongy Kyne
(a) Counter circuit layout
(b) Multi strti-strategy Counter
Figure 10: Human ratings for different agents, Individuals were asked respond the following question: "The agent's actions were human-like: (Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree)". FCP + GAMMA, CoMeDi + GAMMA, and GAMMA-HA-DIT consistently receive ratings for more human-like behavior compared their respective haselines.
Retingo Agenty Behaver Prestreting courter pront Lavaut
tating Agert! Produetin ponter drou
TOP CAMPAA
114
26
1%
36
34
Odlel
316
186
06
126
96
36
CAMMA sM
(a) Counter circuit layout
(b) Multi-strategy Counter
Figure 11: Human ratings for different agents. Individuals were asked respond to the following question: "The agent's behavior was frustrating: (Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree)". FCP + GAMMA and GAMMA HAA consistently receive ratings for less frustrating behavior compared to their respective baselines.
18

---

FCP
FCP GAMMA
CoMeDi
CoMeDi + GAMMA
MEP
MEP GAMMA
Cramped Room
Asymmetric Advantages
Coordination Ring
Reward
Reward
12
23
Forced Coordination
Counter Circuit
Muli-stratagy Counter
Rowand
Rovard
Reward
33
Figure 12: Learning curves for methods using simulated data across six layouts. Error bars are the Standard Error of the Mean (SE). All methods are evaluated using a held-out human proxy model as the partner player. GAMMA consistently shows better equal performance all layouts for both simulated data-generation methods (FCP, CoMeDi, MEP) when evaluated against the human-prow proxy model.
GAMMA (DFT) GAMMA HA FFFT)
GAMMA (DFT)
PPO + BC GAMMA
PPO BC + GANMA
GAMMA HA (FFT)
Counter Circut
Milti-stratayy Counter
Counter Circut
Multistrategy Counterr
(a) Evaluation against human proxy agent.
(b) Evaluation against held-out self-play agents.
Figure 13: Learning curves for methods using human data. When evaluated with held-out human proxy agent (a), human adaptive sampling learns faster on Counter Circuit, but does not reach better final performance since PPO-BC is trained to exploit pooxxy agent. With simulated self eeff-play partners (b), Human-Adaptive GAMMA with DFT shows better performance.
H Additional simulated results
Figure| 12 provides the original data for the performance simulated data.
In addition training the generative model on the human dataset IPPO-BC-GAMMA, we can also leverage Human Adaptive (HA) sampling on the generative model: (GAMMA-HA) HA). The learning
curves of these methods and the baseline PPO-BC 1 are shown in Figure 13 When evaluated
with held-out human proxy agent, our methods do not show great advantage. This is expected, because PPO-BC trained exploit a human proxy agent. However, in the human study, we show that this hurts its adaptation to more diverse real human players. On the other hand, we note that shows much faster learning in both layouts. Figure 133]shows that GAMMA HA also adapts better to held-out self-play agents compared to PPO-BC.
I Compare decoder-only fine-tuning and full fine-tuning
I.1 Full fine-tuning can suffer from insufficient human data
In some early experiments, we find that only fine-tuning the decoder with human data (GAMMA-HHE DFT) provides consistently strong performance on both layouts, whereas full fine-tuning (GAMMA- HA-FFT) provides the strongest performance on the first layout, but weaker performance on the second layout. At first, we hypothesize this is because the second layout is more complex, but the number of human coordination trajectories available for training (N 11) significantly less than the first layout (N 37). With more diverse potential strategies and less human data, we find the
19

---

Original Best: GAMMA HA (DFT)
Original GAMMA HA (FFT)
New GAMMA HA (FFT)
Reward
0
Stop
160
Figure 14: With larger KL Divergence penalty coefficient (3 in Eq [T], the new full-model fine-tuning (FFT) largely improves the original FFT and achieves a comparable performance with the original best decoder-only fine-tuning (DFT) method.
data is insufficient to fully fine-tune the generative model for the second layout. Since the entire model is fully fine-tuned, there is higher chance that the model overfits the training human samples when the amount of human data is extremely small. Therefore, we believe the results of the FFT model could be improved were where limited human data is available, since human data can be quite expensive and difficult to collect. We find that in this low data regime, GAMMA HA DFT still provides excellent performance. Whether to fine-tune the encoder is design choice that can be tuned for a particular domain based on the available data and the performance against simulated agents (since Figure T3 shows that FFT performed poorly here well).
collect more data. However, it is realistic to test the scenario
Agent FCP FCP + GAMMA
Training data source
Counter eircuit
Multi-strategy Counter
32.11 = 1.50
44.22 + 2.15
-CP-generated population
77.75 2.09
74.44 + 2.12
CoMeDi
69.62 = 3.31
32.02 + 1.89
CoMeDi-generated population
CoMeDi GAMMA
77.77 = 2.34
32.34 + 1.79
PPO + BC
61.77 + 2.53
97.73 + 1,90
human data
PPO + BC + GAMMA
72.32 - 1.71
95.72 + 1.75
GAMMA HA DFT
human data
82.82 1.84
103.76 + 1.96
GAMMA HA FFT
FCP-generated population
91.84 + 2.91
34.05 + 2.01
Table 5: Human evaluation results (outdated). Our methods (GAMMA) show significant improve- ments. However, GAMMA HA FFT is not stable on "Multi-strategy Counter".
1.2 Large regularization mitigates the problem of insufficient human data
Given the hypothesis that the human dataset is too small compared to the complexity of the Multi- strategy Counter environment, we find out that with larger regularization over the fine-tuned model on the original model, we can mitigate this issue. See Figure 14]
Can we condition the Cooperator policy on z?
One potential future work to improve the efficiency of online adaptation is to condition the policy z. During testing with novel humans, the Cooperator can then infer the 2 of the human player and adapt that
This method orthogonal our contributions where we focus on sampling partners with different to train the Cooperator. We did some preliminary work to test the z-conditioned Cooperator. As shown Figure [5] the performance of the z-conditioned Cooperator not stable, learns faster but the performance decreases when it is trained longer. Therefore, do not include this method in
20

---

Z-conditioned Cooperator
Mult-strategy
Reward
Reward
1kip
toneed Cooeprator
Best GAMMA HA Cooperator (Uncondilionas)
Figure 15: Performance of z-conditioned Cooperator. The z-conditioned Cooperator reaches higher reward in the Multi-strategy Counter. The performance decreases after the peak since the z-conditioned policy overfits the encoder.
our main experiment, but future work about how to better train the conditioned Cooperator is promising.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide summary of the main claims in the abstract and present our contributions clearly at the end of the introduction section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We describe the limitations of our work Section ??.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
this paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: the information for reproducibility is shown in Section A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: the code will be released on our demo website with detailed instructions (see Section A)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli///uuu guwde//ddee/ee/ Cot//ooodeeesiseeeeeee wwwpibmmsssioooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we provide implementation details and hyperparameters in Section and Section D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: we explicitly show the error bars for all our results with statistical significance (p<0.05).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: the compute resource details are discussed in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: all authors have reviewed the code of ethics and will follow it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: we discuss the societal impacts of our work Section ??.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: all the models released by this work are used for experimental game simulators only and do not have the risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: open-source assets of Overcooked from [1 and open-source code for baselines under MIT/GPL license. No other assets are used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: we include detailed documentation for our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/modell as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: we provide full instructions and consent form to the human subjective and follow the minimum wage requirements for compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: our human study is approved by IRB and we include this in Section 5
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "pJaSfyOk3X", "reviewer": "Reviewer_igMC", "summary": "The authors propose a novel method for training agents that effectively cooperate with humans by leveraging generative models that can sample the space of agent behaviors", "questions": "- how well would this approach generalize to more realistic settings? It seems like moving to the more complex setting led to reduced gains compared to baselines. \n- relatedly, how complex are the strategies learned by the Cooperator? Can they deal with humans who learn, adapt, and change their own strategies?\n- can the Cooperator instead be meta-learned and adapt online?\n- not entirely clear what is meant by \"Error bars [4] use the Standard Error of the Mean (SE) for statistical significance (p < 0.05)\". Are you saying that error bars correspond to 95% CI? or that they correspond to standard error? \n- following up on that, which of the results are statistically significant? hard to tell from the figures.", "limitations": "- Authors note human diversity is critical, but it's somewhat unclear whether human data comes only from WEIRD populations (almost certainly yes since participants were recruited from Prolific). This is worth noting in the human eval/limitations/impact sections since results may not be representative of broader populations (especially since authors mention assistance for elderly or disabled people as potential future application). A potential social impact of this is that a) agents may best serve only the demographics they were trained on, and b) agents may lead to reduction in creativity and diversity of strategies by forcing humans to adapt to the limited range of strategies that the agents are familiar with. \n- Authors do mention wanting to train on larger datasets in limitations, but also worth mentioning that current experiments are in very constrained environments and only in 2-player settings. In addition to dataset size, scaling along both of these dimensions would be important future work.\n- If I understand correctly, the human data used for fine-tuning or for creating the generators is collected either in solo-play or interaction with a simple existing policy. It seems unlikely then, that the GAMMA-based methods could deal with humans learning, adapting, and changing their strategies. Perhaps worth mentioning this as well.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-written and easy to follow\n- Experimental setup is clear, well-documented, and expansive\n- Code and even a live demo are available (very cute demo by the way!)\n- Novel method in an important and growing area", "weaknesses": "- Experimental results are okay, but not clear that GAMMA consistently outperforms baselines. This is true in both the simpler and more complex settings.  Even in cases where GAMMA-based methods have higher performance, the confidence intervals seem to overlap making it difficult to say whether results are simply due to chance. See Questions below for a few more things I'm wondering about regarding results and significance. Happy to raise my score if authors can show that the proposed method fairly consistently outperforms baselines in a statistically significant way (either by correcting me if I'm wrong in my interpretation of current results, or by increasing sample size to decrease confidence intervals). \n\nMinor suggestions (did not affect score):\n - increase text size in fig 2\n - report results in a table (can go in the appendix) as it is hard to tell from the graphs alone whether certain methods outperform others in statistically significant ways", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your detailed and insightful feedback. We are grateful that you took the time to check out our demo. We hope the following responses address your concerns.\n\n> Does the proposed method GAMMA fairly consistently outperform baselines statistically significantly?\n\nGAMMA does outperform baselines, which we clarify in the global response as well. The human evaluation results (Fig 6 and 7) should be considered the gold standard evaluation, and here GAMMA shows a statistically significant improvement over all baselines on both layouts. We provide a table of human evaluation results (Table 1) in our rebuttal PDF which makes this more clear. Other simulated metrics (Fig 4 and 5) are not always predictive of human evaluation performance. \n\n\n> What is the error bar in Fig 6? Which of the results are statistically significant?\n\nBecause the error bars correspond to the Standard Error of the Mean (SE), if two error bars are non-overlapping, the results are considered statistically significantly different at the p < 0.05 level. For more information on this, please see: G. Cumming, F. Fidler, and D. L. Vaux. Error bars in experimental biology. The Journal of cell 377 biology, 177(1):7–11, 2007. Table 1 in the rebuttal PDF bolds the GAMMA models that are significantly better than the baseline model trained with the same data. \n\n> How complex are the strategies that the Cooperator can learn? Can they deal with adaptive humans?\n\nIn our evaluation with real humans, participants are free to change the strategy they play during the episode. In fact, since many of our participants are novice players who have little experience with the game Overcooked or with video games in general, often their performance continues to improve as they play. We show in Figure (e) in the rebuttal PDF that performance improves consistently with the number of trials, providing evidence that humans are changing their gameplay throughout the evaluation. Since our method provides the best performance when playing with real humans, we believe it is best at adapting to their changing game play.\n\n> How is the human training data collected? If I understand correctly, the human data is collected either in solo play or interaction with a simple existing policy. It seems unlikely then, that the GAMMA-based methods could deal with humans learning, adapting, and changing their strategies.\n\nWe would like to clarify that the data is collected by randomly assigning two human players to play together, so it consists of human-human trajectories where players can adapt and change their strategies. Further, as we show above, in our evaluation human players do learn to improve their performance over time, showing evidence of learning and adapting throughout the evaluation.\n\n> The gains are reduced for more complex settings.\n\nWe believe GAMMA is a general approach that can still be applied to more complex settings. Figure 6 shows that GAMMA HA DFT still provides statistically significantly higher performance than competitive baselines in the more complex layout. In the original experiment, FFT is less effective on Multi-strategy Counter due to the relative lack of human data. In Figure (b) in the rebuttal PDF, We solve this issue by using a larger KL Divergence penalty coefficient and now both FFT and DFT achieve the best performance on Multi-strategy Counter.  \n\n> Can the Cooperator instead be meta-learned and adapt online?\n\nCurrently, our Cooperator adapts according to the history of the other agents. Thank you for the suggestion. We agree that we can also make $z_i$ part of $\\pi$. **Here we note, sampling from $z_i$ and making $z_i$ part of $\\pi$ are orthogonal ideas and can be combined to train the cooperator.** We have conducted the following experiment. \n\nWe implement this idea by allowing the Cooperator to condition on the z inferred by the VAE. We run this experiment and present the results in Fig. (c) of the rebuttal PDF. Our findings indicate that this approach improves both learning efficiency and performance on the Multi-strategy Counter layout.\n\n\n\n> Limitations. The human data in this work may not be representative. The current experiments are only in two-player settings.\n\nThank you for pointing out these limitations. We will add a limitations section to the revised version of the paper with the following text:\n\n''In this work, our human studies recruit participants from Prolific, which may not be representative of broader populations. Additionally, our human dataset is limited, which could reduce the diversity of strategies and force participants to adapt to strategies that the Cooperators are already familiar with.\n\nWe focus on the two-player setting in this study following prior work [1] [2] because it is a first step toward enabling an AI assistant that could help a human with a particular task. Scaling up to more agents would exponentially increase the dataset size with our current techniques. Therefore, better sampling techniques are needed to address this issue.''\n\n\n[1] D. Strouse, K. McKee, M. Botvinick, E. Hughes, and R. Everett. Collaborating with humans without human data.\n\n[2] C. Yu, J. Gao, W. Liu, B. Xu, H. Tang, J. Yang, Y. Wang, and Y. Wu. Learning zero-shot cooperation with humans, assuming humans are biased.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R0phd5LsWC", "reviewer": "Reviewer_pvTn", "summary": "The paper addresses the challenge of training AI agents that can effectively coordinate with diverse human partners in cooperative tasks without prior interaction. This work demonstrates that using generative models to create diverse training partners can significantly improve an AI agent's ability to coordinate with novel human partners in cooperative tasks. The proposed approach offers a promising direction for developing more adaptable and effective AI assistants for human-AI collaboration scenarios.​​​​​​​​​​​​​​​​\n\nThe key contributions of the work are listed as follows:\n\n1. The work proposes a novel approach, Generative Agent Modelling for Multi-agent Adaptation (GAMMA), that uses generative models to produce diverse partner agents for training a robust Cooperator agent. Through conducted experiments using the Overcooked environment, the authors demonstrate that GAMMA consistently outperforms baselines in zero-shot coordination with novel human partners, both in terms of task completion and subjective human ratings.\n\n2. The authors show the flexible data integration of GAMMA, i.e. it can be trained on both simulated agent data and real human interaction data. This addresses the dual challenges of limited human data and insufficient coverage of human behaviours in synthetic data.\n\n3. The authors also propose a method, Human-Adaptive Sampling, to efficiently incorporate a small amount of human data to bias the generative model towards producing more human-like partners during training.\n\n4. The study brings together and directly compares two previously separate approaches to zero-shot coordination: training on simulated population data, and training on human data.", "questions": "1. Why is \"training on human data is highly effective in our human evaluation\" counterintuitive?\nThe authors mention in line 95 that \"contrary to past published results, training on human data is highly effective in our human evaluation\". I got confused by this claim as it sounds intuitive to observe better performance by using human data in human evaluation.\n\n2. Can increasing sampling temperature also lead to greater coverage over the strategy space?\nIn line 200, the authors claim that \"the generative model has significantly greater coverage over the strategy space\". A natural alternative could be simply increasing the sampling temperature. So, I wonder if this simple trick could also boost the performance/efficiency of the same baseline methods?\n\n3. Is H3 a hypothesis?\nFrom my perspective, the H3 stated from line 261 to line 263 is not a hypothesis.\n\n4. Why is it useful to evaluate the method by a population of held-out self-play agents?\nAs stated in line 289, the human evaluation is completed on \"a population of held-out self-play agents\". After reading the paragraph from line 282 to line 289, I'm still unclear about the rationale of doing so. More importantly, line 95 state that human data is not so effective in human evaluation. Then, why is fine-tuning on human data a good evaluation?\n\n5. How are the number of policies (8) and the numbers of random seeds (5) decided in human evaluation?", "limitations": "Please refer to the weaknesses section.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. *The research problem is of great significance*:\nThis work studies zero-shot human-AI coordination, which is a key topic for applying AI in the real-world. The key bottleneck the work aims to address is the coverage of behaviours by the simulated humans obtained by behaviour cloning over datasets of human cooperation, or by using MARL. This work offers a promising direction for developing more adaptable and effective AI assistants for human-AI collaboration scenarios.​​​​​​​​​​​​​​​​\n\n2. *The paper is well-written and well-organised*:\nOverall, the paper is well-written, and very-well-organised. Despite the issues mentioned in the weaknesses section below, the motivation, the proposed method, and all experiment results are clearly delivered. Overall, I quite enjoy reading the paper.\n\n3. *The conducted experiments are thorough and the hypotheses are clearly formulated*:\nIt's pleasant to see that the hypotheses to verify are clearly listed at the beginning of Section 5, and the following results are also demonstrated following the same order of hypotheses.\n\n4. *The proposed method is very intuitive*:\nIn this context, the term \"intuitive\" is used to convey a positive quality. It makes a lot of sense to me that human partners can be represented by latent variables, like the user embeddings used in recommendation systems. It also makes sense that depending on the user embeddings leads to higher coverage of the behaviours of simulated human partners.", "weaknesses": "1. *Results lack of statistical significance, thus persuasiveness*:\nDespite all the good aspects of the work, including the motivation and the intuition, my biggest concern is that the improvement of the performance doesn't look significant. First, in the Cramped Room in Figure 4, there seems to be overfitting after $5e7$ steps. In the meantime, GAMMA doesn't lead to improvement on FCP at that time step. Similarly, there seems no (statistically sifnificant) difference between FCP and CoMeDi with and without GAMMA at the $5e7$-th step in Forced Coordination. Considering that the performance of both FCP and CoMeDi decrease after $5e7$ steps in Forced Coordination as well, a more rigorous conclusion could be that GAMMA can prevent the overfitting, instead of improve the performance. Secondly, the performance shown in Figure 5 also suggests that GAMMA doesn't improve the performance, since the orange curves (PPO + BC + GAMMA) twist with the blue curves (PPO + BC). Even though the methodology is convincing, the results make me doubt whether the proposed GAMMA can indeed lead to significant improvement.\n\n2. *Need to illustrate the necessity of sampling from $z_i$*:\nIn Section 4.2, the authors illustrate the human behaviour representation $z_i$ is used to generate batches of MDPs. A straightforward and intuitive alternative practice is simply to make $z_i$ a part of conditionals for $\\pi$. The authors need to provide either a rational in more details or more empirical evidence on the chosen implementation, i.e. using $z_i$ to generate $\\\\{M_{z_i} \\\\}_{i=1}^{N}$. \n\n3. *Length of games during evaluation*:\nAs an experienced player of Overcooked, the 60-second setup used in human evaluation looks too short to me. According to my own experience, for human-human cooperation, it usually takes a whole trial of the game, i.e. >120 seconds. This makes me doubt whether improvement by GAMMA is only on the adaptation efficiency. Moreover, this also makes me doubt whether the improvement observed in the evaluation is only due to some randomness. I would be keen in seeing more trials of each method with human partners in longer games.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed review. We address your questions below.\n\n> Results lack statistical significance\n\n- First of all, for the most important human experiment, GAMMA is statistically significantly better (p < 0.05).  See the common response in detail.\n- For Cramped Room and Forced Coordination in Fig 4, we agree GAMMA does not provide significant improvement. The main reason is that the environments are so simple that the optimal policies can be easily found. Specifically, in Cramped Room, the agents simply need to fill the pot whenever it is available. In Forced Coordination, the left agent only needs to place the onions and dishes on the central table, while the right agent only needs to collect the onions for the pot and use the dish to complete an order. Note that for more complex Coordination Ring or Counter Circuit, GAMMA gives significant gains.\n- Regarding Fig 5, we choose these experiments following [2] [4]. We agree that GAMMA’s performance is only comparable to the strongest baseline. **Note there is a problem with this evaluation.**  As shown in [1], the performance evaluated by the BC human proxy model is not predictive of human study. The BC model trained on limited data, is not reactive or adaptive in the same way as humans and therefore has considerably different behavior. While the proxy metrics are relatively cheaper to run than real human evaluations, both our work and [1] find that they can be misleading for performance against real humans. **Again, we emphasize that the most important experiment is the human evaluation (Fig 6) and GAMMA is statistically significantly better.**\n\n> Illustrate the necessity of sampling from $z_i$\n\nWe first explain the rationale of sampling from $z_i$. Our primary idea is to build a population and train a cooperator against it. We choose VAE to represent this distribution and here $z_i$ serves as the latent vector to generate partner policies. \n\nWe agree that we can also make $z_i$ part of $\\pi$. **Note that sampling from $z_i$ and making $z_i$ part of $\\pi$ are orthogonal ideas and can be combined together.** We have conducted two additional experiments. \n\nFirst, we make the Cooperator condition on the $z$ inferred by the VAE. In Fig (c) in the rebuttal PDF, we found this approach improves both learning efficiency and performance on Multi-strategy Counter.\n\nSecond, as another baseline, we use the $z$-conditioned decoder of the VAE as the policy. The results are presented in Fig (d) in the rebuttal PDF. We find that the performance of the $z$-conditioned decoder is not good compared to the Cooperators trained by sampling partners from the VAE. \n\n> The length of games is short\n\nWe appreciate your insight as an experienced Overcooked player. We chose this time limit to be consistent with prior works; 60s for FCP and 40s for CoMeDi. In our evaluation, the human-AI team can usually finish 2-5 soups (60-100 scores) in 60s.\n\n> Can increasing the sampling temperature also boost the performance?\n\nMaking the partner's actions more random could simulate a less skilled agent. Our VAE training dataset already includes random agents, enabling it to interpolate between expert and random agents.\n\nThe VAE offers a significant advantage over merely adding noise to players' actions, as it can combine different skillful strategies. For example, if partners follow strategies A, B, or C, adding noise to A produces an inexpert A. In contrast, the VAE can create a partner proficient in all strategies, like ACBAAC. We hypothesize the VAE better covers possible skilled partner strategies (Fig 1), and our experimental results support this hypothesis.\n\n\n> Why mention ``training on human data is highly effective’’ particularly in line 95?\n\nWe agree that it is intuitive that adding human data should improve performance. Prior work [2] shows that FCP > PPO-BC in four layouts including our Counter Circuit, and PPO-BC > FCP for Asymmetric Advantage. In our experiments, we found PPO-BC also effective for both Counter Circuit and Multi-strategy Counter. We hope our findings can encourage the community to engage in further discussion and empirical studies on this problem.\n\n> Why use held-out self-play agents in Fig 5b? Line 95 states that human data is not so effective in human evaluation. Then, why is fine-tuning on human data is good?\n\nWe use held-out self-play agents to follow existing works [2, 4]. The reason to use self-play agents is that a human-proxy model is not a good evaluation metric, as shown in prior work [1]. **We emphasize again that the most important experiment is the human evaluation (Fig 6) and GAMMA is statistically significantly better.**\n\nOur finding shows that human data is quite effective, while prior work [2] mentioned in line 95 focuses on training without human data. We hope our findings can encourage more attention to this problem.\n\n>  How are the number of policies (8) and the number of random seeds (5) decided in human evaluation?\n\nWe follow the same number of five random seeds as prior works [2, 3] to ensure the statistical significance of our results We did two separate human evaluation rounds of the two layouts we used. For each round, only one layout would be tested. We first shuffled the order of eight algorithms randomly, and then for each algorithm, we sampled one of five random seeds to play against humans.\n\n> H3 is not a hypothesis.\n\nThanks. We will revise H3 to, “Generalizing to novel humans: we hypothesize that GAMMA will generalize more effectively to novel humans than prior methods, showing improved performance in both game score and human ratings.”\n\n[1] M. Carroll, et al. On the utility of learning about humans for human-ai coordination. \n\n[2] D. Strouse, et al. Collaborating with humans without human data.\n\n[3] R. Zhao, et al. Maximum entropy population-based training for zero-shot human-ai coordination.\n\n[4] C. Yu, et al. Learning zero-shot cooperation with humans, assuming humans are biased.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CwKUTKZaQB", "reviewer": "Reviewer_VM8E", "summary": "This paper proposes GAMMA, a generative model that learns a latent representing partner agents' strategy. Through interpolation, the latent can be used generate diverse partners to train ego agents for human-AI cooperation. Authors studied training the generative model with both simulated data and real-human data and conducted evaluation with both human proxies and real humans. Experiment results show GAMMA improves both objective performance and human subjective ratings.", "questions": "See weaknesses.\n\nPlease at least add MEP [3]  to the baselines in your experiments (since it has very similar pipeline as FCP in your paper and easy to run). I would happily increase my score upon seeing such comparisons.\n\n[3] Zhao, Rui, et al. \"Maximum entropy population-based training for zero-shot human-ai coordination.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023.", "limitations": "Limiations are discussed in the appendix.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written and easy to comprehend.\n\n2. The targeted issue of diversity of partner agents is important in zero-shot coordination and human-AI coordination.\n\n3. Human study is conducted at relatively large scale to show the efficacy of GAMMA in human-AI cooperation.", "weaknesses": "1. The 'state-of-the-art' baselines in this paper are outdated (especially FCP). I would be happy to see comparison against works like [1],[2],[3], which are also targeted at zero-shot coordination.\n\n  [1] Mahlau, Yannik, Frederik Schubert, and Bodo Rosenhahn. \"Mastering Zero-Shot Interactions in Cooperative and Competitive Simultaneous Games.\" arXiv preprint arXiv:2402.03136 (2024).\n\n  [2] Yan, Xue, et al. \"An Efficient End-to-End Training Approach for Zero-Shot Human-AI Coordination.\" Advances in Neural Information Processing Systems 36 (2024).\n\n  [3] Zhao, Rui, et al. \"Maximum entropy population-based training for zero-shot human-ai coordination.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023.\n\n2. In Fig. 4,5, GAMMA's improvements over baselines seem marginal.\n\n3. The interpolation visulization needs better explanation. What do the points in Figure 1 stand for? Is it policy $\\pi$ or hidden vector $z$? I can't find the reletive information to comprehend this figure.\n\n4. Adding to the last question, utilizing interpolation to enhance partner diversity is not new in zero-shot coordination field. PECAN [4] directly interpolates parnter policies through policy ensemble instead of hidden vector $z$ (see figure 4 in [4]). What is the advantage of GAMMA by interpolating at the level of hidden vectors? Does it offer better partner diversity? This should be at least discussed in the paper.\n\n[4] Lou, Xingzhou, et al. \"Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination.\" arXiv preprint arXiv:2301.06387 (2023).\n\n5. Typos:\n\"data-hungry seuential decision making algorithms...\", on page 1\n\n\"We propose a novel, ecnonomical way\" on page 2\n\ne.t.c.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! For typos, we will correct them in the next version. We address your concerns below.\n\n>  The baselines are outdated.\n\nThanks for the suggestion, we have run MEP (see Fig 2 in the rebuttal PDF) and show that GAMMA can still improve its performance.\n\nWe agree all the methods (MEP, E3T, SBRLE, and PECAN) that you mentioned are important and newer works in the human-AI cooperation field. We would like to point out that our GAMMA can not only 1) improve the diversity of the population with generative agents, which those prior works did; but also 2) combine another line of work of human modeling from human data under our human-adaptive sampling techniques.\n\nTo test 1), we need to show that GAMMA can improve diversity by generative agents trained on any synthetic population. Since FCP is the most popular and CoMeDi is one of the latest works, we believe these are two representative populations to test GAMMA. We will include the additional results with MEP in the revised version of the paper. \n\n> GAMMA's improvements seem marginal in Fig. 4,5\n\n- First of all, for the most important human experiment in Fig 6, it is statistically significant that GAMMA is better than baselines (p < 0.05).  See more details in the common response.\n\n- Regarding Cramped Room and Forced Coordination in Figure 4, we agree GAMMA does not provide significant improvement. The main reason is that the environments are so simple that the optimal policies can be easily found for certain algorithms. In more detail, in Cramped Room, the agents simply need to fill the pot whenever it is available. In Forced Coordination, the left agent only needs to place the onions and dishes on the central table, while the right agent only needs to collect the onions for the pot and use the dish to complete a soup order. Note that for more complex environments like Coordination Ring or Counter Circuit, GAMMA gives significant gains.\n\n- Regarding Figure 5, we choose these experiments to follow existing works [1] [4]. We agree that GAMMA’s performance is only comparable to the strongest baseline but not significantly better. **On the other hand, we note there is a problem with this experiment.**  As shown in [5], the performance evaluated by the human proxy model (a BC model) is not predictive of performance with real humans since BC models are trained on limited data, are not reactive or adaptive in the same way as humans and therefore have considerably different behavior than real human agents on evaluation. While the proxy metrics are relatively cheaper to run than real human evaluations, both our work and [5] find that they can be misleading in terms of final coordination performance with real humans. **Again, we emphasize that the most important experiment is the human evaluation (Figure 6) and GAMMA is statistically significantly better.**\n\n\n\n> What do the points in Figure 1 stand for?\n\nWe provide a detailed response in the common rebuttal. It is the latent vector $z$. For all population $P$, each point of $P$ is computed by 1) first sampling an agent in $P$; 2) using the agent to generate an episode with self-play, and 3) encoding the episode into $z$ using VAE. These high-dimensional latent vectors are compressed to 2D using t-SNE. We promise to add this explanation to the paper in the future.\n\n> Compared to PECAN [2], what is the advantage of GAMMA by interpolating at the level of hidden vectors?\n\nThank you for pointing us to the PECAN paper [2], we will include a citation of this work and the following discussion of the advantages of using GAMMA in the revised version of the paper:\n\n''PECAN [18] proposes an ensemble method to increase the partner population diversity. In PECAN, partner agents are categorized into three groups based on their skill levels. With GAMMA, the skill levels are automatically learned and encoded in the latent variable $z$. And by sampling different $z$, generative partner agents with different skill levels are thus sampled.\n\nGAMMA offers several advantages over this method. First, GAMMA provides a controllable way to sample only from a target subpopulation (e.g., humans) by adaptive sampling over the latent space. Second, it is known that with interpolation on the latent variable $z$ [3], generative models can produce novel samples beyond the weighted sum of low-level pixels/actions. ''\n\n\n> Typos: \"data-hungry seuential decision making algorithms...\", on page 1.\n\nThank you for your time and attention to detail. We promise to fix that in the future version.\n\n\n[1] C. Yu, J. Gao, W. Liu, B. Xu, H. Tang, J. Yang, Y. Wang, and Y. Wu. Learning zero-shot cooperation with humans, assuming humans are biased.\n\n[2] Lou, Xingzhou, et al. \"Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination.\"\n\n[3] Radford, Alec, Luke Metz, and Soumith Chintala. \"Unsupervised representation learning with deep convolutional generative adversarial networks.\" \n\n[4] D. Strouse, K. McKee, M. Botvinick, E. Hughes, and R. Everett. Collaborating with humans without human data.\n\n[5] M. Carroll, R. Shah, M. K. Ho, T. L. Griffiths, S. A. Seshia, P. Abbeel, and A. D. Dragan. On the utility of learning about humans for human-ai coordination.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9gesbM5mX2", "reviewer": "Reviewer_UQLZ", "summary": "This work proposes a novel method to train agents to coordinate with humans in team tasks. The main contribution is that instead of training an RL cooperator policy based on a limited number of human models, the authors use a generative model to capture the latent variable representation of human policy. The learned continuous latent space is believed to cover the strategy space of a specific team task, therefore can be used to sample diverse human policies during agent training. Experimental results confirm the benefit of introducing generative models in comparison to previous methods using behavior cloning human models or population-based RL models.\nThe authors also conduct human evaluations to pair trained agents with novel human participants via crowd-sourcing. The team performance and perceived utilities are higher when human players are paired with agents trained with the proposed method.", "questions": "1. How are different policies sampled and visualized on the latent strategy space in Fig 1?\n\n2. When sampling different latent variables of the partner at each episode, could PPO guarantee to learn an optimal cooperator policy?\n\n3. Are trajectories in the human data set all from the same human-human team with a consistent strategy or a population of individual human players with different strategies randomly paired into teams? Are those human strategies optimal or not?\n\n4. How is the held-out performed on human data and self-play agents respectively? Line 475 only mentions a 30%-70% split without mentioning if agent policy type is stratified.\n\n5. What does this sentence mean in lines 286-288? If the held-out is properly performed on human data, e.g. removing trajectories independently collected from different human participants, PPO-BC has no chance of exploiting information about the unseen evaluation data set. This question might be related to the previous concerns about human data distribution and collection details.\n\n6. How novel are the strategies in human evaluation compared to the original training human data?\n\n7. How accurate is the BC model? What is the action prediction score?", "limitations": "In the supplementary materials, the authors acknowledge the limitation of using limited human data which prevents the generative model from learning from scratch. I actually see this as a strength of the paper since human data is usually expensive to collect in practice. While the limited amount of human data is acceptable, the limited characteristics of human data might hurt the further application of the proposed method. As discussed earlier, if the authors could consider examining the quality and features of the used data set, it would shed light on learning to collaborate with humans in ad-hoc teamwork.\n\nThere are a few inconsistent patterns within the experimental results (e.g. Figure 5 (a) and Figure 6 (b)) where the proposed method does not achieve better performance than the baselines. The authors attribute those failed cases to limitations of the training data collection process. This actually illustrates the limitations of the proposed method in solving the 2 main challenges claimed in the paper: 1) lack of real (high-quality) human data and 2) the difficulty of synthetically covering the large strategy space of human partners.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well written and easy to follow. The introduction and related work sections motivate the work and position it among previous literature well. The problem definitions and methods section are clear. I enjoyed reading it.\n\n2. The problem of learning to cooperate with (unseen) humans is generally underexplored. This work is relatively novel by introducing a generative model and the adaptive sampling method to capture diverse human strategies.\n\n3. The experiments are well designed. The proposed method is evaluated among a variety of task scenarios against traditional methods. Results show that the proposed GAMMA method improves the team performance in comparison to baseline methods such as population RL or behavior cloning. The evaluation environment, i.e. Overcooked-AI, is a rather common benchmark facilitating comparison with other work in the future.\n\n4. The authors conducted evaluations with real human participants, instead of purely relying on simulated human models. The results confirm the actual ad-hoc teamwork performance of the proposed agent with unseen human participants, yielding high ecological validity.", "weaknesses": "1. This work lacks a formal analysis about the constructed latent space of human policy given the training data set. There seems to be no theoretical guarantee or quantitative analysis about the generalizability of the proposed generative model over training data, as claimed in lines 198-203. If the collected trajectories in D are biased / do not cover the entire strategy space, the performance of the proposed method might be impacted. Provide more details about Fig. 1, or quantitative analysis of the generalization / interpolation properties would help the readers to value the proposed method.\n\n2. Details about the human data used during training are missing. Are they independently collected from human teams with optimal / consistent team strategies? The setup is very important because the policy distribution of the human data would impact the learned generative models and human proxy models used during training. Explaining the data collection setups and comparing the nature of collected trajectories from agents / humans would enhance the paper.\n\n3. Training a Cooperator policy via RL given a partner policy conditioned on a distribution of latent variables is not trivial. There seems to be a hidden assumption about the distribution parameters to make sure partner policies do not differ too much so the RL policy can converge. The paper does not report details about this process.\n\n4. The hidden assumption of human-adaptive sampling is that human data are closer to the target partner policy compared to synthetic data, so we want to fine-tune the generative model and Cooperator policy on it. This might not be the case if human trajectories are noisy.\n\n5. The authors mention type-based reasoning in the related work section but do not include them in the baselines [1]. It is worth discussing how the proposed method and type-based methods perform differently, given different human policy distributions.\n\n6. One potential future direction is to extend the framework to be online adaptive to the target partner during interaction. For example, using a similar technique to infer the human-centered latent Gaussian distribution of a specific individual human, and change the agent action policy accordingly on the fly.\n\n[1] Zhao, M., Simmons, R., & Admoni, H. (2022, October). Coordination with humans via strategy matching. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 9116-9123). IEEE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and positive feedback on our comprehensive real-human study. We're glad you find it has high ecological validity.\n\n> There are a few experiment results (Fig 5a and 6b) where GAMMA + HA (FFT) is not good on Multi-strategy Counter.\n\nWe acknowledge that the FFT method originally performed poorly due to limited human data. Our further experiments found that increasing the regularization can mitigate the problem of lack of human data, enabling GAMMA HA FFT to provide good performance in Multi-Strategy Counter as well. See global response and Fig (e) in the rebuttal PDF for details.\n\n> One potential future direction is to infer the human latent variable […]\n\nThank you for pointing out this! We agree that we can also make $z$ part of $\\pi$. **Note sampling from $z_i$ and making $z_i$ part of $\\pi$ are orthogonal ideas and can be combined to train the cooperator.**\n\nWe implement this idea by allowing the Cooperator to condition on $z$ inferred by the VAE. We present the results in Fig. (c) of the rebuttal PDF and find that this improves both learning efficiency and performance on Multi-strategy Counter. \n\n> Are trajectories in the human dataset all from the same human-human team […]\n\nFor layouts proposed by [1], we use their open-source human from randomly paired human-human teams. For the new Multi-strategy Counter, we collect data similarly, sampling players randomly and explaining the game rules before collecting their gameplay data. These strategies are not always optimal; in Figure (f) of the rebuttal PDF, 49.5% of the players are novice overcooked players. People played according to their own strategies without prompts. We will add this information to the paper.\n\n> Details of Fig 1.\n\nPlease refer to our common response.\n\n> Could PPO guarantee to learn an optimal cooperator policy?\n\nIn general, gradient-based RL is not guaranteed to converge in multi-agent settings [3].  Although we can not ``guarantee’’ PPO to learn optimally, we give it enough training time (1.5e8 steps) to converge.\n\n> There seems to be no theoretical guarantee [...]\n\nWe will soften the claims in lines 198-203. We acknowledge assuming the human data for training is representative of the target population during the human study. We will add the following limitation to the paper:\n\n\"We assume the human dataset for VAE training and human-adaptive sampling matches the distribution of human players during evaluation. This may not hold if the dataset quality is low. Addressing distribution shift from training to testing is important and will be our future work.\"\n\n> How is the held-out performed on human data and self-play agents respectively? [...]\n\nThe 70%-30% split is only used to determine the validation dataset for VAE training. The held-out agents will never be used for training VAEs or Cooperators.\n\n> [...] examining the quality and features of the used data set, [...]\n\nWe provide a detailed introduction to human data here and will add this to the paper. For the human dataset in the original Overcooked paper [1], their open-sourced dataset contains ``16 joint human-human trajectories for Cramped Room environment, 17 for Asymmetric Advantages, …, and 15 for Counter Circuit.’’ with length of $T\\approx 1200$. In Multi-strategy Counter, we collect 38 trajectories with length $T\\approx 400$ which is closer to the actual episode length during training.\n\nFor the human study, we have made additional plots that show the participants’ distribution of experience levels for playing Overcooked. In Figure (f) in the rebuttal PDF, 49.5% of players agreed that they had no experience of playing Overcooked. We also recommend looking at the demos on our website, where we show videos of humans using different strategies. \n\n>  What does this sentence mean in lines 286-288? [...]\n\nWhile the proxy metrics are relatively cheaper to run than real human evaluations, both our work and [1] find that they favor the PPO-BC method, and thus can be misleading for final coordination performance with real humans. **Again, we emphasize that the most important experiment is the human evaluation (Figure 6) and GAMMA is statistically significantly better.**\n\n> How novel are the strategies in human evaluation […]?\n\nFor all human players, we only give them basic instructions about the game, so they are free to use unique strategies. Especially for Counter Circuit, we recruit an entirely new population of participants using a different platform and potentially different recruiting criteria of [1]. Therefore, this test-time population could be quite different from the training dataset [1] released several years ago.\n\n> How accurate is the BC model? [...]\n\nThe accuracy is around 55-60%. Under self-play, our BC model reaches 40-80 scores (2-4 soups). We use the same dataset [1] to train the BC model as prior work.\n\n> Compare GAMMA with type-based reasoning work [2].\n\nWe thank the reviewer for pointing us to [2], which has a similar setting to our PPO + BC + GAMMA method. During training, [2] clusters human trajectories into different strategies and learns the best response for each, while GAMMA uses a VAE over the entire human dataset to train a universal Cooperator. When facing a novel human, [2] infers the human's strategy and applies the best response to it, whereas GAMMA uses the universal Cooperator. Thus, while [2] is limited to partner strategies in the human dataset, PPO + BC + GAMMA can potentially generate novel partner policies through interpolation. Moreover, unlike [2], which only uses human data, GAMMA can combine synthetic and human data with controllable sampling like GAMMA HA. We will cite this work and include the discussion in the paper.\n\n\n[1] M. Carroll, et al. On the utility of learning about humans for human-ai coordination. \n\n[2] Zhao, et al. Coordination with humans via strategy matching.\n\n[3] Mazumdar, et al. \"On finding local nash equilibria (and only local nash equilibria) in zero-sum games.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their detailed reviews and feedback. 

To summarize the positive feedback, reviewers recognize the importance of the problem we studied and find the paper well-written with well-designed experiments. We are thankful to reviewer igMC for taking the time to explore the demo. We also appreciate that reviewers find our method intuitive (pvTn) and novel (igMV), and commend our large-scale human study (VM8E) and the high ecological validity (UQLZ) of our work.

We first address a common question asked by all reviewers.

> Which results are statistically significant showing GAMMA is better than baselines?

All reviewers ask whether GAMMA is better than baselines in a statistically significant way. Yes, GAMMA is. Below we provide the results in tabular format as well as an additional explanation to make this more clear.

The real human experiment (the most important evaluation metric for zero-shot human-AI cooperation) is shown in Figure 6. For our error bars, we use the **standard error of the mean (SE), which is calculated as $\text{SE} = \frac{ \text{std dev} }{ \sqrt{n} }$**. We use a total of $n=80$ participants recruited online. Figure 6 in the original draft and the Table below shows we have 1) FCP + GAMMA > FCP; 2) CoMeDi + GAMMA > CoMeDi; 3) GAMMA FFT/DFT > PPO-BC. **Furthermore, for these comparisons 1), 2), 3), the error bars do not overlap, and thus they are statistically significant (p<0.05).** That is, for the same training source, GAMMA consistently improves performance. These results demonstrate that when GAMMA is applied to an existing population method like FCP or CoMeDi it consistently improves their performance, and it can also make better use of human data than the best existing baseline designed for use with human data (PPO-BC). We also note that GAMMA is also consistently rated higher according to subjective human ratings (Fig 7).

| Agent           | Training data source                  | Counter circuit    | Multi-strategy counter  |
|-----------------|---------------------------------------|--------------------|-------------------------|
| FCP             | FCP-generated population | 32.11 ± 1.50       | 44.22 ± 2.15            |
| FCP + GAMMA     |                                       | **77.75 ± 2.09**   | **74.44 ± 2.12**        |
|-----------------|---------------------------------------|--------------------|-------------------------|
| CoMeDi          |CoMeDi-generated population | 69.62 ± 3.31       | **32.02 ± 1.89**        |
| CoMeDi + GAMMA  |                                       | **77.77 ± 2.34**   | **32.34 ± 1.79**        |
|-----------------|---------------------------------------|--------------------|-------------------------|
| PPO + BC        | human data          | 61.77 ± 2.53       | 97.73 ± 1.90            |
| PPO + BC + GAMMA|                                       | 72.32 ± 1.71       | 95.72 ± 1.75            |
|-----------------|---------------------------------------|--------------------|-------------------------|
| GAMMA HA DFT    | human data + FCP-generated population | 82.82 ± 1.84       | **103.76 ± 1.96**       |
| GAMMA HA FFT    |                                       | **91.84 ± 2.91**   | 34.05 ± 2.01            |




We also provide the results of 4 new experiments in the rebuttal PDF, as well as additional information about the population of human participants in our study, which we believe will help address the concerns raised in the reviews. 


Next, we provide more details of the new experiments.

> New results for an additional baseline MEP.

As requested by reviewer VM8E, we add MEP as a new baseline. See Fig (a) in the rebuttal PDF. We find GAMMA can improve the performance of MEP.

> New results for GAMMA FFT HA on Multi-strategy Counter.

To answer the question of reviewer UQLZ, we improve GAMMA FFT HA and run new experiments. See Fig (b) in the rebuttal PDF. We show that with larger regularization, GAMMA FFT HA can achieve comparable performance as the best method on the Multi-strategy Counter layout.

> New results for $z$-conditioned policy.


 As requested by reviewer pvTn, we add new results of the $z$-conditioned policy. See Figure (c) and (d) in the rebuttal PDF. In Figure (c), we show the performance of $z$-conditioned Cooperators. In Figure (d), we show the performance of $z$-conditioned decoders. We found that the $z$-conditioned Cooperator can improve performance on the Multi-strategy Counter layout.


> Figure showing humans are adapting during evaluation.


See Fig (e) in the rebuttal PDF.  Human performance improves with the number of trials, indicating that the humans learn, change, and adapt their gameplay during the course of the evaluation. In our evaluation with real humans, each user can change their strategy at any point in the game. A significant proportion of our users self-identify as novice players, both for video games and for Overcooked (17.9% and 49.5%, respectively, Figure f), thus often exhibit improved performance over the course of an increased number of trials. Figure (e) in the rebuttal PDF provides evidence of this pattern, demonstrating that humans change their gameplay style over the course of the evaluation. We hope this help address the concern of reviewer igMC.

> Clarification about how Figure 1 is created.

Reviewer VM8E and UQLZ ask about the details of Figure 1. This is how we get the visualization results in Figure 1.

The point in Figure 1 is the latent vector $z$ of different populations. Taking the population of simulated agents (e.g., the FCP population with 8 FCP agents) as an example, a point is generated by: 1) first, sample an agent in the FCP population; 2) use the agent to generate an episode with self-play; 3) use the VAE encoder to encode the episode into $z$ and map to the 2D space using t-SNE.

For each subfigure, we use the same VAE encoder so all points fall into the same latent space.

Meta-review: The reviews for this paper were split, with two reviewers on the borderline negative side, and two recommending acceptance. The main concerns expressed were insignificant / non-SOTA performance improvement and the 60-second human evaluation setup in Overcooked, which one reviewer argued is too short. (Even if prior work has used 60 seconds, the reviewer argued that the authors should provide their own rationale, which I tend to agree with.) On the other hand, another reviewer strongly championed the paper during the AC-reviewer discussion phase, praising the idea of learning a latent strategy space and arguing that the notion of performance in human-AI collaboration is multifaceted, since explainability, trustworthiness, satisfaction, etc., are all important factors. They also praised the evaluations as being comprehensive relative to other work in this domain. Given the variance in scores, I looked at this paper closely, and broadly agree with the most positive reviewer. This is interesting, mature work and I agree about the comprehensiveness of the evaluation. While the reviewers have pointed out valid minor issues, I don't see any major flaws that warrant blocking publication.
Decision: Accept (poster)