Learning to Cooperate with Humans using Generative Agents

Authors: yancheng  liang
daphne  chen
abhishek  gupta
simon s du
natasha  jaques
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:05:43

Abstract: Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL). Current algorithms focus on training simulated human partner policies which are then used to train a Cooperator agent. The simulated human is produced either through behavior cloning over a dataset of human cooperation behavior, or by using MARL to create a population of simulated agents. However, these approaches often struggle to produce a Cooperator that can coordinate well with real humans, since the simulated humans fail to cover the diverse strategies and styles employed by people in the real world.  We show \emph{learning a generative model of human partners} can effectively address this issue. Our model learns a latent variable representation of the human that can be regarded as encoding the human's unique strategy, intention, experience, or style. This generative model can be flexibly trained from any (human or neural policy) agent interaction data. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents. We evaluate our method---Generative Agent Modeling for Multi-agent Adaptation (GAMMA)---on Overcooked, a challenging cooperative cooking game that has become a standard benchmark for zero-shot coordination. We conduct an evaluation with real human teammates, and the results show that GAMMA consistently improves performance, whether the generative model is trained on simulated populations or human datasets. Further, we propose a method for posterior sampling from the generative model that is biased towards the human data, enabling us to efficiently improve performance with only a small amount of expensive human interaction data.

Main paper: 
 introduction: 
 Producing agents that can cooperate well with unseen partners such as humans [28] is an important problem for a variety of multi-agent systems across domains like robotics or software agents. While being cooperative is a notable characteristic of human intelligence [7, 32] , training an artificial agent that cooperates well with humans poses a significant challenge. Human behaviors are uncertain and diverse, encompassing a wide range of preferences, abilities, and intentions. While humans can rapidly adapt to different partners, AI agents are particularly poor at generalizing to working with a novel human partner. Solving this issue of distribution shift between the human player's strategy and those seen during the training of an AI agent is crucial to achieving human-AI cooperation.It is tempting to tackle this problem of learning to cooperate with humans using only data of humanhuman interactions [1] . While this may certainly provide some leverage, in many situations the amount of human-human data available is far less than the amount of data required by data-hungry sequential decision making algorithms. This suggests that we need a way to generate synthetic data at scale, while still providing relevance to the problem of coordinating with new, real human teammates on deployment. On the opposite end is the paradigm of self-play, that generates entirely synthetic data by iteratively training agents against copies of themselves. Self-play, while initially envisioned as a paradigm for learning how to compete with novel human players [27, 33] , has been applied to cooperative multi-agent reinforcement learning to find joint strategies for a team of agents when all agents are trained together [10, 23] . Self-play is an appealing approach for learning cooperative strategies, since it can be done in simulation, and does not require the expensive and time-consuming process of collecting and training on human cooperation data [1] . However, agents trained with self-play may fail to coordinate effectively with novel humans [1, 29] . During self-play, the agent learns to form a convention with a copy of itself and relies on that convention to achieve seamless cooperation [6] . When a human adopts a different strategy at test time, the agent fails to adapt, and merely continues to follow the convention formed in training. Such scenarios are fairly common, as self-play often generates "non-human" conventions [1, 11] . This points to the dual problems of 1) human-only data being expensive, and 2) synthetic-only data lacking coverage over human behaviors.To tackle these challenges, a popular approach to solving the distribution shift problem in human-AI cooperation has become training a Cooperator agent against a population of simulated agents rather than just itself [2, 15, 18, 19, 24, 29, 37, 40] . The goal of this work is often to create a diverse enough population of agents to cover the strategy space used by humans [2, 24, 37] . However, the bottom line remains that truly covering the space of strategies with discrete samples of strategies can quickly become untenable. As we move towards more complex real-world tasks, with a myriad of possible strategies, representing every strategy by an individual agent in the population and coordinating with them becomes computationally difficult. Indeed, we empirically compare the two lines of work on using simulated versus human data for zero-shot coordination, and find that contrary to past results, using human data provides better performance, especially for more complex environments.The key insight we will make in this work is that generative models offer a solution to the dual problems mentioned above. By training a generative model that can simulate cooperation partners, we can easily incorporate both human and simulated data. The resulting model can generate a diverse range of partner strategies, by interpolating between the policies it has been trained on, as well as composing strategies to create novel partners. Therefore, we propose to train a generative model of partner behavior using a variational autoencoder (VAE) [12] on a collection of coordination trajectories, either human or synthetically generated. The inference model of the VAE can be used to infer the latent variable z of a partner from its interaction data, encoding information about that partner's unique style or skill levels. The decoder can be used to generate the corresponding actions taken by the partner. Because the generative model can be trained on any combination of synthetic or human data, it can overcome the challenges of data scale and coverage to train adaptive Cooperators that much better cover the space of human behavior (see Figure 1 ).Given the ability to generate this diverse distribution of partner policies, a single adaptive Cooperator can be trained to adapt to a range of partners, by sampling a novel partner strategy from the generative model at each episode during training. We call our full method GAMMA: Generative Agent Modeling for Multi-agent Adaptation. While the interpolation properties of the generative model enable GAMMA to train more robust Cooperator agents, the availability of a controllable latent encoder also allows our desired adaptation behavior to be targeted to real human behavior. We propose a novel, ecnonomical way of incorporating a small amount of human-provided data into the sampling procedure of the generative model. This Human-Adaptive sampling method enables training Cooperators that are more targeted to partner with real human coordinators.We test our method using Overcooked [1] , a collaborative multiplayer cooking game requiring close coordination between two partners to successfully prepare recipes. In an evaluation in which trained agents play against novel humans in real-time, we find that GAMMA improves the performance of state-of-the-art coordination methods which rely on simulated populations of agents [24, 29] , or on human data [1] . We summarize our contributions below:• We present GAMMA, a novel approach for learning a generative model of partner strategies which can be trained on data from humans or a simulated population of agents. We sample from the generative model to simulate novel partners that are used to train a robust Cooperator agent. See Section 4.1 and Section 4.2.• We propose a data-efficient and human-adaptive sampling technique to steer the model to generate more human-like partners, even with limited amounts of human data. See Section 4.3. Figure 1 : We show the latent space covered by different methods. 2 For either simulated data or human data, the generative agents produced by GAMMA can cover a larger strategy space. Generative models can provide novel agents by interpolating the agents in the simulated population (a). On human data (b), the human proxy model only covers a subset of all human player behavior patterns, while the generative model can capture the diversity in the data. We can also control the latent space sampling (c) to model a target population of agents (e.g., human coordinators).• We conduct an empirical evaluation via a user study with real human players. We bring together and directly compare two lines of prior work on zero-shot coordination with humans: training on simulated population data and training on human data. We find that contrary to past published results, training on human data is highly effective in our human evaluation, especially for the new, more complex layout we propose. However, regardless of the data source, GAMMA consistently provides significant performance improvements over multiple lines of past work when evaluated with real humans. See Section 5. 
related work: 
 Training against simulated populations. Building cooperative agents that generalize well to humans and novel partners is a long-standing problem of AI, and is known as ad-hoc team-play [28] , or zero-shot coordination [11] . Recently, FCP [29] adopted the idea of using a population of policies to train a strong zero-shot Cooperator agent, as a diverse population effectively prevents the Cooperator from exploiting one specific convention. Following this framework, many techniques [2, 37] have been proposed to improve the diversity of the population, such as Maximum Entropy Population (MEP) [40] . Reward shaping [30, 37] and quality diversity [22, 31, 35] utilize domain knowledge to create agents with diverse behaviors. Statistical diversity based on trajectory distributions [19] or population entropy [40] are straightforward optimization objectives. However, as indicated in [2] , agents with different behavior distributions do not necessarily use different high-level strategies.To address this, the LIPO algorithm [2] instead tries to minimize the cross-play reward between different agents in the population to encourage them to form incompatible conventions. However, because LIPO agents can potentially sabotage the game, follow-up works [3, 24] propose strategies for preventing this, including Cross-play Optimized, Mixed-play Enforced Diversity (CoMeDi) [24] . While this extensive literature presents different population creation methods, in contrast we propose a novel approach to modeling this population with a generative model, and thus benefit the training process through the ability to flexibly sample unlimited new partners. Our experiments directly compare to FCP [29] , CoMeDi [24] , and MEP [40] , and show that GAMMA can enhance performance above each of these population-generation techniques, even using the same underlying population of simulated agents, while also allowing for the incorporation of human data.Training with human data. Another line of work has focused on collecting and training on human cooperation data [1] . Learning from human data is inherently challenging due to limited scale, arbitrary human behavior, subjective preferences, and relative speed of human actions compared to agents. Another challenge of coordination comes from the heterogeneity, uncertainty and suboptimality [8, 21] of humans. The agent might fail to coordinate well when it does not properly infer the preferences or intentions of humans. Our work takes a novel approach of combining these two lines Inferring the partner type. Our work is also related to the many works on multi-agent learning which attempt to learn a latent partner embedding to aid coordination. Some works use Bayesian inference [26, 36] or weight update [39] to decide the partner type from historical experience. Their methods require a manually-designed or predefined set of all possible partner policies. The idea of embedding-conditioned agent modeling is used in imitation learning [14] to learn more interpretable policies from human demonstrations. Grover et al. [5] train a latent variable model to learn agent representations from interaction data of different agents, and then infer a cooperation partner's latent vector and use it to condition a multi-agent RL policy. Papoudakis et al. [20] learn to predict the partner's representation from only the agent's own observation, avoiding accessing the global states during execution time. The idea of partner modeling has also been extended to offline RL [9] to train a zero-shot RL agent from a dataset. It is also known that generative models can learn diverse cooperative strategies from human-human cooperation demonstrations [34] . While our work also infers hidden context about partner type, our aims are different; we use a generative model to simulate novel partners during training time as a way to train a Cooperator to be robust for zero-shot coordination with real humans. 
problem formulation: 
 A multi-agent system is typically modeled as a Markov game [17] . In this work, we focus on two-player Markov games, but the formulation and our methods can be easily extended to more agents. The state space is S. At each step, two agents execute their actions a t ∈ A, b t ∈ B from their policies π : S → ∆(A), µ : S → ∆(B). They then receive a shared reward r t : S × A × B → R.The next state is determined by a transition function T : S × A × B → ∆(S). The value function V (π, µ) of two policies is defined as the expectation of discounted cumulative reward T t=0 γ t r t . Under this formulation of Markov games, the learning objective for the fully cooperative joint policy max π,µ V (π, µ) can be perfectly defined. However, it is unclear how to describe the learning objective of human-AI cooperation under the framework of Markov games.Inspired by the latent Markov Decision Process [13] , we model the problem of cooperating with humans as an MDP with latent variable z. We assume the human policy is conditioned on a latent variable z ∈ Z. This latent variable z can be regarded as a representation of the partner's unique style, skill level, reaction speed, etc. Then the human population can be defined as a distribution D(Z) over the latent space. Given the human policy µ : S × Z → ∆(B) conditioning on the latent variable z, the conditional transition function T z (s ′ | s, a) = b∈B µ z (b | s)T (s ′ | s,z = {T z , r z } is V z (π) = V (π, µ z ), the learning objective can be defined as π * ∈ arg max π E z∼D(Z) [V z (π)] . 
gamma: generative agent modeling for multi-agent adaptation: 
 We will leverage generative models as our key tool for strategy diversification, to overcome the dual challenges of 1) lack of real human data and 2) the difficulty of synthetically covering the large strategy space of human partners. In being able to generate diverse strategy profiles beyond the data, these generative models can be used to train a Cooperator that can be deployed to coordinate with novel users, each with their own diverse styles, preferences and capabilities. Here, we first describe our procedure for learning a generative model from training data of discrete agents (or human-human data), and then describe how this can be used for targeted coordination with real human partners. We call our approach Generative Agent Modeling for Multi-agent Adaptation (GAMMA).The first step of GAMMA is learning the generative model from pre-existing coordination data. We assume access to a dataset of trajectories D coordination = {τ i } N i=1 , where each trajectory τ = {(s t , a t , b t )} T t=0 is a sequence of multi-agent coordination behaviors. This dataset can be derived from human playing records. Alternatively, if there is a population of simulated agents available, this dataset can be collected by pairing them together to generate joint trajectories. For instance, in our experimental evaluation, we use agents generated through techniques like fictitious co-play (FCP) [29] to generate this dataset. The purpose of generative modeling here is to be able to model the multimodal marginal distribution over various strategy profiles in D coordination , a challenging distribution to model with standard maximum likelihood methods. Notably, the generative model is able to sample a landscape of agents that can be used to train a Cooperator, going beyond the quantity and diversity of the training data.We develop a variant of the Variational Autoencoder (VAE) [12] to model the diverse strategies underlying the training data D. As is typical in variational inference frameworks, we propose to learn an encoder-decoder generative model with an approximate posterior (encoder) q(z | τ ; ϕ) that identifies the agent style from the trajectory. The decoder in this model, p(a t | z, τ 0..t-1 ; θ) uses the agent's own past experience τ 0..t-1 = (s 0 , a 0 , ..., s t-1 , a t-1 , s t ) and the latent variable z to predict the agent's next action. This structure of the variational inference model can be thought of as multi-modal behavior cloning, where the encoder history provides the latent variable required to model the distribution of generated actions. The encoder-decoder architecture described above, and shown in Figure 2 can be trained using an evidence lower bound (ELBO) loss:EQUATIONImportantly, this generative model is able to generate behaviors that go far beyond the training data, both in quantity and diversity, as is shown in Figure 1 . We show in Fig 1 that while simulated behavior may not cover the space by itself, the generative model has significantly greater coverage over the strategy space. Importantly, we hypothesize that human behavior is more likely to lie within the span of strategies generated by the generative model. In this way, the interpolation and generalization of the generative model (even if it is not perfect) provides the expansion of the data required to effectively coordinate with humans.Once a generative model has been obtained, it can be used to train a robust Cooperator agent. This is accomplished by treating the generative model as a partner generator, using it to simulate partner behavior that covers the space of real human behaviors, and training the Cooperator to optimize performance with these partners in simulation. In particular, the generative agent model p(a t | z, τ 0..t-1 ; θ) can now be used as a generator of partner policies µ z to train our Cooperator agent. As shown in Figure 2 , for every episode we can sample latent variables from the prior z ∼ p(z), and use the conditioned action distribution p(a t | z, τ 0..t-1 ; θ) as a partner µ z for that episode. We aim to leverage these sampled partners from the generative model to train a single Cooperator π C , treating the sampled partners µ z as part of the environment. At each iteration, a batch of agents {µ zi } N i=1 are generated using latent variable samples z i ∼ D z . Then a batch of MDPs {M zi } N i=1 can be derived from these training partner policies to learn our Cooperator policy π. Then we use PPO [25] to optimize π C over these training MDPs:EQUATIONImportantly, the Cooperator π C is not trained with imitation learning, but is rather trained with reinforcement learning to learn task-specific coordination behavior. The generative model from Section 4.1 is simply used to provide the quantity and diversity of agents necessary for meaningful generalization to real human behavior.While the Cooperator described in Section 4.2 is trained by sampling partner agents from the generative model as z ∼ p(z), a t ∼ p(a t | z, τ 0..t-1 ; θ), this does not make use of human specific data if available, instead simply treating any human and synthetic data as equivalent. However, when coordinating with real human partners, it is useful to target model adaptation to be human-specific rather than to span the entire space of potentially irrelevant synthetic strategies (as shown in Figure 1 ). The key insight we will leverage to do so is that the latent space afforded by our generative model provides the ability to do controllable sampling from any latent distribution. This distribution can be chosen to incorporate additional information about the desired population. In particular, when coordinating with real human partners, the latent prior distribution for sampling p(z) can be replaced with a human-centric one p h (z), which is more focused on the part of the latent space relevant to human behavior. This suggests that given a small amount of human data D h , a human-centered latent distribution p h (z) can be quickly inferred by encoding the human data and estimating the latent Gaussian distribution that best explains encoded human data. The latent Gaussian mean is given by:EQUATIONGiven this human-centered latent distribution p h (z) = N (z, I), a targeted Cooperator that is meant for a particular target population can be trained by maximizingJ(π C ) = E z∼p h (z) [V (π, µ z )] .Human-adaptive sampling makes the model focus less on adaptation to irrelevant synthetic partners and more on "human-centric" partners sampled from the generative model. As our experiments will show, this approach outperforms training a partner simulator using only human data, since with limited human data it is easy for the model to go out of distribution during generation and it can thus be brittle. In contrast, GAMMA is able to train a robust partner simulator using large amounts of synthetic data. Adding human-adaptive sampling to GAMMA can provide considerable generalization benefits even with modest amounts of data. Note that to better capture human data, we do not just target the latent space, but also perform some fine-tuning on the encoder and decoder of the VAE using human data. 
experiments: 
 We evaluate GAMMA using the Overcooked environment [1] as a popular benchmark for prior work on human-AI cooperation [1, 24, 29, 36, 37] . In Overcooked, two players need to cooperate to divide the work of cooking and avoid getting in each other's way. This involves anticipating the intended goal and actions of the other player, and inferring which task would most usefully assist them. The layouts proposed in previous work are shown as the first five environments in Figure 3 . The Counter Circuit layout poses a hard coordination challenge with multiple strategies such as passing items across the countertop to the partner or moving clockwise/counterclockwise around the ring. Additionally, we include a custom layout based on Counter Circuit, termed Multi-strategy Counter. This new layout involves recipes with multiple ingredients, significantly increasing the complexity of the strategy space. Failing to infer the intentions of the partners and adding the wrong ingredients to the pot could ruin the entire dish, which increases the importance of coordination.Our experiments investigate the following questions:H1: Using simulated agents. Will training a Cooperator against a generative model of partner strategies trained on a population of simulated agents outperform training the Cooperator against the simulated agents directly? H2: Using real human data. Can the generative model be used to effectively leverage (small amounts of) human data, and also combine it with simulated agents? H3: State-of-the-art. Can we obtain better performance than competitive baselines using both simulated and human data?Learning from a Simulated Agent Population. We first investigate H1, and test whether generative agents can improve the performance of the Cooperator agent when training with simulated agent partners. We include three state-of-the-art approaches for creating a population of simulated agents. Learning from Human Data. To investigate H2, we test whether the generative model can be used to model human behaviors from data produced by real human players. In this work, we assume the human dataset contains 20 to 50 trajectories, reflecting the popular open-source dataset provided by Carroll et al. [1] . We use PPO-BC [1] as the baseline where a BC model over human data is used as the partner during training. We test whether replacing the BC agent in this framework with our generative model trained on the human data (PPO-BC-GAMMA) provides better performance. Since we assume the amount of human data is limited, we also fine-tune a generative model pretrained from the simulated agent population with human data with both decoder-only (DFT) and full fine-tuning (DFT), as described in Section 4.3. We apply our Human-Adaptive (HA) sampling method to the fine-tuned generative model to sample from the human-centered latent distribution p h (z).Simulated Agent Evaluation. Following prior work [1] , we create an automatic evaluation mechanism by using held-out human data to train a behavior cloning policy as a human proxy agent, and report the performance of the Cooperator when it is paired with this test human proxy agent. However, we note that methods which are explicitly trained against a human-proxy agent (PPO-BC) can easily exploit this metric in a way that is not indicative of actual performance with real humans, so we do not use this automatic evaluation to assess those methods. Instead, we conduct evaluations with real human players as the gold standard evaluation technique.Human Evaluation. We run a user study with real human players in order to determine which method can most effectively coordinate with humans, and which method is rated as subjectively better by humans (H3). We conducted a study with 80 users recruited via online crowdsourcing from Prolific.Our study follows guidelines set by a UW IRB protocol. During the study, each user is instructed to play multiple rounds of Overcooked with a partner via a web interface, where in each round the partner is an agent following one of the 9 policies, in randomized order. We trained 5 random seeds per agent, and used a different randomly-selected seed for each of the 9P P game rounds. For the human study, we focus on the most complex layouts, Counter Circuit and Multi-strategy Counter.Each game lasts for 60 seconds. After each round, the user answers Likert scale survey questions [16] to rate their experience playing with the agent. At the end of the 8 rounds, humans also answer qualitative questions about the performance of the agents. Users are required to pass an attention check to ensure quality of their data. Using these answers, we conduct a qualitative analysis to understand which factors most heavily influence overall performance and users' preferences when playing with real humans.6 Results 
evaluation in simulation: 
 In this section, we present the evaluation results against a human proxy (behavior cloning) agent.H1: Using Simulated Data. We train the generative models FCP+GAMMA, CoMeDi+GAMMA and MEP+GAMMA on the simulated agent population of FCP, CoMeDi and MEP, respectively. Figure 4 shows the learning curves of each method averaged on all layouts, and the final performance of different methods for each of the different layouts. The Cooperator agent is evaluated by the zero-shot cooperation performance with a human proxy BC model. GAMMA consistently improves performance over the baselines, across layouts. This demonstrates that GAMMA provides a more efficient way to utilize the simulated agents by providing a landscape of partners to train the Cooperator.We also find the improvement gap increases as the layouts become more complex.As described in Section 5, we conduct an evaluation with real human players, recruiting new participants that were not in the training data. We evaluate all agents against the novel human players, and plot the cooperative scores achieved by each agent-human team in Table 3 .Training with Simulated Data.We find that GAMMA offers significant improvements over prior techniques for training against simulated populations (FCP, CoMeDi, MEP). Although the trend of these results is consistent with the previous results for H1, we find that here GAMMA provides significantly enhanced performance improvements when tested with real humans, reaching the new state-of-the-art performance for both Counter Circuit and Multi-Strategy Counter.On our newly proposed, more complex layout, Multi-Strategy Counter, we find that the CoMeDi baseline performs so poorly that it cannot discover strategies which make use of the new tomato ingredient. The reason is because playing the onion-soup strategy and the tomato-soup strategy together can recreate an unrecoverable game state, which is not favored by the CoMeDi algorithm to include both types of agents in the population. Therefore, when we use GAMMA to train a generative partner model using data generated from the CoMeDi population, it also fails to learn any strategies involving tomatoes, and both models generalize poorly to playing with humans, although CoMeDi+GAMMA still offers a performance benefit over CoMeDi. This points to the fact that GAMMA, which is based on training a generative model on cooperation data, can fail to perform well if the cooperation data is not sufficiently diverse. This problem can be aptly described by the well-known adage "garbage in, garbage out".However, when GAMMA is trained using the high-quality population found by MEP, we see that it performs extremely well, reaching a score of 88 on Multi-Strategy COunter, while MEP, the most competitive simulated population baseline, only achieved 64. This represents a 38% improvement.Comparing methods that make use of human data reveals some interesting findings. First, our results directly compare two lines of prior research: training on simulated populations vs. training against a BC model trained with a limited amount of human data (PPO-BC). While previous works show that using only simulated data can exceed PPO-BC and reach state-of-the-art performance [29] , we find that on the more complex layout, PPO-BC is actually the best performing baseline. Modeling Whether training with simulated or human data, GAMMA shows consistent, statistically significant advantages over the baselines. GAMMA-HA is able to efficiently use the real human dataset to learn a better sampling of its latent space, achieving the best performance when cooperating with real humans.the human data with the generative model (PPO+BC+GAMMA) only provides performance improvements half the time. However, combining simulated and human data with Human Adaptive GAMMA provides significantly higher performance in both layouts, surpassing state-of-the-art zero-shot coordination techniques.As revealed in Figure 5 , GAMMA HA achieves the highest performance in both layouts, surpassing the most competitive baselines by 60% and 43% in Counter Circuit and Multi-Strategy Counter, respectively (See Table 3 ). We conducted a statistical analysis of these results using Holm-Bonferroni correction, which can be found in Table 4 .As demonstrated in prior work [1] , humans can adapt to deficiencies in the policies of AI agents and narrow the apparent performance gap between different agents by simply completing the task themselves. This means cooperation performance alone cannot measure whether a particular agent is frustrating or cumbersome for the human to cooperate with. Therefore, we also collect subjective ratings of the agents from the human participants. The results for the question about overall cooperation are plotted in Figure 6 ; additional results are available in the Appendix, which pertain to agent's ability to adapt (Fig. 9 ), and whether it was human-like (Fig. 10 ) or frustrating (Fig. 11 ).For the overall ratings in Figure 6 ,the subjective ratings mirror the cooperative performance scores: when GAMMA is trained with the same data available to a baseline technique it improves performance in terms of the human ratings, and GAMMA HA gives consistently good performance on both layouts, in the sense that it receives a higher proportion of positive human ratings. We note that while PPO+BC+GAMMA did not show a performance benefit over PPO+BC on Multi-strategy Counter, human ratings of PPO+BC+GAMMA were more positive. One exception to the previous trends is that CoMeDi obtains high subjective ratings in Counter Circuit, although it performs poorly in Multi-strategy Counter. From the additional figures in the appendix, we can observe that GAMMA methods are rated as more adaptive, human-like, and less frustrating than other techniques.Analyzing the qualitative results reported by participants in the human study, we find that adaptation to the human partner was a core theme distinguishing agents that humans liked. Participants reported that the FCP + GAMMA agent demonstrated an ability to learn from the user's actions, such as mimicking the user's strategy of placing onions on the table to save time. This adaptive behavior was positively received by a study participant: "I noticed that once I started to put back onions on the table that it did the same as I wanted to save time rather than going back for onions 3 times for soup. I thought it was interesting that it learned about my behavior." Because GAMMA models have been trained over a more diverse range of partners, they consequently exhibit the ability to better adapt to real humans during gameplay. "Overall, I felt that the agent's ability to coordinate with me was: {Very poor, Poor, Neutral, Good, Very good}". FCP + GAMMA, PPO + BC + GAMMA, and GAMMA-HA consistently receive higher ratings for ability to coordinate compared to their respective baselines.Another common theme that emerged from the qualitative analysis was consistency and predictability. Erratic behavior was a common observation regarding the baseline methods. Users observed the AI performing random actions, such as moving onions around without an evident purpose or failing to complete necessary steps in the cooking process. In contrast, users report that agents using GAMMA behave in a logical, consistent manner. For example, for FCP + GAMMA, participants provided the following feedback that the agent was "more deliberate in its actions" and "its actions were logical".In contrast users reported that the baseline FCP agent, "didn't behave logically" and "the agent this time was inconsistent and did not help me with any of my orders at all." We hypothesize the reason for this "inconsistency" that occurred in baseline methods may be due to the human's behavior going out-of-distribution (OOD) of their training data, causing the resulting policy to make errors and behave in an unpredictable way. This points to the importance of obtaining better coverage of the human data distribution, as provided by GAMMA (see Figure 1 ). 
conclusion: 
 In this work, we propose GAMMA, a novel approach to training a coordinator agent by using generative models to produce training partner agents. We conduct a comprehensive analysis using data from a study with real human cooperation partners, and show GAMMA outperforms baselines over both subjective human ratings and quantitative measurements of cooperation performance. We also provide a new perspective to compare different populations under the latent space of a generative model, showing how the simulated populations may not provide sufficient coverage of the range of human players.Limitations. As shown by the performance of GAMMA+CoMeDi on Multi-Strategy Counter, obtaining good performance with our approach depends on having a reasonably diverse amount of cooperation data to train the model. If the quality of the simulated population data is too low, the approach can fail to provide significant benefits.In this work, our human studies recruit participants from Prolific, which may not be representative of broader populations. Additionally, our human dataset is limited, which could reduce the diversity of strategies and force participants to adapt to strategies that the Cooperators are already familiar with.We focus on the two-player setting in this study following prior work [29, 37, 40] because it is a first step toward enabling an AI assistant that could help a human with a particular task. Scaling up to more agents would exponentially increase the dataset size with our current techniques. Therefore, better sampling techniques are needed to address this issue.Future work. Several potential directions are interesting for future work: 1) In this work, the amount of human data is limited, which restricts the performance of the generative model that learns human data from scratch. 2) An orthogonal direction is to condition the policy of the Cooperator on the embedding of the partner policy. We provide some preliminary results in Figure 15 .Social impact. Our work focuses on how to train AI agents that can effectively cooperate with diverse humans to assist them with tasks. We believe this is a critical component of eventually enabling assistive robots that could operate in human environments to assist the elderly or disabled to live more comfortably, or reduce the burden of domestic labour for all people. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: we discuss the societal impacts of our work in Section ??.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: all the models released by this work are used for experimental game simulators only and do not have the risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: we use open-source assets of Overcooked from [1] and open-source code for baselines under MIT/GPL license. No other assets are used in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Our demo website is https://sites.google.com/view/human-ai-gamma-2024/ and contains the code and more experiment results. We also provide information about the implementation details B and hyperparameters used in our experiments D to help reproduce our results.Generative models. The dataset used to train the VAE model contains the joint trajectories of two players. For the simulated agent population {π 1 , ..., π N }, we create this dataset by evenly sampling π i × π j to generate the trajectories. A simulated dataset contains 100K joint trajectories. To train a VAE on it, the dataset is split into a training dataset with 70% data and a validation dataset with the rest of 30% data. To compute the ELBO loss 1, the trajectories are truncated to length 100 for better optimization for the recurrent module. A linear scheduling of the KL penalty coefficient β is adopted to control a target value for the KL divergence of the posterior distribution. The target KL value is set to 7 for the layout Forced Coordination over the CoMeDi population. All other VAE models are chosen by a target KL value of 32.Train Cooperator agents. On Overcooked, the Cooperator is trained by PPO [25] . We based our implementations on HSP [37] . Reward shaping for dish and soup pick-up is used for the first 100M steps to encourage exploration. All results are reported with averaged episode reward and the standard error of at least 5 seeds.Simulated agent populations. We use MAPPO [38] to create the FCP [29] agent populations. Eight self-play agents are trained and three checkpoints for each agent are added to the population, making the population size 8 × 3. For the CoMeDi agent population [24] , we download the population proposed by the authors 3 for the original five layouts [1] . The population size is 8 for CoMeDi. For our custom layout Multi-strategy Counter, we use CoMeDi's official implementation and keep the population size the same.For the human dataset in the original Overcooked paper [1] , their open-sourced dataset contains " We use MAPPO to train our Cooperator agent. The architectures and hyperparamers are fixed throughout all layouts. All policy networks follow the same structure where an RNN (we use GRU) is followed by a CNN.The generative model follows a similar architecture to the policy model. An encoder head and a decoder head are used to produce variational posterior and action reconstruction predictions from the representations.We Since diversity of humans is a key component in our approach, in this section we report the demographics of our participants. The study demographics include a population of 54% female and 46% male participants. The user demographics skewed towards younger ages, with 39% of participants between ages 18 -26, 44% of participants between 27 -37, 11% of participants between 38 -48, and 6% of participants ages 49 and above. The game experiences of the participants are shown in Figure 7 . In our evaluation with real humans, each user can change their strategy at any point in the game. A significant proportion of our users self-identify as novice players, both for video games and for Overcooked (17.9% and 49.5%, respectively, Figure 8 , thus often exhibiting improved performance over the course of an increased number of trials. Figure 8 provides evidence of this pattern, demonstrating that humans change their gameplay style over the course of the evaluation.Training data source 3 : Human evaluation results. Our methods (GAMMA) show significant improvements.We conducted statistical significance tests and computed the p-value using the Holm-Bonferroni correction. See Table 4 We include additional analyses of users self-reported responses for qualitative questions from our user study. Figure 9 shows the results for users' response to the agents' ability to adapt. The results indicate that two of our methods, FCP + GAMMA and GAMMA-HA-DFT, consistently receive higher ratings indicating better ability to adapt than their respective baseline agents, across both layouts. Figure 10 displays users' ratings for whether the agents demonstrated human-like behavior. The responses show that FCP + GAMMA, CoMeDi + GAMMA, and GAMMA-HA-DFT exhibit the most human-like gameplay in both layouts. Figure 11 includes responses for whether the agents' behavior was frustrating. Individuals consistently reported that FCP + GAMMA and GAMMA-HA-DFT demonstrated the least frustrating behavior. Furthermore, we provide additional participant feedback for our agents from the user study as follows, starting with feedback for agents using our methods: Feedback for FCP + GAMMA:• "It was very coordinated."• "This agent figured things out the fastest and worked with me well."• "When it was in my way, it moved out of the way instead of blocking the path, which was an issue with other agents." • "Much better cooperation by comparison." Feedback for CoMeDi + GAMMA:• "He was the best teammate."• "This one did well, it moved around me enough to complete tasks and we were relatively efficient passing items around." • "The agent figured out the next step I would have done." Feedback for PPO + BC + GAMMA:• "It was very efficient and placed things down so I could grab it on the other side."• "This agent was very collaborative. All I did was put onions out and he did the rest of the work -super smooth." • "It behaved smoothly and intentionally. It responded to my moves and helped rather than getting in the way."In contrast, the feedback for the baseline agents is overall more negative, including themes such as frustration, inefficiency, and lack of cooperation.Feedback for FCP (baseline):• "Frustrating."• "He got in the way and didn't help at all."• "Just seemed to lack coordination and felt less intelligent."Feedback for CoMeDi (baseline):• "The agent was helpful, but still did not fully cooperate as well as I thought it should have."• "The agent was executing some random move actions before actually cooperating."Feedback for PPO + BC (baseline):• "The agent was not helping much, it was doing its own thing, which meant we were not coordinating well." • "Agent did everything by himself; didn't see the onions on the counter that I had placed; got in my way a bunch." • "Very inefficient and lacking intelligence." "The agent's behavior was frustrating: {Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree}". FCP + GAMMA and GAMMA-HA consistently receive ratings for less frustrating behavior compared to their respective baselines. Figure 12 provides the original data for the performance on simulated data.In addition to training the generative model on the human dataset (PPO-BC-GAMMA, we can also leverage Human Adaptive (HA) sampling on the generative model: (GAMMA-HA). The learning curves of these methods and the baseline PPO-BC [1] are shown in Figure 13 . When evaluated with a held-out human proxy agent, our methods do not show a great advantage. This is expected, because PPO-BC is trained to exploit a human proxy agent. However, in the human study, we show that this hurts its adaptation to more diverse real human players. On the other hand, we note that GAMMA-HA shows much faster learning in both layouts. Figure 13b shows that GAMMA-HA also adapts better to held-out self-play agents compared to PPO-BC.I Compare decoder-only fine-tuning and full fine-tuning I.1 Full fine-tuning can suffer from insufficient human dataIn some early experiments, we find that only fine-tuning the decoder with human data (GAMMA-HA-DFT) provides consistently strong performance on both layouts, whereas full fine-tuning (GAMMA-HA-FFT) provides the strongest performance on the first layout, but weaker performance on the second layout. At first, we hypothesize this is because the second layout is more complex, but the number of human coordination trajectories available for training (N = 11) is significantly less than the first layout (N = 37). With more diverse potential strategies and less human data, we find the data is insufficient to fully fine-tune the generative model for the second layout. Since the entire model is fully fine-tuned, there is a higher chance that the model overfits the training human samples when the amount of human data is extremely small. Therefore, we believe the results of the FFT model could be improved were we to collect more data. However, it is realistic to test the scenario where limited human data is available, since human data can be quite expensive and difficult to collect. We find that in this low data regime, GAMMA HA DFT still provides excellent performance. Whether to fine-tune the encoder is a design choice that can be tuned for a particular domain based on the available data and the performance against simulated agents (since Figure 13 shows that FFT performed poorly here as well). 5 : Human evaluation results (outdated). Our methods (GAMMA) show significant improvements. However, GAMMA HA FFT is not stable on "Multi-strategy Counter".Given the hypothesis that the human dataset is too small compared to the complexity of the Multistrategy Counter environment, we find out that with a larger regularization over the fine-tuned model on the original model, we can mitigate this issue. See Figure 14 .One potential future work to improve the efficiency of online adaptation is to condition the policy on z. During testing with novel humans, the Cooperator can then infer the z of the human player and adapt to that z. This method is orthogonal to our contributions where we focus on sampling partners with different z to train the Cooperator. We did some preliminary work to test the z-conditioned Cooperator. As shown in Figure 15 , the performance of the z-conditioned Cooperator is not stable, it learns faster but the performance decreases when it is trained longer. Therefore, we do not include this method in our main experiment, but future work about how to better train the zconditioned Cooperator is promising.Answer: [Yes] Justification: the code will be released on our demo website with detailed instructions (see Section A). Guidelines: Justification: the compute resource details are discussed in Section E.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: all authors have reviewed the code of ethics and will follow it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).See our website for human-AI study videos and an interactive demo. The training code is also available. 38th Conference on Neural Information Processing Systems (NeurIPS2024).Points are the latent vectors. Taking the population of simulated agents (e.g., the FCP population with 8 FCP agents) as an example, a point is generated by: 1) sampling an agent in the FCP population; 2) using the agent to generate an episode with self-play;3) using the VAE encoder to encode the episode into a latent vector and mapping the latent vector to the 2D space using t-SNE.https://github.com/Stanford-ILIAD/Diverse-Conventions/tree/master 


Decision: The reviews for this paper were split, with two reviewers on the borderline negative side, and two recommending acceptance. The main concerns expressed were insignificant / non-SOTA performance improvement and the 60-second human evaluation setup in Overcooked, which one reviewer argued is too short. (Even if prior work has used 60 seconds, the reviewer argued that the authors should provide their own rationale, which I tend to agree with.) On the other hand, another reviewer strongly championed the paper during the AC-reviewer discussion phase, praising the idea of learning a latent strategy space and arguing that the notion of performance in human-AI collaboration is multifaceted, since explainability, trustworthiness, satisfaction, etc., are all important factors. They also praised the evaluations as being comprehensive relative to other work in this domain. Given the variance in scores, I looked at this paper closely, and broadly agree with the most positive reviewer. This is interesting, mature work and I agree about the comprehensiveness of the evaluation. While the reviewers have pointed out valid minor issues, I don't see any major flaws that warrant blocking publication.

Reviews: review: paper_summary: paper_summary: The authors propose a novel method for training agents that effectively cooperate with humans by leveraging generative models that can sample the space of agent behaviors
strength_weakness: strength_weakness: - Experimental results are okay, but not clear that GAMMA consistently outperforms baselines. This is true in both the simpler and more complex settings.  Even in cases where GAMMA-based methods have higher performance, the confidence intervals seem to overlap making it difficult to say whether results are simply due to chance. See Questions below for a few more things I'm wondering about regarding results and significance. Happy to raise my score if authors can show that the proposed method fairly consistently outperforms baselines in a statistically significant way (either by correcting me if I'm wrong in my interpretation of current results, or by increasing sample size to decrease confidence intervals). 

Minor suggestions (did not affect score):
 - increase text size in fig 2
 - report results in a table (can go in the appendix) as it is hard to tell from the graphs alone whether certain methods outperform others in statistically significant ways
questions: questions: - how well would this approach generalize to more realistic settings? It seems like moving to the more complex setting led to reduced gains compared to baselines. 
- relatedly, how complex are the strategies learned by the Cooperator? Can they deal with humans who learn, adapt, and change their own strategies?
- can the Cooperator instead be meta-learned and adapt online?
- not entirely clear what is meant by "Error bars [4] use the Standard Error of the Mean (SE) for statistical significance (p < 0.05)". Are you saying that error bars correspond to 95% CI? or that they correspond to standard error? 
- following up on that, which of the results are statistically significant? hard to tell from the figures.
limitations: limitations: - Authors note human diversity is critical, but it's somewhat unclear whether human data comes only from WEIRD populations (almost certainly yes since participants were recruited from Prolific). This is worth noting in the human eval/limitations/impact sections since results may not be representative of broader populations (especially since authors mention assistance for elderly or disabled people as potential future application). A potential social impact of this is that a) agents may best serve only the demographics they were trained on, and b) agents may lead to reduction in creativity and diversity of strategies by forcing humans to adapt to the limited range of strategies that the agents are familiar with. 
- Authors do mention wanting to train on larger datasets in limitations, but also worth mentioning that current experiments are in very constrained environments and only in 2-player settings. In addition to dataset size, scaling along both of these dimensions would be important future work.
- If I understand correctly, the human data used for fine-tuning or for creating the generators is collected either in solo-play or interaction with a simple existing policy. It seems unlikely then, that the GAMMA-based methods could deal with humans learning, adapting, and changing their strategies. Perhaps worth mentioning this as well.
paper_summary: paper_summary: The paper addresses the challenge of training AI agents that can effectively coordinate with diverse human partners in cooperative tasks without prior interaction. This work demonstrates that using generative models to create diverse training partners can significantly improve an AI agent's ability to coordinate with novel human partners in cooperative tasks. The proposed approach offers a promising direction for developing more adaptable and effective AI assistants for human-AI collaboration scenarios.​​​​​​​​​​​​​​​​

The key contributions of the work are listed as follows:

1. The work proposes a novel approach, Generative Agent Modelling for Multi-agent Adaptation (GAMMA), that uses generative models to produce diverse partner agents for training a robust Cooperator agent. Through conducted experiments using the Overcooked environment, the authors demonstrate that GAMMA consistently outperforms baselines in zero-shot coordination with novel human partners, both in terms of task completion and subjective human ratings.

2. The authors show the flexible data integration of GAMMA, i.e. it can be trained on both simulated agent data and real human interaction data. This addresses the dual challenges of limited human data and insufficient coverage of human behaviours in synthetic data.

3. The authors also propose a method, Human-Adaptive Sampling, to efficiently incorporate a small amount of human data to bias the generative model towards producing more human-like partners during training.

4. The study brings together and directly compares two previously separate approaches to zero-shot coordination: training on simulated population data, and training on human data.
strength_weakness: strength_weakness: 1. *Results lack of statistical significance, thus persuasiveness*:
Despite all the good aspects of the work, including the motivation and the intuition, my biggest concern is that the improvement of the performance doesn't look significant. First, in the Cramped Room in Figure 4, there seems to be overfitting after $5e7$ steps. In the meantime, GAMMA doesn't lead to improvement on FCP at that time step. Similarly, there seems no (statistically sifnificant) difference between FCP and CoMeDi with and without GAMMA at the $5e7$-th step in Forced Coordination. Considering that the performance of both FCP and CoMeDi decrease after $5e7$ steps in Forced Coordination as well, a more rigorous conclusion could be that GAMMA can prevent the overfitting, instead of improve the performance. Secondly, the performance shown in Figure 5 also suggests that GAMMA doesn't improve the performance, since the orange curves (PPO + BC + GAMMA) twist with the blue curves (PPO + BC). Even though the methodology is convincing, the results make me doubt whether the proposed GAMMA can indeed lead to significant improvement.

2. *Need to illustrate the necessity of sampling from $z_i$*:
In Section 4.2, the authors illustrate the human behaviour representation $z_i$ is used to generate batches of MDPs. A straightforward and intuitive alternative practice is simply to make $z_i$ a part of conditionals for $\pi$. The authors need to provide either a rational in more details or more empirical evidence on the chosen implementation, i.e. using $z_i$ to generate $\\{M_{z_i} \\}_{i=1}^{N}$. 

3. *Length of games during evaluation*:
As an experienced player of Overcooked, the 60-second setup used in human evaluation looks too short to me. According to my own experience, for human-human cooperation, it usually takes a whole trial of the game, i.e. >120 seconds. This makes me doubt whether improvement by GAMMA is only on the adaptation efficiency. Moreover, this also makes me doubt whether the improvement observed in the evaluation is only due to some randomness. I would be keen in seeing more trials of each method with human partners in longer games.
questions: questions: 1. Why is "training on human data is highly effective in our human evaluation" counterintuitive?
The authors mention in line 95 that "contrary to past published results, training on human data is highly effective in our human evaluation". I got confused by this claim as it sounds intuitive to observe better performance by using human data in human evaluation.

2. Can increasing sampling temperature also lead to greater coverage over the strategy space?
In line 200, the authors claim that "the generative model has significantly greater coverage over the strategy space". A natural alternative could be simply increasing the sampling temperature. So, I wonder if this simple trick could also boost the performance/efficiency of the same baseline methods?

3. Is H3 a hypothesis?
From my perspective, the H3 stated from line 261 to line 263 is not a hypothesis.

4. Why is it useful to evaluate the method by a population of held-out self-play agents?
As stated in line 289, the human evaluation is completed on "a population of held-out self-play agents". After reading the paragraph from line 282 to line 289, I'm still unclear about the rationale of doing so. More importantly, line 95 state that human data is not so effective in human evaluation. Then, why is fine-tuning on human data a good evaluation?

5. How are the number of policies (8) and the numbers of random seeds (5) decided in human evaluation?
limitations: limitations: Please refer to the weaknesses section.
paper_summary: paper_summary: This paper proposes GAMMA, a generative model that learns a latent representing partner agents' strategy. Through interpolation, the latent can be used generate diverse partners to train ego agents for human-AI cooperation. Authors studied training the generative model with both simulated data and real-human data and conducted evaluation with both human proxies and real humans. Experiment results show GAMMA improves both objective performance and human subjective ratings.
strength_weakness: strength_weakness: 1. The 'state-of-the-art' baselines in this paper are outdated (especially FCP). I would be happy to see comparison against works like [1],[2],[3], which are also targeted at zero-shot coordination.

  [1] Mahlau, Yannik, Frederik Schubert, and Bodo Rosenhahn. "Mastering Zero-Shot Interactions in Cooperative and Competitive Simultaneous Games." arXiv preprint arXiv:2402.03136 (2024).

  [2] Yan, Xue, et al. "An Efficient End-to-End Training Approach for Zero-Shot Human-AI Coordination." Advances in Neural Information Processing Systems 36 (2024).

  [3] Zhao, Rui, et al. "Maximum entropy population-based training for zero-shot human-ai coordination." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023.

2. In Fig. 4,5, GAMMA's improvements over baselines seem marginal.

3. The interpolation visulization needs better explanation. What do the points in Figure 1 stand for? Is it policy $\pi$ or hidden vector $z$? I can't find the reletive information to comprehend this figure.

4. Adding to the last question, utilizing interpolation to enhance partner diversity is not new in zero-shot coordination field. PECAN [4] directly interpolates parnter policies through policy ensemble instead of hidden vector $z$ (see figure 4 in [4]). What is the advantage of GAMMA by interpolating at the level of hidden vectors? Does it offer better partner diversity? This should be at least discussed in the paper.

[4] Lou, Xingzhou, et al. "Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination." arXiv preprint arXiv:2301.06387 (2023).

5. Typos:
"data-hungry seuential decision making algorithms...", on page 1

"We propose a novel, ecnonomical way" on page 2

e.t.c.
questions: questions: See weaknesses.

Please at least add MEP [3]  to the baselines in your experiments (since it has very similar pipeline as FCP in your paper and easy to run). I would happily increase my score upon seeing such comparisons.

[3] Zhao, Rui, et al. "Maximum entropy population-based training for zero-shot human-ai coordination." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023.
limitations: limitations: Limiations are discussed in the appendix.
paper_summary: paper_summary: This work proposes a novel method to train agents to coordinate with humans in team tasks. The main contribution is that instead of training an RL cooperator policy based on a limited number of human models, the authors use a generative model to capture the latent variable representation of human policy. The learned continuous latent space is believed to cover the strategy space of a specific team task, therefore can be used to sample diverse human policies during agent training. Experimental results confirm the benefit of introducing generative models in comparison to previous methods using behavior cloning human models or population-based RL models.
The authors also conduct human evaluations to pair trained agents with novel human participants via crowd-sourcing. The team performance and perceived utilities are higher when human players are paired with agents trained with the proposed method.
strength_weakness: strength_weakness: 1. This work lacks a formal analysis about the constructed latent space of human policy given the training data set. There seems to be no theoretical guarantee or quantitative analysis about the generalizability of the proposed generative model over training data, as claimed in lines 198-203. If the collected trajectories in D are biased / do not cover the entire strategy space, the performance of the proposed method might be impacted. Provide more details about Fig. 1, or quantitative analysis of the generalization / interpolation properties would help the readers to value the proposed method.

2. Details about the human data used during training are missing. Are they independently collected from human teams with optimal / consistent team strategies? The setup is very important because the policy distribution of the human data would impact the learned generative models and human proxy models used during training. Explaining the data collection setups and comparing the nature of collected trajectories from agents / humans would enhance the paper.

3. Training a Cooperator policy via RL given a partner policy conditioned on a distribution of latent variables is not trivial. There seems to be a hidden assumption about the distribution parameters to make sure partner policies do not differ too much so the RL policy can converge. The paper does not report details about this process.

4. The hidden assumption of human-adaptive sampling is that human data are closer to the target partner policy compared to synthetic data, so we want to fine-tune the generative model and Cooperator policy on it. This might not be the case if human trajectories are noisy.

5. The authors mention type-based reasoning in the related work section but do not include them in the baselines [1]. It is worth discussing how the proposed method and type-based methods perform differently, given different human policy distributions.

6. One potential future direction is to extend the framework to be online adaptive to the target partner during interaction. For example, using a similar technique to infer the human-centered latent Gaussian distribution of a specific individual human, and change the agent action policy accordingly on the fly.

[1] Zhao, M., Simmons, R., & Admoni, H. (2022, October). Coordination with humans via strategy matching. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 9116-9123). IEEE.
questions: questions: 1. How are different policies sampled and visualized on the latent strategy space in Fig 1?

2. When sampling different latent variables of the partner at each episode, could PPO guarantee to learn an optimal cooperator policy?

3. Are trajectories in the human data set all from the same human-human team with a consistent strategy or a population of individual human players with different strategies randomly paired into teams? Are those human strategies optimal or not?

4. How is the held-out performed on human data and self-play agents respectively? Line 475 only mentions a 30%-70% split without mentioning if agent policy type is stratified.

5. What does this sentence mean in lines 286-288? If the held-out is properly performed on human data, e.g. removing trajectories independently collected from different human participants, PPO-BC has no chance of exploiting information about the unseen evaluation data set. This question might be related to the previous concerns about human data distribution and collection details.

6. How novel are the strategies in human evaluation compared to the original training human data?

7. How accurate is the BC model? What is the action prediction score?
limitations: limitations: In the supplementary materials, the authors acknowledge the limitation of using limited human data which prevents the generative model from learning from scratch. I actually see this as a strength of the paper since human data is usually expensive to collect in practice. While the limited amount of human data is acceptable, the limited characteristics of human data might hurt the further application of the proposed method. As discussed earlier, if the authors could consider examining the quality and features of the used data set, it would shed light on learning to collaborate with humans in ad-hoc teamwork.

There are a few inconsistent patterns within the experimental results (e.g. Figure 5 (a) and Figure 6 (b)) where the proposed method does not achieve better performance than the baselines. The authors attribute those failed cases to limitations of the training data collection process. This actually illustrates the limitations of the proposed method in solving the 2 main challenges claimed in the paper: 1) lack of real (high-quality) human data and 2) the difficulty of synthetically covering the large strategy space of human partners.

Comments: title: statistical significance test with correction
Exact p-values.
Exact p-values
Concern about statistical significance
Response to rebuttal
comment: We thank you again for your response to our rebuttal. We provide the p-values with Holm-Bonferroni correction below, with the aim of reducing the family-wise error rate. We also include an additional comparison of PPO + BC + GAMMA against PPO + BC as requested. The p-value is compared to 0.0028 as there are n=18 comparisons, thus p=0.05/18=0.0028. Please let us know if you have any additional comments or clarifications. 

### Comparison of best-performing GAMMA against baselines with Holm-Bonferroni correction:
| **Comparison**            | **Counter Circuit** | **Multi Strategy Counter Circuit** |
|---------------------------|-------------------------------|----------------------------------------------|
| **GAMMA HA DFT/FFT > FCP**      | **1.09e-57**          | **7.58e-71**       |
| **GAMMA HA DFT/FFT > CoMeDi**| **1.23e-06**          |  **2.18e-105**     |
| **GAMMA HA DFT/FFT > PPO + BC**  |  **8.01e-14**       | 0.12   |

### Comparison based on data source with Holm-Bonferroni correction:
| **Comparison**            | **Counter Circuit** | **Multi Strategy Counter Circuit** |
|---------------------------|-------------------------------|----------------------------------------------|
| **PPO + BC + GAMMA > PPO + BC** | 0.0031                   | 2.18                         |
| **FCP + GAMMA > FCP**                    | **1.30e-53**          | **3.00e-21**
| **CoMeDi + GAMMA > CoMeDi**         | 0.20                      | 5.41
Thank you, I appreciate this additional analysis. However, it is not correct to pick the best of multiple models and compute significance. This breaks the iid assumption. At a minimum, you would need to apply a multiple comparison correction. Furthermore, you haven't included the comparison of PPO+BC+Gamma against PPO+BC. As a result, it's still not clear whether GAMMA actually improves performance in many of the described settings.
Thank you for your rebuttal. It resolves some of my concerns and helps me better understand the experiment details. However, my concerns about formal analysis of latent space and the learning process of the cooperator policy remain. I will keep my original rating.
We appreciate your response to our rebuttal. In order to address your concerns, we provide the exact p-value for each comparison between methods for both layouts. In summary, our method GAMMA is statistically significant (p < 0.05) for the majority of comparisons between methods for both layouts, when comparing based on population generation method, and when comparing our best-performing GAMMA model across all baselines per layout. See the following tables for the exact p-values using a t-test. Our human study data has $n=80$ participants.

### Comparison based on data source:
| **Comparison**            | **Counter Circuit** | **Multi Strategy Counter Circuit** |
|---------------------------|-------------------------------|----------------------------------------------|
| **FCP + GAMMA > FCP**      | **6.48e-5**         | **9.99e-22**       |
| **CoMeDi + GAMMA > CoMeDi**| **0.0427**        |  0.902     |
| **GAMMA HA DFT/FFT > PPO + BC**| **4.08e-14**    | **0.030**                                            |

### Best-performing GAMMA:
| **Comparison**              | **Counter Circuit** | **Multi Strategy Counter Circuit** |
|-----------------------------|-------------------------------|----------------------------------------------|
| **GAMMA HA DFT/FFT > PPO + BC**  |  **4.08e-14**       | **0.030**   |
| **GAMMA HA DFT/FFT > FCP**       | **2.04e-57**    | **3.79e-71**         |
| **GAMMA HA DFT/FFT > CoMeDi**    |  **7.14e-07**  | **2.18e-105**       |
We appreciate your response to our rebuttal. In order to address your concerns, we provide the exact p-value for each comparison between methods for both layouts. In summary, our method GAMMA is statistically significant (p < 0.05) for the majority of comparisons between methods for both layouts, when comparing based on population generation method, and when comparing our best-performing GAMMA model across all baselines per layout. See the following tables for the exact p-values using a t-test. Our human study data has $n=80$ participants.

### Comparison based on data source:
| **Comparison**            | **Counter Circuit** | **Multi Strategy Counter Circuit** |
|---------------------------|-------------------------------|----------------------------------------------|
| **FCP + GAMMA > FCP**      | **6.48e-5**         | **9.99e-22**       |
| **CoMeDi + GAMMA > CoMeDi**| **0.0427**        |  0.902     |
| **GAMMA HA DFT/FFT > PPO + BC**| **4.08e-14**    | **0.030**                                            |

### Best-performing GAMMA:
| **Comparison**              | **Counter Circuit** | **Multi Strategy Counter Circuit** |
|-----------------------------|-------------------------------|----------------------------------------------|
| **GAMMA HA DFT/FFT > PPO + BC**  |  **4.08e-14**       | **0.030**   |
| **GAMMA HA DFT/FFT > FCP**       | **2.04e-57**    | **3.79e-71**         |
| **GAMMA HA DFT/FFT > CoMeDi**    |  **7.14e-07**  | **2.18e-105**       |
Copying this over from my individual response below:

With regard to standard errors, a gap of one standard error **between the edges of the standard error bars** corresponds to p=0.05. A gap of one standard error **between the means** does **not** correspond to p=0.05. See Fig 5 of the Cumming et al. (2007) paper for an example of this. It is still unclear which results are statistically significant. Consider applying a traditional hypothesis test to get the exact p-values rather than estimating them based on the gap between SEs.
Thank you for the rebuttal!
Some of my concerns have now been addressed, but the most major one still remains:
With regard to standard errors, a gap of one standard error **between the edges of the standard error bars** corresponds to p=0.05. A gap of one standard error **between the means** does **not** correspond to p=0.05. See Fig 5 of the Cumming et al. (2007) paper for an example of this. It is still unclear which results are statistically significant. Consider applying a traditional hypothesis test to get the exact p-values rather than estimating them based on the gap between SEs.