Improving Human-AI Coordination through Online Adversarial Training and Generative Models
Authors: Paresh R. Chaudhary
Yancheng Liang
Daphne Chen
Simon Shaolei Du
Natasha Jaques
Venue: ICLR 2026 Poster (2026)
Keywords: multi agent
adversarial training
zero-shot coordination
human-AI interaction
cooperation
reinforcement learning
Primary area: reinforcement learning

Abstract: Being able to cooperate with diverse humans is an important component of many economically valuable AI tasks, from household robotics to autonomous driving. However, generalizing to novel humans requires training on data that captures the diversity of human behaviors. Adversarial training is a promising method that allows dynamic data generation and ensures that agents are robust. It creates a feedback loop where the agent’s performance influences the generation of new adversarial data, which can be used immediately to train the agent. However, adversarial training is difficult to apply in a cooperative task; how can we train an adversarial cooperator?
We propose a novel strategy that combines a pre-trained generative model to simulate valid cooperative agent policies with adversarial training to maximize regret. We call our method \textbf{GOAT}: \textbf{G}enerative \textbf{O}nline \textbf{A}dversarial \textbf{T}raining. In this framework, the GOAT dynamically searches the latent space of the generative model for coordination strategies where the learning policy---the Cooperator agent---underperforms. GOAT enables better generalization by exposing the Cooperator to various challenging interaction scenarios. We maintain realistic coordination strategies by keeping the generative model frozen, thus avoiding adversarial exploitation. We evaluate GOAT with real human partners, and the results demonstrate state-of-the-art performance on the Overcooked benchmark, highlighting its effectiveness in generalizing to diverse human behaviors.

IMPROVING HUMAN-AI COORDINATION THROUGH ONLINE ADVERSARIAL TRAINING AND GENERATIVE MODELS
Paresh Chaudhary, Yancheng Liang, Daphne Chen, Simon S. Du, Natasha Jaques University of Washington, Seattle
pareshre, yancheng, daphc, sadu, njecs. wwwwwngton..ddeeee
ABSTRACT
Being able to cooperate with diverse humans is an important component of many economically valuable AI tasks, from household robotics to autonomous driving. However, generalizing to novel humans requires training on data that captures the diversity of human behaviors. Adversarial training is a promising method that allows dynamic data generation and ensures that agents are robust. It creates a feedback loop where the agent's performance influences the generation of new adversarial data, which can be used immediately to train the agent. However, adversarial training is difficult to apply in cooperative task; how adversarial cooperator? We propose a novel strategy that combines a pre- pre-trained generative model simulate valid cooperative agent policies with adversarial training to maximize regret. We call our method GOAT: Generative Online Adversarial Training. In this framework, GOAT dynamically searches the latent space of the generative model for coordination strategies where the learning policy- the Cooperator unnent-unddrrrfoomms GOAT enables better generalization by exposing the Cooperator to various challenging interaction scenarios. We maintain realistic coordination strategies by keeping the generative model frozen, thus avoiding adversarial exploitation. We evaluate GOAT with real human partners, and the results demonstrate state-of-the-art performance on the Overcooked benchmark, highlighting its effectiveness in generalizing to diverse human behaviors.
train
INTRODUCTION
In multi-agent human-Al cooperation (Carroll et al., 2020), training a cooperative agent to generalize across diverse human behaviors remains a formidable challenge. Distribution shift, limited training data, and highly dynamic human decision-making make training these systems difficult. Without adequate coverage of all possible human interactions, Al agents learn brittle policies and struggle generalize to novel, unseen behaviors in real human cooperation partners.
Adversarial training (Tucker al., 2020; Cui et al., 2023; Fujimoto & Pedersen, 2022; Rutherford et al., 2024) could provide a computationally efficient solution for training robust Cooperators by exposing both the Cooperator agent's 's limitations and underexplored areas of the partner strategy space. It can be used to automatically generate a curriculum of challenging tasks targeted to the learner's weaknesses, resulting in a more robust policy Dennis et al. (2021). Yet despite its potential, adversarial training within cooperative AI introduces unique challenges; what does it mean to train an adversarial cooperator? In a single-agent task, Adversary's 's sole objective is to challenge the learning agent much possible by minimizing its performance. However, in collaborative task, the adversarial agent has to find weaknesses in the learner while still simulating a realistic cooperative partner policy. If the adversary is not constrained to such policies, then it can become overly combative, resorting to strategies that sabotage the cooperation task, and is thus no longer a valid simulation of what a human cooperation partner might do. The challenge lies in simulating
Please check out our live interactive demo, game play videos, and training code at our website https: // i tes googl e vvi ew/ wgtptttttaaoo 2025/0220o

---

Pre- Trained Generative Model
Adversarially Training Cooperator
Jap(a,
angmaa Regret(9, #)
Japla, R)
Japla, d)
Sci-PPay
Sirvitaad Agara
Regret
Z
Z
Adheanary Sangler
Variatooall Autzesooder
Jxp(a.e)
Figure 1: Adversarial training framework for cooperative agents. (Left) A generative model encodes simulated agents into a latent space to learn diverse agent strategies, which are then used to generate different types of training partners. (Right) GOAT samples new partners to maximize the Cooperator agent's 's regret, defined as the performance gap between self- (the partner playing with itself) and cross-play (the partner playing with the Cooperator). The key idea is that the adversarial objective is constrained by the frozen generative model, which prevents it from generating self-sabotaging partners. But by applying regret-based adversarial training to search over the policies that can be generated by the model, we can expose the Cooperator to curriculum of challenging training partners, ensuring it is robust to interacting with diverse human partners at test time.
realistic yet challenging partners that maintain cooperative intentions while simultancously correcting weaknesses in the AI agent's policy.
For single-agent tasks, techniques like Unsupervised Environment Design (UED) (Dennis et al., 2021; Parker- Holder et al., 2022; Jiang al., 2021) constrains the Adversary so that it can only generate valid tasks for which there is a solution. This is accomplished by maximizing the learning agent's regret, which is the performance gap between the optimal agent and the learning agent. The regret objective proves effective because it challenges the learning agent by continuously identifying difficult but achievable scenarios where the agent currently underperforms. In the cooperative setting, attempts have been made to create a regret-like objective (Charakorn et al., 2023; Villin et al., 2025; Rutherford et al., 2024; Wang et al., 2025; Erlebach & Cook, 2024). In methods like Charakorn et al. (2023), diverse population is trained by optimizing for the difference between self-play and cross-play performance. However, naively applying this type of adversarial objective can lead to policies learning sabotage the game. While attempts have been made to add additional objectives to counterbalance this effect (Sarkar et al., 2023). However, this approach is not able to address the issue that minimizing cross-play inherently incentivizes overly adversarial, uncooperative behavior.
Parker-Holder
How can we ensure that the adversarial agent can only generate cooperative behaviors that do not engage in self-sabotage but can still fully explore the space of challenging partner strategies? We propose a novel adversarial training method to achieve these criteria. We first pre-train generative model of partner strategies on diverse cooperation data, following Liang et al. (2024). We then place this generative model into an adversarial training loop, where we train an Adversary to search for embedding vectors that, when passed through the generative model, decode into simulated partners that maximize the Cooperator's s regret. Because we only update the weights of the Adversary and not the generative model, we ensure that all generated partners are cooperative, thereby addressing the intentional sabotage seen in adversarial training. In GOAT, the Adversary is motivated to maximize the regret, searching for valid coordination strategies where the Cooperator agent underperforms. In contrast, the Cooperator attempts to adapt the Adversary and minimize the regret. Unlike previous adversarial approaches that risk learning degenerate strategies, this cyclic training method provides natural curriculum for the Cooperator agent and unlocks the benefits of adversarial training for robustly generalizing to cooperate with diverse partners.
We evaluate GOAT three distinct domains: One-Step Cooperative Matrix Game (CMG); 2) Cooperative Reaching Game (CRG); and 3) Overcooked (Carroll et al., 2020), a popular human-Al AI zero-shot coordination benchmark. We compare
competitive baselines (Zhao al., 2022; Sarkar et al., 2023; Liang et al., 2024; Strouse et al., 2022; Carroll et al., 2020) and show that GOAT achieves improved performance across all three popular environments. In the Overcooked environment, we tested GOAT live against real, novel human partners and found that it significantly improves cooperation performance.
The contributions of this paper are as follows:
2

---

We introduce novel adversarial training method for zero-shot coordination that trains both the Adversary and Cooperator agents online in a common payoff game, resulting in a dynamic curriculum where the Adversary searches over the space of partner policies that a generative model can simulate. The generative model is limited to simulating cooperative partners, which ensures the adversarial training process generates challenging but realistic cooperative partners.
We outperform competitive basclines across distinct popular cooperative environments.
We perform real-time evaluation with novel human partners, using the popular Overcooked benchmark (Carroll et al., 2020) commonly used in human-Al cooperation research like Strouse et al. (2022); Charakorn et al. (2023); Zhao et al. (2022); Sarkar et al. (2023); Liang et al. (2024). As compared to recent high-scoring techniques, we show that our method leads to the best cooperation performance with diverse people.
2 RELATED WORK
Zero-shot coordination (ZSC). Human-Al coordination requires training cooperative agents that can generalize to interactions with different humans zero-shot settings, and thus navigate a vast coordination behavior space. (Hu et al., 2020; Kirk et al., 2023). Behavior cloning (Carroll et al., 2020) is limited by expensive and often scarce human data. Self-Play (SP) (Samuel, 1959; Silver et al., 2017; 2018; OpenAl I et al., 2019) is an automatic training algorithm that does not need human data but converges to rigid policies that generalize poorly to novel human partners. Population- Based Training (PBT) (Jaderberg et al., 2017; Vinyals et al., 2019; Parker-Holder et al., 2020; Jung et al., 2020) has become a popular automatic training paradigm for ZSC, in which a Cooperator policy is trained using large population of simulated partners (Hong et al., 2018; Strouse et al., 2022; Lupu et al., 2021;
Charakorn et al., 2023; Zhao et al., 2022; Sarkar et al., 2023).
Population diversity objectives. In effort to ensure the simulated agent population covers diverse human strategies, various methods (Hong et al., 2018; Parker-Holee et al., 2020; Wu et al., 2023; Lupu et al., 2021: Zhao et al., 2022; Charakorn al., 2023; Sarkar al., 2023; Rahman et al., 2023; Yuan et al., 2023) improve PBT by optimizing the diversity of the agents in the population. The population can be created by reward-shaping (Tang et al., 2021; Yu al., 2023), manual design (Ghosh et al., 2020; Wang et al., 2022; Xie et al., 2021), and quality diversity (Canaan et al., 2019; Pugh et al., 2016; Wu al.. 2023; Fontaine & Nikolaidis. 2021). However, these methods typically use domain knowledge and fail to generalize and scale. Statistical methods like (Hong et al., 2018; Eysenbach et al., 2018; Horder-Holder& et al., 2020; Derek Isola, 2021; Lupu et al., 2021; Zhao et al., 2022) automate the diverse population generation, but Sarkar et al. (2023) demonstrated that trajectory distribution methods can result in minor variations in the policies rather than behaviorally different policies.
Min-Max optimization. Another line of work maximizes diversity by training self-play policies while minimizing cross-play rewards with the policies in the population (Cui et al., 2023; Charakorn et al., 2023; Rahman et al., 2023; 2024; Villin et al., 2025; Sarkar et al., 2023). However, agents might learn to sabotage the cross-play play game while retaining high self-play performance with handshaking protocols, such that if their partner fails to complete the handshake correctly, they throw the game. To address this, Sarkar al. (2023) introduces mixed-play, balancing self-play and cross-play by randomly sampling initial game states from both. However, such a solution still leads to rigid policies that fail to generalize, and does not address the fundamental problem that training a partner to minimize cross-pla play incentives sabotage. In this work, we only sample partners from a frozen, pre-trained generative model; since we do not train the partner policy on the adversarial objective, we prevent handshaking and sabotage. In addition, unlike prior work on this topic, we use online adversarial training to find and exploit weaknesses in the cooperator, instead of using adversarial objectives to pre-train a diverse partner population.
Generative models for Cooperation. Derek & Isola (2021); Wang et al. (2023); Liang et al. (2024) show that generative models can be trained policies to simulate diverse and valid collaborative policies. Recently, Liang et al. (2024) proposed training a Variational Autoencoder (Kingma & Welling, 2022), simulated cooperative agent trajectories to generate synthetic training data to improve human-Al coordination. This approach outperforms previous methods because the generative model enables sampling random novel partner strategies that interpolate between or combine strategies
3

---

from the data. GOAT leverages this approach to proactively generate challenging novel partners strategies to train the cooperator robustly.
3 PRELIMINARIES
Two-player Markov game: We consider the two-player Markov game (Littman. 1994) as a tuple (S, AI, A2, r; T, 7) where S is the state space shared by all the agents. Ai and A2 denote the action space of agents such that at each time step in the environment, and each agent selects one action a1,t E AA, 02, E Ag sampled from their own action space according to their policies Ti T + P(A1) and 72 S + P(A2). After taking these actions, they receive a shared immediate reward S x A1 x A2 - R. The system then transitions to the next state S1+1 according to the transition dynamics (ss+1 | st, 01 02,1) E P(S) where P(S) denotes the set of probability distributions over S. The parameter 7 [0, 1) is the discount factor that weights immediate versus future rewards. The expected joint return is captured by TT) = IE Li= which sums the discounted rewards from each timestep.
oo 7 r8tt, a1 (2,0)
Generative model of cooperation strategies: Following Liang et al. (2024), train Variational (VAE) (Kingma & Welling, 2022) on diverse cooperative strategies. The VAE is trained to reconstruct the partner's actions from dataset of coordination trajectories D, which is obtained from interactions between simulated PBT (Population-Basse environment. Here,
Training) agents within the fixed agent population with training methods like Maximum Entropy Population (Zhao et al., 2022) and CoMeDi (Sarkar et al., 2023) to provide the simulated partner agents. The VAE consists of an encoder ()) which maps trajectory a latent space, where & represents the parameters of the decoder. The decoder p(a, |2, To...-1: Ø) predicts actions a autoregressively using the latent and the agent's history To.. The VAE model is optimized using the ELBO loss described in (Kingma & Welling, 2022). The distribution of partner policies is therefore learned through the encoder's 's posterior g(=[T) over the dataset, which maps trajectories with different cooperation styles to different regions of the latent space. The KL term regularizes these posteriors towards the prior p(z) = (0, I). Sampling 2 ~ (0, the decoder p(a,1z, 1o Φ) generates partner policy corresponding to cooperation style, and diverse policies can be generated by sampling multiple latents from the prior. VAE thus learns a smooth distribution over partner policies in dataset D. The 33 parameter that balances reconstruction accuracy against the KL divergence in the ELBO loss, ensuring the latent space resembles a standard normal distribution (0, I). The choice of / leads to tradeoff between reconstruction quality and encoding disentangled latent representations (Burgess et al., 2018). We experiment with various / values to get better disentangled representations.
(so, ao. .... ST) to
GOAT: GENERATIVE ONLINE ADVERSARIAL TRAINING
We introduce GOAT, which combines generative modeling with online regret- based adversarial training to enable robust human-Al coordination.
Adversarially minimizing Cross-Play (XP) performance. We begin by introducing an adversarial training objective based on the XP performance of the Cooperator Tे and an adversarial partner AA. The adversary म A attempts minimize cooperation performance, while a Cooperator policy me attempts to maximize it. This type of optimization is widely known as minimax optimization in game theory:
JXP = max min JTTC TA)
As may be obvious, attempting to train agents to cooperate via XP minimization leads to a fun- damentally misaligned optimization objective because it conflicts with the cooperative objective of maximizing common payoff. The Adversary agent is incentivized to develop behaviors that intentionally sabotage interactions rather than generate meaningful cooperation trajectories.
Generating adversarial cooperative agents. To address the above issues, we propose adding a pre-trained generative model (VAE) with frozen weights into the adversarial training loop. We hypothesize that adding a generative model trained solely on cooperative policy data will ensure that adversarial training generates only valid cooperative partners. By design, the generative model is not
4

---

capable of simulating self-sabotage handshaking behaviors. Instead, train separate Adversary policy to search for embedding vectors & that decode into partners that minimize XP performance for the specific Cooperator agent seek to train. The Adversary can thus search over well-structured, continuous latent space, allowing a smooth optimization landscape.
We formalize this adversarial training algorithm by modifying the previous Jx formulation in Eq. 1. Now, the Adversary policy conditions on a randomly sampled 2 ~ [(0, I), and transforms it to pro- duce an Adversary embedding z' т T (z). This modified æ' is fed the decoder p(az|z', To.e-110)) of the generative model to produce the actions of the simulated partner policy p.. Let xp (a(()) be the partner policy produced by having the adversary л A choose an embedding which then simulated with the generative model. Then, the optimization problem is as follows:
max min E (O.1( WC
Σ rsts Te(af To.--),, (Ta Tor- 1))
(2)
HA
max min E [ ]
This objective no longer directly trains the partner policy on the adversarial objective, but instead uses the adversary to steer the pre-trained generative model to select partners.
Regret-bassee -baee adversarial cooperative training, Because Eq. is still fundamentally a minimax objective, it does not guarantee effective exploration the partner space because it optimizes for worst-case returns. Therefore, the Adversary will only search for the worst agents that the generative model can simulate. Instead, we would like to formulate a better objective that encourages searching for meaningful partner policies that could have good performance, but for which the Cooperator still fails to perform well. Thus, we propose using a regret objective to generate a realistic curriculum for the Cooperator agent. As proposed initially by Robbins (1952), regret is the performance gap between the returns of an agent-chosen strategy and the returns of the optimal strategy that could have been selected with perfect foresight. We formalize regret in the cooperative setting the performance gap between the cross-play (XP) performance an agent achieves with a partner T and Tpps 's optimal self-play (SP) performance, (TJ TP)], which is the score achieves when paired with itself.
Regret(np, TC) = [[][mp J(T,, TC) = JSP JXP
GOAT objective. At the start of the training process, GOAT samples from the standard normal distribution and transforms it to the effective latent vector 2'. This latent vector 2' then used to generate the partner policy Ty using the frozen weights of the VAE decoder p(a,|=',m 10 0). The partner policy Tp is then paired with itself evaluate SP returns and with Fe to evaluate XP returns. Regret is evaluated after collecting the returns from both games. The Adversary is trained to choose z' to maximize regret, while the Cooperator policy is trained to maximize its cooperation performance. The full objective of the GOAT game is thus:
min max Regret( (TP ..)) Re
Tals) Ja(x)
(5)
RA
The online optimization, shown in algorithm ensures that the Adversary is incentivized to continually discover new partner strategies that are challenging for the Cooperator, by searching the latent space maximize regret. The Cooperator, in turn, learns to adapt to these partner strategies by minimizing the regret. This cyclic hehavior naturally trains the Cooperator to learn to cooperate with diverse strategies, from more straightforward to increasingly complex ones. For invalid partners, the optimal policy cannot achieve positive returns, resulting in zero regret and providing no incentive to the Adversary. This property of regret promotes exploration that leads to curriculum generation to increasingly generate complex yet solvable tasks to train the Cooperator agent.
5 EXPERIMENTS
Our experimental evaluation designed to assess how well our adversarial training procedure can improve zero-shot cooperation performance, both in simulation and with real humans.
Environments. We evaluate GOAT using three popular coordination benchmarks: 1) One-Step Cooperative Matrix Game (CMG) (Lupu al., 2021; Zhao al., 2022; Charakorn et al., 2023), 2)
5

---

Agent Action
Agent Action
Agent R Action
Agent Action
12-
1.
24
14
30
36
40
54
12 15
36 48 5460
12
60
18 25
12 24
Agent Action
Agent C Action
Agent Action
Agent C Action
(a) CMG Payoff Matrix
(b) Self-Play
(c) MEP
(d) CoMeDi
Agent R Action
Agent R Action
Agent R Action
B
Expected Reward
24
33
as
B
i
MER
CoMeD!
GAMMA
MINIMAX
GOAT
18 36 44
23 34
31 at Agent C Action
Agent C Action
Agent C Action
(c) GAMMA
(f) MiniMax
(g) GOAT
(h) Expected Reward
Figure 2: a) Cooperative Matrix Game. b) to g) Policy probability distribution to show coverage of different methods on the CMG payoff matrix h) Total expected rewards for each method, assuming we uniformly sample partners and each method gets a payoff for each partner proportional to amount of coverage they have for the reward block.
Cooperative Reaching Game (CRG) (Rahman et al., 2023; 2024; Charakorn al., 2023), and 3)
Overcooked (Carroll et al., 2020).
One-Step Cooperative Matrix Game (CMG). CMG is single-step game where agents (Row and Column) independently select an action by choosing a row and a column, respectively, and the intersection of their choices returns the associated reward according to the payoff matrix. Both agents receive the same reward. A typical payoff matrix in CMG is defined by (M, [mm) frm)) where M is the number of solutions, km the number of compatible actions, and r. is the reward associated with each solution. We use a challenging layout shown in figure 2a with three scenarios: no reward, suboptimal rewards, and global maximum reward.
Cooperative Reaching Game (CRG). CRG is 5x5 grid world, with |A| = 5, where two agents need to reach the same goal location simultancously to receive rewards. For agents cooperate and receive rewards, they must reach and stay in any of the same goal coordinates. An ideal Cooperator should chase their partner the goal persuade them to follow the same goal. We evaluate GOAT and prior baselines against 11 heuristic agent teammates described in detail in appendix E.
Overcooked. A state-of-the-uu collaborative, real-time benchmark where two players need to cooperate by dividing cooking tasks and avoiding obstructing each other. Identifying the opposite player's intent and behavior key to determining how coordinate with the other agent
Algorithm GOAT
Training VAE on trajectory dataset D = - where = {ss a; N Initialize: VAE Decoder p(a, zz, To..--1:00 Adversary TA, Cooperator TC
while not converged do
Sample 2 (0,
Map z' = (2)
Generate тр = p(a 22 To...1;0)
Collect Partner self-play returns J (TTP, TP)
Collect Partner- Coater-Coooprrtorr returns [[Tp, TC)
Compute REGRET = J[TP, TP) - [[Tp, TT)
Train Adversary Policy (TT) with RL to maximize REGRET Train Cooperator Policy (TC) with RI. minimize REGRET
end while

---

effectively. We test GOAT the most challenging tasks with a diverse strategy space. Counter Circuit introduced in Carroll et al. (2020), and a more complex layout introduced by Liang et al. (2024), Multi- Saltt--trategy Counter.
GOAT implementation & Baselines: Training the Adversary is RL-algorithm-agnostie it does not depend on state and is one-step optimization process. To keep the optimization loop simpler, the Cooperator is trained using PPO, and the Adversary is trained using REINFORCE. Details the KL- regularization of the VAE, training procedure for GOAT, and other architecture and hyperparameters available in the Appendix. We compare to 5 competitive prior baselines on human-Al coordination: Behavior Cloning (BC) partners with RI. cooperator (Carroll et al., 2020) (BC+RL), PBT methods Fictitious Co-Play (FCP) (Strouse et al., 2022), maximum entropy population (MEP) (Zhao et al., 2022), CoMeDi Sarkar et al. (2023), and GAMMA trained on MEP (MEP+GAMMA) Liang et al. (2024), which previously showed state-of-the-ar performance in real human evaluations.
Human Evaluation: We conducted a user study with 40 participants to discover which method best coordinated with humans in a real-time evaluation in Overcooked, recording the team scores. Participants were recruited from the Prolific online crowdsourcing platform following an IRB- approved protocol. Each participant was tasked with completing 6 total rounds of the Overcooked game with an Al agent partner, where each round presented a different agent model among one of the baselines or our method. In order to minimize trends due to gameplay order, the agent models are loaded in randomized order, with randomly sampled checkpoint chosen from I of 5 random seeds.
EXPERIMENTAL RESULTS
Can GOAT discover and exploit multiple coopera- tive strategies? For GOAT to be effective at generalizing novel partners, the adversary should be able to discover and exploit all rewarding cooperative modes. We first test this question with the CMG environment, using a population size of 8 for all baseline PBT methods. To train the VAE using the population of CoMeDi agents, we use the same procedure as GAMMA Liang et al. (2024), which pairs each of the 8 row and column agents in the population together to generate trajectory data. Since VAE does not have a population, can
RQI:
1.0
HOT
0.35
0.35
0.85
0.75
0.60
Sum H11 H1O HD9 HD8 HD7 HO6 HO5 H04 H03 H02
0.60
0.95
0.55
1.00
1.00
0.00
0.20
0.60
1.00
1.00
0.8
Heuristic Agent
1.00
1.00
0.75 0.75
0.6
0.30
0:75
0.75
0.75
0.75
Score
0.60
0.60
0.80
1.00
1.00
generate many more agents than PBT methods.
0.4
1.00
0.75
0.95
1.00
1.00
As seen in Figure 2b and 2c, the baselines' Self-Play and MEP fail to cover all modes, implying they would not generalize to partners that used those modes. CoMedi 2d and GAMMA 2e discover all the regions and give the highest priority to the global maximum, but spread probability mass roughly equally across all other modes, regardless of their payoff. Whereas MiniMax 2 converges to the worst-case scenario. In contrast, GOAT not only covers all the modes but also is better at assigning higher probability to more rewarding modes (for example, assigning low probability to modes 3, 4, and 7, which have low payoff). Thus, GOAT achieves good coverage while also focusing on maximizing return. This is also evident in Figure 3, which shows the results of evaluating each method in the CRG environment. Here we see that GOAT achieves the highest average score when teamed with all 11 heuristic partner agents.
1.00
0.60
0.80
1.00
1.00
0.15
0.60
0.2
0.15
1.00
1.00
5.50
5.95
7.30
9.40
9.70
CoMebi
GAMMA
GOAT
0.0
Method
Figure 3: CRG: Average reward ob- tained against the 11 Heuristic Agent teammates across 5 seeds of each method. Sum row highlighted in red shows the summation of rewards, where the maximum possible reward is 11.
RQ2: Will adversarial training result in a more robust Cooperator with better sample complexity in simulation? PBT methods rely on simulated populations of self-play partners, and have poor coverage of actual human behaviors. Training Cooperator against them may not achieve satisfactory performance with people, and may take a long time to converge since the training does not focus correcting weaknesses in the Cooperator, but instead randomly samples from the population. We hypothesize that our adversarial training technique can more efficiently search the space of partners to make an agent that is more robust and
7

---

FGP
IPP
MLP-GAWM
100
Average Score
75
Reward
60
40
25
0.00
0.25
0.50
0.75
1.00
BC
CoMeDi
MEP
GAMMA
GOAT
Step
168
(a) Counter Circuit
(b) Simulation Results
(c) Human evaluation
FCP
G0X
Do/ACi
MEP GANAA
100
Average Score
Reward
50
60
25
20
%.000
0.25
0.50
0.75
1.00
BC
PCP
CoMeDi
MEP
GAMMA
GOAT
Step
168
(d) Multi-Strategy Counter
(c) Simulation Results
(f) Human evaluation
Figure 4: Overeooked: Carroll et al. (2020) first introduced the most challenging Counter Circuit layout (a). To increase coordination complexity, Liang et al. (2024) introduced the Multi-Stratteyy Counter layout (d), where players could choose between tomato and onion ingredients to cook soup. This adds an additional coordination challenge where agents have to adapt to soup-making strategies based on different ingredients.
(b) (e) are evaluation of GOAT against a human proxy model trained with BC. We compare baselines, which are trained using simulated population data, including the previous state-of-the-art method, GAMMA (Liang al., 2024). Error bars show std. emr. over random seeds. (c) & (f) shows the evaluation of the performance of methods when tested against real humans in two layouts: counter circuit and multi-strategy counter, respectively.
requires less training time. To validate this hypothesis, investigate the Overcooked learning curves of all simulated training methods, including our own, in Figure 4b and 4e. Here, we evaluate the cooperation performance of the agents against Human Proxy model (behavior-cloned (BC) model trained on human data) as in prior work (Carroll et al., 2020; Strouse et al., 2022; Liang et al., 2024). The human proxy enables the automatic evaluation of agents' performance throughout training.
As is evident in the curves, GOAT achieves significantly higher performance than all baselines in both Overcooked layouts and increases performance more quickly than other techniques. We hypoth- esize that adversarial exploration in the generative model's latent space enables rapid coverage of diverse human behaviors, ranging from simple coordination strategies complex, context-dependent strategies. By constraining the Adversary to valid cooperative policies, the method avoids degenerate strategies and exposes the Cooperator all the semantically diverse strategies that are beneficial for generalizing to diverse partners. It efficiently targets training to weak points in the Cooperator's policy, and thus has lower sample complexity than traditional PBT methods.
RQ3: Can GOAT achieve improved cooperation performance when evaluated in real-time with novel humans? The real test of Auman-Al coordination algorithm is whether can work with actual people. As described in Section 5, evaluated the Cooperator agents trained with GOAT and the 5 basclines alongside real human partners in the Counter Circuit and the Multi- Strategy Counter Overcooked layouts. We recruited novel human players with prior exposure to the agents. Figure 4c and 4f depict the evaluation results for the two layouts and all the agents against human players. GOAT outperforms the previous baselines both layouts. Even though GOAT shows significant improvement over baselines on Counter Circuit simulation results, the magnitude of the improvement over GAMMA is smaller (3%) this layout. This is because both methods are reaching close to optimal performance in this simpler environment. However, on the more complex layout, Multi-Strategy Counter, GOAT leads markedly higher cooperation performance with real humans, with 38% improvement over the previous state of the art. We encourage readers to test these differences for themselves by visiting our webpage, which provides an interactive demo in which you play Overcooked with both GOAT and the baselines https: / //sites. google. wwwcco//////ooeo.e www.vwww///c/ccotcoo...o.......e vwtp///wtwww/tte/.t/ooo/// goat 2025 home.
8

---

30
1750
20
1500
10
1250
Episodes
750
-20
-30
GAMMA GOAT
250
Minimax Regret
-40
-00
20
10
20
30
-90
-20
-15
(a) GOAT exploring the latent space
(b) Regret vs Minimax Exploration
Figure 5: a) GOAT explores the latent space of the generative model across training episodes, progressing from episode (purple points) to episode 1750 (yellow points). At the same time, GAMMA (red) remains stationary, maintaining standard normal distribution cluster. b) Minimax (red) mostly fixates on regions of low reward or poor behavior, while Regret (blue) explores more broadly and generates variety of challenging training scenarios.
RQ4: Will adversarial training create a curriculum of increasingly challenging strategies by discovering them in latent space? The regret-driven adversarial framework is designed create a dynamic curriculum where the Cooperator must continually adapt to novel challenges. GOAT identifies underperforming coordination strategies and generates increasingly complex cooperation challenges, compelling the Cooperator to explore and master new behaviors. This iterative process prevents the Cooperator from stagnating into rigid conventions, common pitfall in self-play or population-bass methods. Figure 5a assesses how GOAT explores the latent space of the generative model over the whole training interval. Each point represents latent vector collected during training episodes 0 to 1750. The Figure 5a compares GOAT embeddings to the distribution of embeddings of original GAMMA method (Liang et al., 2024) samples throughout training (red ball). We can see that at the start of training, GOAT samples latent vectors distributed throughout this ball (shown in purple). As training progresses, it chooses a particular region to exploit. After a few iterations, moves to new region, presumably because the Cooperator has adapted to that strategy, and it is longer able to maximize regret. Finally, at the end of the training step in yellow, GOAT has traveled across the latent space explore particular regions (those with high regret) to find valid strategies that exploit weaknesses in the Cooperator.
RQ5: Ablation study of Minimax vs. Regret. In Section 4, we hypothesized that even with generative model in the loop, the purely adversarial minimax objective (Eq. 3) is misaligned with effectively training agents to cooperate. While the generative model can prevent sabotage behavior, because minimax focuses on worst-case scenarios, we hypothesized that a minimax adversary would still focus too much on poorly performing or incompetent partners. In this experiment, we directly compare training the GOAT adversary with either minimax (Eq. 3) or regret (Eq. 4) on the Multi- Strategy Counter Overcooked layout. We further investigate the embeddings chosen by both types of adversary throughout the training period in Figure 5b. We find that the minimax adversary mostly fixates on a single region (red cluster in the plot) where the policies result in low-reward or unproductive behaviors. One such example, shown as video agent avoids working with its partner and converges to doing just enough not to sabotage the task. The team still receives rewards the partner policy keeps completing the task uninterrupted. In contrast, we see that the regret objective causes GOAT to move to multiple modes in the embedding space over the course of training, exploring the latent space more effectively and thus covering wider range of interesting behaviors. As shown in game videos on our website, it starts by doing one task, then shifts another task, and then does both tasks simultaneously. We can also see robust training instances like intentionally coming in the way or taking roles.
website, is where the minimax

---

7 CONCLUSION
Our work demonstrates that combining generative modeling with adversarial training provides an effective approach for training Al agents that can coordinate with diverse human partners. Using a generative model and regret-based optimization to constrain the adversary enables GOAT to generate valid, meaningful cooperation partners while posing curriculum of increasingly challenging scenarios, adapted to the weaknesses of the Cooperator's policy. Thus, we enable robust learning without the risk of degenerate strategies. Experimental results on three cooperation tasks show that GOAT achieves state-of-the-art performance in both simulated evaluations and studies with real human partners. The improved coverage the partner space demonstrated by our agents suggests this approach can successfully and robustly achieve generalization to human behavior. Future work could explore extending this framework to more complex collaborative tasks and investigating ways to incorporate explicit human feedback during training.
ACKNOWLEDGGENTS
This research was supported by the Cooperative AI Foundation, the UW-Amazon Science Gift Hub, Sony Research Award, UW-Tsukuba Amazon NVIDIA Cross Pacific AI Initiative (XPAI), the Microsoft Accelerate Foundation Models Research Program, Character.AI, DoorDash, and the Schmidt A12050 Fellows program. This material is based upon work supported by the Defense Advanced Research Projects Agency and the Air Force Research Laboratory, contract number(s), s): FA8650-23-C-7316. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of AFRL or DARPA.
10

---

REFERENCES
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in B-vae, 2018. URL wtw.thtts///rrat.. org/abs, 0obs//800 1804. 03599. 4
Rodrigo Canaan, Julian Togelius, Andy Nealen, and Stefan Menzel. Diverse agents for ad-hoc cooperation in hanabi. In 2019 IEEÉ Conference on Games (CoG), pp. 1-8. IEEE, 2019. 3
Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination, 2020. URL
https: /tt////axivvvv//vv//////eeeeaaaa org/ abs/ / 1910. 05789. 1,6,7,6,7,
Rujikom Charakorn, Poramate Manoonpong, and Nat Dilokthamakul. Generating diverse cooperative agents by learning incompatible policies. In The Eleventh International Conference on Learning Representations, 2023. URL https: httpss///pppeevveeeeeee net/ forum? cworummid=Ukknn ddUkkU0GGHOHHHH C 5,
Brandon Cui, Andrei Lupu, Samuel Sokota, Hengyuan Hu, David Wu, and Jakob Nicolaus Fo- erster. Adversarial diversity in hanabi. In The Eleventh International Conference on Learning Representations, 2023. URL
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design, 2021. URL pettps:/aaxivv.oot
www.oo//g///ba//oooo/ooosoee abtp/////2...2 02096. 1,2
Kenneth Derek and Phillip Isola. Adaptable agent populations via a generative model of policies.
Advances in Neural Information Processing Systems, 34:3902-3913, 2021. 3
Hannah Erlebach and Jonathan Cook. Raccoon: Regret-based adaptive curricula for cooperation. 2024. 2
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: /802,06070 2018.
Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. Advances in Neural Information Processing Systems, 34: 10040-10052, 2021. 3
Ted Fujimoto and Arthur Paul Pedersen. Adversarial attacks in cooperative ai, 2022. URL https:
//taaxiiirg/aaaaaaaaaaaaaaaaaaaa // //arxivv.co org. abs /211: 14833.
Ahana Ghosh, Sebastian Tschiatschek, Hamed Mahdavi, and Adish Singla. Towards deployment of robust cooperative ai agents: An algorithmic framework for learning adaptive policies. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 20, pp. 447-455, Richland, SC, 2020. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 1453351884.
Zhang-Wei Hong, Tzu- Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, and Chun-Yi Yi Lee. Diversity- driven exploration strategy for deep reinforcement learning, 2018. URL httpst http/arxvvvv.oo//////o org/ abs/1802 04564. 3
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "'other-pla"" for zero-shot coordination. In International Conference on Machine Learning, pp. 4419-4410. PMLR, 2020. 3
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M. Czarnekki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks, 2017. URL https:// arxiv. orgoorgggao abs /alb//110 /11///11.11 39
Minqi Jiang, Michael Dennis, Jack Parker- Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances Neural Information Processing Systems, 34:1884 14:1888-188 18971897, 2021.
11

---

Whiyoung Jung, Giseung Park, and Youngchul Sung. Population-guidee parallel policy search for reinforcement learning. arXiv preprint arXiv:200.002977, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https:
//arxiv... org/ abs/ /132. 6114.111.6 3,
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76: 201-264, 2023. 3
Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S. Du, and Natasha Jaques. Learning to cooperate with humans using generative agents, 2024. URL 2411. 13934. 3, 8,
Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In William W. Cohen and Haym Hirsh (eds.), Machine Learning Proceedings 1994, pp. 157-163.
Morgan Kaufmann, San Francisco (CA), 1994. ISBN 978-1-55860-3356 doi: https://doi. org/10. 100066687788--55500-333------66- sci
URL https: www. scienced rect. com/ / 55 (911//898811686300088
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7224-7213. PMLR, 18-24 Jul 2021. URL wwtptt:ttpptpeeeeeeeeeeepo proceed ngs. ml r. w.w.psssvvee. vv39// lupu2la. html. 3,5
OpenAI, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning, 2019, URL https: //trxava.vv.////a///a//eaaeaaava org/ abs 1912.. 06880. 3
Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique
Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 15:18800-18862 2020. 3
Jack Polder-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In International Conference Machine Learning, pp. 1477-17498. PMLR, 2022. 2
Justin Pugh, Lisa Soros, and Kenneth O Stanley. Quality diversity: new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V. Albrecht. Generating teammates for training robust ad hoc teamwork agents via best-response diversity, 2023. URL https:
//arxiv. org. abs 2207 14138. 3,
Muhammad Rahman, Jiaxun Cui, and Peter Stone. Minimum coverage sets for training robust ad hoc teamwork agents. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances Artificial Intelligence, AAAAT241AAAT24/AAATTT AAAI Press. 2024. ISBN 1-57735--8777 doi: 10.1609/aaii..38881.... org/10.11 100./169//40 aaai v38i16 29702. 3, 6
URL https: /doi.
Herbert E. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58:527-535, 1952. URL https: / /ppi. semanti wnchocccccllla. org/
CorpusID: 155.561:...55593..
Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, and Jakob Foerster. No regrets: Investigating and improving regret approximations for curriculum discovery, 2024. URL https: wwwpaaawwvv.oorrareeaeeeeesee wwwo//a///////eooooerooooooeaeee 2200 15099. 1
12

---

A. L. Samuel. Some studies in machine learning using the game of checkers. IBM Journal of Research and Development, 3(3):210-222 1959. doi: 10.1147/hd.3300210.
Bidipta Sarkar, Andy Shih, and Dorsa Sadigh. Diverse conventions for human-ai collaboration, 2023. URL https::pp/..... //aari//aviii/ org/ abs 2310 15414. 4,7
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL /tt.ttts////xii.orrrr 00..0///////0000.3..... 15
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm, 2017. URL https:
abs/ 1712. ..2.000 01 01
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou. Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, (426641))1110--111 1144, 2018. doi: 10.1 2a111///cience.aar6//. URL https: // www. sciesieee encience... org/ doi aabs/ / sci ecceence.oo aa 6404..
DJ Strouse, Kevin R. McKee, Matt Botvinick, Edward Hughes, and Richard Everett, Collaborating with humans without human data, 2022. URL hwtps::t///eeeeeee http://axvvvvvo org/ abs /ttp/////..////////pp//oooo 08106 2, 7,8
Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang. Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. arXiv preprint arXiv:2
2021. 3
Mycal Tucker, Yilun Zhou, and Julie Shah. Adversarially guided self-play for adopting social conventions. arXiv preprint arXiv:2001.0599 2020.
Victor Villin, Thomas Kleine Buening, and Christos Dimitrakakis. A minimax approach ad hoc teamwork, 2025. https::::
arxiv .ev.vooore www.oorggr///////0/
2,
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnekki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Hwalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P Agapiou, Max Jaderberg, Alexander S Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L Paine, Caglar Gulcehre. Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap. Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agen reinforcement learning. Nature, 575(7782):
350-354, November 2019. 3
Caroline Wang, Arrasy Rahman, Jiaxun Cui, Yoonchang Sung, and Peter Stone. Rotate: Regret-driven open-ended training for ad hoc teamwork. 2025. URL https: wttp:///wwwwo..aaa///a/a//// /2505 23686. 6. 2
Chen Wang, Claudia Pérez-D' Arpino, Danfei Xu, Li Fei-Fei, C. Karen Liu, and Silvio Savarese, Co- gail: Learning diverse strategies for human-robot collaboration, 2023. URL https: //axxiv.. orgg/baa 2108.000 06.38.
Woodrow Zhouyuan Wang, Andy Shih, Annie Xie, and Dorsa Sadigh. Influencing towards stable multi-agent interactions. In Conference on robot learning, pp. 1132-1144 1143. PMLR, 2022. 3
Shuang Wu, Jian Yao, Haobo Fu, Ye Tian, Chao Qian. Yaodong Yang, Qiang Fu, and Yang Wei. Quality-similar diversity via population based reinforcement learning. In The eleventh international conference on learning representations, 2023. 3
Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representa- tions to influence multi-agent interaction. Conference on robot learning, pp. 575-588. PMLR,
2021. 3
13

---

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games, 2022. URL https: //arxiv. wttpxivvogg/bb//110000 01955. 15
Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning zero-shot cooperation with humans, assuming humans are biased, 2023. URL https:
///axxxi/ org/ 22t///bbb/22..e.////// 01605.
Lei Yuan, Lihe Li, Ziqian Zhang, Feng Chen, Tianyi Zhang, Cong Guan, Yang Yu, and Zhi-Hua Zhou. Learning to coordinate with anyone. In Proceedings of the Fifth International Conference on Distributed Artificial Intelligence, DAI '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400708480. doi: 10.1145/3276666.6276888 URL https:
//doi.org/10. 0tt.///20..11//7776666663 362777766666778888
Rui Zhao, Jinming Song, Yufeng Yuan, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Maximum entropy population-based training for zero-shot human-ai coordination, 2022.
URL httpt::////aatta org/ /ttp////1... 11701. 2, 3, 4,5, 7
14

---

A
REPRODUCIBILITY
We have live interactive demo, extended experimental data, and code on our webpage, link first page.
B IMPLEMENTATION DETAILS
Generative models. The VAE model trained dataset containing joint trajectories of two players. This dataset was created by evenly sampling from the pairs in the simulated agent population (T1...., ) generating 100k joint trajectories. The dataset was split in 70/30 format for training and evaluation. The VAE was trained using ELBO loss, and linear scheduling of the KL penalty coefficient 3 was applied to control the KI. divergence checkpoints of VAE models of different strengths were saved. For CMG, the training data used for VAE was 8 policies generated by CoMeDi on CMG. Due to a smaller amount data, we used data augmentation to train the VAE.
the posterior distribution. Automatically.
Cooperator Agent The Cooperator agent was trained using PPO (Schulman al., 2017). promote exploration, the first 100M steps use reward shaping for dish and soup pick-up. We adapted our codebase on the previous implementation of HSP (Yu et al., 2022) and GAMMA (Liang et al., 2024).
GOAT For GOAT policy training, keep the policy constrained in the VAE latent space. This objective requires two separate passes of the environment gather returns for self-play and cross-play.
the REINFORCE objective with KL divergence penalty to
C HYPERPARAMETERS
GOAT uses simple policy and was trained using REINFORCE to generate latent vectors. The architecture and hyperparameters are defined below.
hyperparameter network architecture
value
4-layer MLP
hidden dimension activation function output dimension optimizer
128
ReLU
2x z dim
Adam
learning rate weight decay
0.0005
0.0001
latent dimension (z_dim)
KL coefficient
5.0
Table 1:
Hyperparameters for GOAT Generator
The Cooperator agent is trained using MAPPO with all overcooked layouts having similar low-level implementation details like architecture and hyperparameters. Every policy network has the same structure, with CNN coming after an RNN (we use GRU).
The architecture of the policy model and the generative model are comparable. The representations are transformed into variational posterior and action reconstruction predictions using an encoder head and decoder head.
D COMPUTATIONAL RESOURCES
Our primary tests were carried out on AMD EPYC 64-Core Processor and NVIDIA L40s/L40 clusters. One Cooperator agent can be trained in roughly day. The primary tests require roughly 24-36 GPU hours for Overcooked and hour for rest of the environments. We perform some preliminary experiments to determine the best training frameworks and hyperparameters.
15

---

hyperparameter
value
CNN kernels
[3, 3], [3, 3], [3,
CNN channels
[32, 64, 32]
hidden layer size
[64]
recurrent layer size
64
activation function weight decay environment steps parallel environments episode length PPO batch size
ReLU
0
100M (simulated data) or 150M (human data)
200
400
2 x 200 x 400
PPO epoch
15
PPO learning rate
0.0005
Generalized Advantage Estimator (GAE) À
0.95
discounting factor y
0.99
Table 2: Policy hyperparameters
hyperparameter
value
CNN kernels
[3, 3]. [3, 3J. [3, 3] [32, 64, 32]
CNN channels
hidden layer size
[256]
recurrent layer size
256
activation function
ReLU
weight decay
0.0001
parallel environments
200
episode length epoch chunk length
400
500
100
learning rate
0.0005
KL penalty coefficient /
0 +
latent variable dimension
16
Table 3: Hyperparameters for VAE models
E COOPERATIVE REACHING GAME: GAME LAYOUT & 11 HEURISTIC AGENTS
Figure 6: Cooperative Reaching Game (CRG): Red and Blue the agents that should cooperate to reach and stay on one of the goal coordinates in the corner. Yellow goal coordinates lead to reward 1, and Green goal coordinates lead reward of 0.75.
Heuristic H01: Selects actions moving teammates toward the nearest reward- providing coordinate.
Heuristic H02: Selects actions moving teammates toward the reward-providiee coordinate most distant from their episode starting position.
Heuristic H03: Moves teammates toward the nearest optimal reward- reward-providing coordinate.
Heuristic H04: Moves agents toward the optimal reward-providing coordinate most distant from teammates' initial episode location.
Heuristic H05: Same as H4, but considers only suboptimal reward- -rrviddiooviiingg coordinates instead of optimal ones.
16

---

Heuristic H06: Same H5, but teammates toward the nearest suboptimal teward-providing coordinate.
Heuristic H07: At episode start, agents randomly select a reward-providing coordinate and move toward it.
Heuristic H08: Moves teammates toward the reward-providing coordinate nearest their counter-
part agent's s location.
Heuristic H09: Same as H8, but considers only optimal
coordinates as destina-
tions.
Heuristic H10: Moves teammates directly toward their counterpart agent's location.
Heuristic HIl: Always randomly selects actions from teammates' available action set.
F
TESTS ON CMG-S
Agencc Actes
(a) CMG-S layout
(b) Self-play
(c) TrajeDi
(d) MEP
(c) CoMeDi
GOAT
Figure 7: We provide additional tests of the CMG-S problem space. In the problem space, rewards flow in a direction that creates a smooth learning landscape for the algorithms to exploit. The policies for the methods were aggregated and normalized into heatmap. GOAT successfully assigns priorities to regions.
G MINIMAX AND REGRET COMPARISON ON OVERCOOKED
GOAT
Minimax
Multi-strategy Counter
80
Reward
20
0.0
0.2
0.4
0.6
0.8
10
Step
160
Figure 8: Performance comparison between minimax GOAT objective Multi-Strategy Counter when evaluated against a held-out human behavior cloned model as a partner. Error bars are the Standard Error of the Mean. GOAT outperforms MiniMax, but here MiniMax produces poor- performing agents that fail to participate coordination. See game videos on our website.
17

---

H MODE DISCOVERY WITH RESPECT TO POPULATION SIZE.
Discovered Modes
6
4
SP
CoMeDi
2
GAMMA
24 8 16
32
64
Population Size
Figure Number of modes discovered by each baseline in Cooperative Matrix Game (CMG) with respect to the population size. CoMeDi. GAMMA, and GOAT discover all the modes with population size of 8, but each one receives different rewards as shown earlier in Figure 2.
I GOAT USING DIFFERENT VAES TRAINED ON DIFFERENT AGENT DATA
(MEP) COAT (CoMsDD)
75
Reward
50
0.4
1.0
Step
1e8
Figure 10: Performance comparison between GOAT using VAE trained with MEP agents data (purple line) vs GOAT using VAE trained with CoMeDi agents data (brown line) on Counter Circuit Overcooked Layout.
J COOPERATIVE STRATEGIES LEARNED BY VAE
G
0.004
12
(Row) Acten
18
0.006
Agant (Row) Action
18
24
24
0.003
Coverage
30
0.004
36
36
0.002
0.003
42
48
0.002
0.001
54
0.001
60
12 18 30 42 48 Agent C (Column) Action
12 18 24 30
36 42 48 54 60 Agent C (Colum)) Action
Figure 11: Left: We collect 4 cooperative policies CMG-M using CoMeDi and then aggregate them. Right: The policies are used for training the VAE, where the latent dimension is 2 and we estimate the policy reconstruction from the VAE.
K BROADER IMPACT STATEMENT
Our work on human-Al coordination aims to provide humanity with better cooperative AI that can adapt, understand, and generalize diverse humans. While this could enhance collaborative robots,
18

---

autonomous vehicles, and assistive technologies, method specifically improves generalization to new partners without requiring extensive human data. GOAT has the potential to improve capabilities for household robots, vehieles, and collaborative Al systems where scamless human interaction adds value to people's lives. This type of technology enhances human modeling, which can be deployed widely to help people from all walks of life and of all age groups. We would also like note that the development of such an AI system will accelerate technological progress and requires responsible development and deployment.
L LIMITATION
The effectiveness of this method depends on the training of VAE models and enough data available for VAE to learn the distribution, instead of overfitting to a few examples. It would be promising method for fields like robotics and human-Al interaction, where data availability not an issue. We also found that in sparse reward problems, this method discovers solutions and trains the cooperator robustly.
0.016
1.5
0.010
var, 2(11
0.0
0.008
0.006
-1.0
0.004
-15
-1.5
1.0
mean, 2[0]
Figure 12: After training a VAE on 4-population pool of CoMeDi agents, we visualize the generation by interpolating the dimensions of the latent vector. Particularly for this example, we choose the latent dimension those policies in the latent space. As we move from one side of the plot to the other, we can see that different cells are activated, and it covers the cooperative policies as seen in the Figure 11
2. We can see that VAE trained on cooperative policies has only encoded
19


Reviews:
[{"review_id": "1tnELdcUzJ", "reviewer": "Reviewer_Zoti", "summary": "The paper tackles the problem of learning cooperative agents that are robust to a diverse distribution of human and artificial agents. The models uses a pre-trained generative model to simulate valid cooperative agent policies with adversarial training to maximize regret. The policy itself tries to minimize regret, this induces an adversarial setup, but the challenge is to not the adversary be adversarial _and_ cooperative - two conflicting objectives. If the adversary is not constrained to such policies, then it can become overly combative, resorting to sabotage instead of making the policy robust and foolproof. In contrast to zero-sum games like chess where self-play can improve the performance of the policy by exploiting weaknesses against itself, in cooperative games there is no such incentive for a policy. \nTo prevent cross-play performance from self-sabotaging, the model essentially samples adversarial players from a learned distribution of agents using a variational autoencoder. this limits the capability for self-sabotage since the agents are sampled from a predefined distribution. This makes the contribution of the work relevant to the community.", "questions": "**Questions**\n\n1. Since the distribution of the partner policies is learned using a VAE, why cant the cooperative agent be trained to minimize regret over the entire distribution of partner policies (by sampling partner policies from the VAE) instead of using an adversarial method to sample partner policies? Is that going to be slower or faster than the proposed training in terms of environment interactions and training time? \n2. Line 376 - \"We hypothesize this is because the generative model, trained on cooperative trajectories, encodes a latent space rich in strategic variations.\" - the paper posits that the generative model encodes a rich space of strategic variations is what leads to fast convergence - this approach is also used in other prior work (some of which are mentioned in the paper). Does GOAT perform better than prior work leveraging generative policies due to its adversarial sampling? \n\nMinor questions:\n1. Line 45/74: Why is Adversary capitalized? \n2. uncommon notation for distribution, i.e. $\\Delta$ is used. This could be replaced with a better notation in my opinion\n3. Line 363: \"PBT methods rely on simulated populations of self-play partners, are often computationally expensive, and have poor coverage of actual human behaviors.\"  - why is cross play and adversarial training less expensive than self play? I thought the tradeoff was to spend more compute on adversarial training to have better mode coverage of adversarial policies. Figures 4b and 4e show that GOAT is indeed faster but is there any argument or justification as to why that is the case, especially because adversarial training can be slow and brittle.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Strengths**\n1. The VAE model and generative policy allows for sampling policies parametrically with a smooth latent space, preventing expensive zeroth-order methods to sample policies. This makes the optimization objective more tractable and inexpensive. This idea is also used in controllable image generation and editing where latent variables are optimized to achieve a test-time objective with a pretrained network. \n2. The formulation is pretty straightforward - given a cooperator policy $\\pi_C$, the adversarial policy chooses a partner policy $\\pi_P$ to minimize the value function with cross play of the cooperator and partner, while maximize the self-play performance of the partner policy - this is presumably to prevent selecting incompetent policies. This is also interpreted (accurately) as the regret of choosing a cooperative policy over itself in a cross-play scenario. Regret for bad partner policies will be lower (due to low first term in Equation (4)) than a good partner policy that does not work well with the cooperator policy. \n3. The experiment setup is satisfactory - including a cooperative matrix game allows easy visualization of one-step policies and analytically verifying optimal adversarial policies for a given cooperative policy, a 5x5 cooperative reaching game with more complexity but still tractable, and a hard game with a larger action space (Overcooked) with real-time dynamics on which performance is also shown with human cooperative players. \n4. Extensive analysis with related methods - GAMMA and MinMax is shown in RQ4 and RQ5. Figure 5 shows the adversarial objective drifting to new policies that significantly deviate from the normal distribution (in the projection space), showing that the model is indeed aiming to find harder partner policies.", "weaknesses": "1. The robustness of the policy depends on the coverage / support of the autoencoder and its simulated policies. The paper uses fixed agent populations with training methods (Line 178).\n2. The paper is not self-sufficient in terms of mentioning how the distribution of partner policies are learned before the adversarial sampling is performed. \n3. Figures 5 and 6 are nice, but they also show that the policies deviate a lot from random normal as training progresses. Since the generative model works off a VAE, are the policies that have a latent so far from the normal distribution meaningful at all? Is a regularization apart from the self-play regret (as mentioned in Line 722) enough to constrain the space of sampled policies?\n4. Figures 5 and 6 also show that the coverage of partner policies is very concentrated and lacks diversity - could it lead to a possible oscillation of the cooperative policy if certain partner policies are not compatible with each other? \n\nMinor nits: \n1. Line 75: \"The regret objective proves effective because it challenges the learning agent with a curriculum of increasingly difficult, yet still feasible tasks.\" - Regret minimization by itself does not motivate a curriculum learning approach. This line could say something like \"it challenges the learning agent and expands the frontier / coverage of harder yet feasible state configurations\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNvQav7319", "reviewer": "Reviewer_hn3Y", "summary": "The authors present GOAT, a method that generates adversarial agents to train a cooperator agent for Human-AI Coordination. The method first utilizes a GAMMA-style VAE to encode an agent population in latent space. Next, the authors propose to train an Adversary to transform the GAMMA latents to Adversarial embeddings, which is then used by the GAMMA decoder to simulate an adversarial agent. The Cooperator Agent and Adversary are jointly trained together in a minimax objective where the adversary tries to maximize the regret (SP to XP gap) between the adversary and Cooperator agent. The authors then evaluate GOAT on 3 cooperative environments, a matrix game, a cooperative reaching game and Overcooked-AI.", "questions": "- In Figure 5, the authors show that the adversarial latent vectors sampled by GOAT is significantly far away from the cluster of GAMMA latent vectors. As I understand, the GAMMA latent vectors can be interpreted as an interpolation between actual agent trajectories encoded in latent space during the VAE training process. Does this imply that GAMMA could extrapolate new agents out of the standard trained distribution?\n- Why is REINFORCE chosen over PPO to train the Adversary?\n- The authors state that the main limitation of GOAT is the reliance on a trained VAE model. Could the authors comment on what might the minimally viable population size/amount of data to train a viable VAE model for GOAT?\n- Relatedly, does the type of agents (CoMeDi vs MEP) matter when it comes to VAE training?  Does it affect the quality of latent vectors of the VAE?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The authors provide a intuitive solution to a long-standing problem in the Human-AI/ZSC community, that of generating viable adversarial cooperative agents without sabotaging effects.\n- The paper is very well written and structured.", "weaknesses": "- The authors cites two recent/concurrent related works [1] and [2] but did not compare GOAT with the two methods in the experimental sections. It will be interesting to see how GOAT compares to the two methods especially ROTATE as they also proposes a regret like objective to generate a curriculum of adversarial agents to train a cooperator agent.\n- GOAT is evaluated on fairly simple, fully observable grid-world environments which limits its generalizability to more complex environments such as those with partial observable states [3] or continuous actions [4]. \n\n[1] Wang, C., Rahman, A., Cui, J., Sung, Y., & Stone, P.  ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork. arXiv preprint arXiv:2505.23686.\n\n[2] Villin, V., Buening, T. K., & Dimitrakakis, C.  A Minimax Approach to Ad Hoc Teamwork. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems (pp. 2105-2114).\n\n[3] Gessler, T., Dizdarevic, T., Calinescu, A., Ellis, B., Lupu, A., & Foerster, J. N. OvercookedV2: Rethinking Overcooked for Zero-Shot Coordination. In The Thirteenth International Conference on Learning Representations.\n\n[4] Kang, X., Lee, S. W., Liu, H., Wang, Y., & Kuo, Y. L.  Moving Out: Physically-grounded Human-AI Collaboration. arXiv preprint arXiv:2507.18623.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P9WGDMjjgd", "reviewer": "Reviewer_HUTe", "summary": "This paper focuses on the problem of training a cooperative agent in a two player cooperative Markov game. It combines a generative procedure for sampling cooperative partner agents with an adversarial procedure for finding agents whose strategies challenge a learning agent to train an agent that is robust to diverse partner strategies while playing cooperatively and maximizing returns.\n\nThe generative procedure uses a VAE similar to previous work. The adversarial procedure works by finding embedding values that would lead to partner agents being sampled by the generator that maximize regret. This regret is calculated by calculating the difference between self-play scores and cross-play scores with the learning agent.\n\nEvaluation is done on a toy matrix game, a reaching environment, and two overcooked environments. Evaluation with human participants in overcooked show that the proposed technique, GOAT, outperforms evaluated baselines in terms of dealing with human strategies.", "questions": "* In the paragraph preceding equation 4, the paper claims that self-play is a valid proxy for optimal score without actually stating that this is a proxy. Would self play actually be optimal in all scenarios? I can imagine scenarios where the two agents need very different policies in order to act optimally. E.g. being when agents need to fulfill different roles. Perhaps this paragraph needs to be phrased more carefully.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper combines ideas from previous works in a clear and fairly principled manner.\n* The idea is explained well. Figure 1 is a good overview of the technique. Section 3 is succinct but clear, and the figures referring to the experiments are also clear and fairly well explained. Some caveats here will be expounded on the weaknesses section\n* Human evaluations give the idea a lot of credence. The idea seems to be performing as advertised in the overcooked domain.\n* The human evaluation procedure is well designed.\n* Appreciate the error bars and their explanation in Figure 4.\n* As far as I can tell, all required relevant work seems to be cited.", "weaknesses": "* Figures 5 and 6 are a little hard to follow. After reading the text in the experiments and intuiting the intention, I am able to guess what the authors are trying to show. Figure 6 can be understood well in this manner. But Figure 5 was still a little confusing due to the two different dots for GAMMA and GOAT in addition to the gradient colored dots for the episode numbers. If Figure 5 can be simplified it would get the point across much better.\n* Nit about how the value function is defined at the end of the first paragraph of Section 3: Value functions are generally conditioned on the state. Perhaps better to phrase this expression as expected returns?\n* Figure 3 was also a little difficult to look at. But I appreciate the summation across methods in the last row. Perhaps this row can be separated or highlighted so a reader can look at the summary first before going through the detailed comparison for each heuristic agent. The description would also benefit from mentioning that the maximum sum would be `11`.\n\n\nMinor: Line 83, the citation should be a `\\citep`", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)