Adaptively Coordinating with Novel Partners via Learned Latent Strategies
Authors: Benjamin Li
Shuyang Shi
Lucia Romero
Huao Li
Yaqi Xie
Woojun Kim
Stefanos Nikolaidis
Charles Michael Lewis
Katia P. Sycara
Simon Stepputtis
Venue: NeurIPS 2025 poster ()
Keywords: multi-agent reinforcement learning
human-ai collaboration
multi-agent systems
zero-shot coordination
Primary area: reinforcement_learning

Abstract: Adaptation is the cornerstone of effective collaboration among heterogeneous team members. In human-agent teams, artificial agents need to adapt to their human partners in real time, as individuals often have unique preferences and policies that may change dynamically throughout interactions. This becomes particularly challenging in tasks with time pressure and complex strategic spaces, where identifying partner behaviors and selecting suitable responses is difficult.
In this work, we introduce a strategy-conditioned cooperator framework that learns to represent, categorize, and adapt to a broad range of potential partner strategies in real-time. 
Our approach encodes strategies with a variational autoencoder to learn a latent strategy space from agent trajectory data, identifies distinct strategy types through clustering, and trains a cooperator agent conditioned on these clusters by generating partners of each strategy type.
For online adaptation to novel partners, we leverage a fixed-share regret minimization algorithm that dynamically infers and adjusts the partner's strategy estimation during interaction.
We evaluate our method in a modified version of the Overcooked domain, a complex collaborative cooking environment that requires effective coordination among two players with a diverse potential strategy space.
Through these experiments and an online user study, we demonstrate that our proposed agent achieves state of the art performance compared to existing baselines when paired with novel human, and agent teammates.

Adaptively Coordinating with Novel Partners via Learned Latent Strategies
Benjamin Li"
Shuyang Shi
Lucia Romero
Huao Li
Yaqi Xie Simon Stepputtis
Woojun Kim
Stefanos Nikolaidis
Michael Lewis"
Katia Sycara'
'Carnegie Mellon University UUniversity of Pittsburgh
University of Southern California *Virginia Tech
[benjil, shuyangs, yaqix, woojunk, sycara] Gandrew cmu edu fluciaromero, hu152, cmlewis) edu nikolaidQusc.edu, stepputtis@vt.en edu
Abstract
Adaptation is the cornerstone of effective collaboration among heterogeneous team members. In human-agent teams, artificial agents need to adapt to their human partners in real time, as individuals often have unique preferences and policies that may change dynamically throughout interactions. This becomes particularly challenging in tasks with time pressure and complex strategic spaces, where identifying partner behaviors and selecting suitable responses is difficult. In this work, we introduce a strategy-conditione represent, categorize, and adapt to broad range of potential partner strategies in real-time. Our approach encodes strategies with a variational autoencoder to learn latent strategy space from agent trajectory data, identifies distinct strategy types through clustering, and trains cooperator agent conditioned on these clusters by generating partners of each strategy type. For online adaptation to novel partners, we leverage a fixed-share regret minimization algorithm that dynamically infers and adjusts the partner's strategy estimation during interaction. We evaluate our method in a modified version of the Overcooked domain, a complex collaborative cooking environment that requires effective coordination among two players with a diverse potential strategy space. Through these experiments and an online user study, we demonstrate that our proposed agent achieves state of the art performance compared to existing baselines when paired with novel human, and agent teammates.
cooperator framework that learns to
Introduction
As an increasing number of Al agents and robots enter our daily lives, it is becoming increasingly important to explore methods of effective collaboration between agent and human. When an agent attempts to collaborate with an unknown partner (whether human or artificial), the challenge can typically be divided into two steps: (1) accurately predicting the partner's behavior, and (2) choosing actions that advance the team toward its common goal. Our approach focuses on understanding and responding to behavioral patterns that distinguish different types of collaborators. Equipping an agent with the ability to identify partner type can enable it to exploit knowledge about that type when choosing its own actions, enabling adaptation for coherent and fluid collaboration. This type of adaptation is essential, as misaligned actions between teammates inevitably decrease overall performance. The problem of adapting to a previously unknown human is an instance of the more general problem of ad hoc teamwork, cooperating with any previously unknown teammate (38). The difficulties of the problem arise if the agent begins without prior knowledge of its teammate's behavioral patterns, yet must select its own actions match the teammate's intended strategy
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

TALENTS
Behavior Embeddings
Action
Action Matching
Action Decoder
Cond.
Component &
Movement
State
Movement
Component
Action
Unknown Teammate
Figure I: Overview of TALENTS: Provided an observation of teammate, the VAE's latent strategy clusters are queried to generate action predictions. At subsequent timesteps, the teammate's actual actions are compared to these predictions, and the belief over the teammate's s latent strategy is progressively updated.
for completing the collective goal. This presents significant challenges given the multiplicity of possible strategies for any collaborative task. The difficulty intensifies human-agent partnerships, where humans typically employ diverse, non-stationary, and often suboptimal approaches that can shift rapidly during interaction. Consequently, effective collaborative agents must be capable of interpreting their partners' intentions and behaviors in real-time while demonstrating rapid adaptive responses.
In coordinated team settings, roles and specializations can be learned by agents in a centralized manner (23 43). However, human-agent teams are often decentralized, in that roles and strategies are often implicit and need to be inferred [24). To account for the diverse set of teammates an agent may encounter test-time, an effective cooperator agent this setting must be able to successfully learn and play the best response to each potential teammate's 's strategy. A popular class of methods that seeks to address this is population-basel training, in which population of agent policies is generated, from which agents are drawn to serve as teammates to train a cooperator [8] 41. At their core, these methods seek to construct agent populations with sufficient behavioral diversity span the distribution of strategies exhibited by human players, often aided by a secondary objective during population training that enforces diversity in the set of training partners 25 39 ST]. To further
increase the strategic diversity of training partners, these agent populations can be used to train a generative model, which enables sampling from continuous latent strategy distribution simulate a approaches, agent modeling methods focus on online adaptation by inferring teammate behaviors and conditioning agent actions on these inferred strategies (29] 46 50 52). However, these approaches often rely on a limited or hand-defined collection of strategy types to operate over, or lack the diverse synthetic training data provided by population- based methods. We argue that unifying these two paradigms can enable more effective team collaboration: leveraging diverse population data to learn a latent strategy space from which unique partners can be generated during training, while explicitly reasoning over teammate types at test-time to enable rapid online adaptation and best-response action selection.
much wider range of novel teammates [22]. Complementary to population-
In this work, we introduce Team A Adaptation via LatEnt No-regreT Strategies (TALENTS) (see Fig. [ a novel method for zero-shot coordination that models partner behaviors semantic strategy space. At the core of our TALENTS agent is a latent behavior embedding created by a variational autoencoder. This autoencoder is trained from offline trajectory data in the respective environment, collected from population of baseline agents. With this dataset, we train a variational auto-encoder (VAE) with the goal of predicting each agent's 's next action and utilize its latent space as behavior representation. Next, we determine appropriate number and location of behavior clusters utilizing K-Means with silhouette analysis [32]. At test time, the type of new partner, with respect to the behavior clusters, is determined by assessing each latent cluster mean's 's decoded prediction accuracy with respect to the partner's 's actual action (observed at the next timestep). Our agent's 's belief of its partner is updated according to this per-step loss, and determines the most likely partner type which then used as a conditioning term for its own action.
We evaluate our agent in modified Overcooked environment that extends prior work 2 10 45 with
additional temporal pressure through order timers and bonus rewards for fast deliveríes. This setting requires the coordinated use of three cooking stations prepare two distinct recipes, significantly

---

increasing the demands for effective teamwork and strategic coordination. We conduct experiments in both agent-agent and human- -agent zero-shot coordination settings. In agent-agent evaluations, our method demonstrates overall superior performance compared to existing baselines. Furthermore, we deploy these same agents in human studies, demonstrating their ability to collaborate effectively with diverse unseen human partners. In summary, the work introduces the following contributions:
We introduce novel method for learning -preciiy-specffic best responses for human- -gget collaboration using generative model that characterizes agents based on latent strategies.
Our method learns a strategy-onddttioned cooperator that assesses teammate type and adapts its behavior through an online regret- perretminimizationn framework.
In extensive teaming experiments, including human-agent subject study, we show that our agent achieves state-of-the performance in a challenging Overcooked environment, adapting to new partners in similar strategy space in a zero-shot manner.
2 Related Work
Collaborating with partners that were unseen by agents during training is the central goal of Zero-Shot Coordination (ZSC) [4.3 Self-play (SP) is a common approach to address this problem, in which agents establish joint strategy by pairing with a copy of itself during training [33]. These approaches have been shown to surpass human performance in zero-sum games in which agents can exploit sub-optimalities of opponents at test time 11: 37 but are often unable to achieve comparable performance in common-paye -aaooff cooperative settings since these agents tend to develop rigid behavior patterns (2). iapullttionbbssed training (PBT) methods combat this rigid convention formation by optimizing over diverse partner sets [8 41. Fictitious Co-Play (FCP) [39] leverages intermediate checkpoints of partner policies to simulate teammates of different skill levels, while other methods increase the strategy coverage of the population by means of diversity or entropy related objectives [25 30 51. cross-play performance [4] 34 or reward shaping 44
Zero-Shot Coordination
48)
E3T [47 applies the entropy objectives of PBT methods to the more efficient SP training paradigm, defining the trainer partner as a mixture between the ego policy (maximizing coordination) and random policy (maximizing entropy). GAMMA (22) further enhances the breadth of PBT strategy coverage by learning generative model over the population of policies, encoding trajectories into latent representations. They are then able to perform targeted sampling over these latents to generate novel agents spanning the strategy space of the original population. Our proposed TALENTS method applies the same approach of GAMMA by training a generative model from PBT agent trajectory rollouts, but differs in that we then cluster the learned latent space and perform controlled sampling over the clusters. We treat these clusters as a population of strategy types, training a cooperator agent conditioned on these clusters to explicitly learn role-compatible behaviors.
Strategy Inference Inferring the partner type allows agents to form beliefs about other agents in the environment and optimize its own behavior in response to those beliefs, whether in adversarial or cooperative environments. A common strategy is to learn latent embeddings of partner type, role, or
skill from a dataset of trajectories [3] 28
36
46 42. Other methods directly estimate the opponent's
policy updates [6]. perform Bayesian inference of partner type [3) 7 9 15 52), or learn a partner
model through interaction during training [16 26 47]. In MeLIBA (52) Bayesian filtering objective
is used to model confidence in the partner's 's type as the trajectory rolls out, allowing a best response agent to respond optimally given its knowledge about its opponent. Grover et al. [TT] proposes triplet loss term to encourage distinguishability between different types of opponents, and Papoudakis and Albrecht [29] expands on this by using reinforcement learning train an agent conditioned on this latent embedding. Zhao et al. [50] use apprenticeship learning to model the different types of observed strategies, then online mixture of experts select compatible policies over the course of an episode. Li et al. [99 trains a library of teammate types, each with a different role that, when faced with an unknown human, use cross-entropy to choose strategies that are highly compatible with the human's 's inferred current strategy. A separate but adjacent line of work learns Theory of Mind Our work proposes to use the same latent space that generated training partners to perform strategy inference. At train-time, the cooperator agent is conditioned on the latent cluster its teammate is generated from in order to learn ttratey-speciin best responses. At test-time, utilize tracking-regret minimized approach
models to predict partner preferences, intentions, or beliefs 20 21 31
3

---

to enable intra- -iira-epioodic adaptation, improving our agent's ability when paired with teammates that may change their strategy several times over the course of single episode.
3
Preliminaries
In this section, we establish the mathematical framework for multi-agent reinforcement learning setup conditioned on latent strategies.
3.1 Two-Player Markov Decision Process
A Markov Decision Process (MDP) is defined by tuple M = A2 P, H). While we present the two-agent formulation for clarity, this framework naturally extends an arbitrary number of agents. In this formulation, (S) represents set of states, and (A,) denotes the set of actions available to agent (i), where (i E 1, 2). The transition probability function P S A x A x S - [0, 1] defines the probability P(s'ls, a) of transitioning to state s' where joint action is taken in state s. The team reward function R S x Ai x Ag x S + I) specifies the reward R(s, 41 02 s') received by both agents when transitioning from state to state s' after executing the joint actions ai and dg. The discount factor y E [0. 1) determines the significance of future rewards, influencing how current decisions evaluated relative to potential long-tent outcomes. Lastly, H represents the horizon of the problem, which is typically omitted in infinite horizon problems.
The goal in multi-agent MDP is to find policy π S + A1, that maximizes the expected discounted sum of rewards, defined as FT 00 7 R(st, 01,t, 02,1; 8t+1)]
3.2 Hidden Parameter Markov Decision Process
Hidden Parameter Markov Decision Process (HiP-MDP) extends the standard MDP framework to incorporate unobserved parameters that influence the transition and reward dynamics. Formally, HiP- MDP is defined as tuple M(Z)- = (S, A1, A2, Z, P. P R1, R2, 7, H) where S, A, 7, and H are defined in the standard MDP. Z is of hidden parameters and : SS Ai x Ag 8 Z [0, 1] is a transition probability function that depends on the hidden parameter Z E Z. p : S x Ai X Ag Z is transition probability function for the hidden parameter 2 E representing how may evolve over the course of an episode. R is defined as S x A₁ x A2 x S Z - R, representing the reward function that depends on the hidden parameter 2 E Z.
For a specific, constant value of the hidden parameter E 2, the HiP-MDP reduces to a standard MDP M, (8,AL, A2, P(-1z), R(-1=), ?; H). Given the two-player HiP-MDP framework, the learning objective for each agent i can be formulated as:
DC
max 74
Ra(Se; 01,t 02,, 8141,2)
(I)
3.3 Tracking Regret Minimization
To address the strategic adaptations and non-stationarity inherent in our HiP-MDP setting, where teammates may shift between different strategies during an episode, we incorporate tracking regret minimization. This permits a limited number of optimal response policy changes over the course of an episode while maintaining performance guarantees. In online learning, at each time step I, an agent selects a decision from a convex set X' and subsequently observes loss function fa X - IR. Unlike static regret, which compares against single fixed decision, tracking regret compares the agent's 's performance against sequence of decisions that can change a limited number of times.
Formally, for a partition I {1, 12, ... 1m} of the time horizon [T] into m intervals, and sequence of comparators u = (n,, U2 ..., Um)) where each uj E X, the tracking regret is defined as:
T
Regi (I, u) =
Σ f((x)) - Σ Σ f((ug).
(2)
111

---

The worst-case tracking regret against any sequence with at most m expert policies (i.e. m expert changes) is
Regi(m) max Regs (I, u).
I,u
Fixed Share 12 with cetimally-selecterr switching rate a achieves the following bound for any comparator sequence of m segments:
Reg (m) V
T T(m mIn ) ) m
4 Team Adaptation via No-Regret Strategies
We introduce TALENTS, a method of training general cooperator agent that is conditioned on wide range of strategies and behaviors. In the online setting, the cooperator agent can assess the strategy of its partner and adapt to it by playing the corresponding best response policy. Given a dataset of offline trajectories exhibiting a wide range of strategies, we first use a variational autoencoder (VAE) [17 to learn a latent space of strategy vectors. We then partition the latent space into discrete clusters, with each cluster representing unique high-level strategy. Next, we train best response agent over the different strategy clusters, generating a strategic partner agent for every episode by selecting a strategy cluster and sampling from its latent mean. We condition the cooperator agent on its training partner's 's mean in order to learn strategy -deeeeeedd best response. At test time, we use the fixed share algorithm to select a latent mean condition our cooperator agent on, allowing intra-episodic adaptation to the new partner.
4.1 Strategy Learning
To form the latent strategy space, we utilize offline dataset of trajectories generated by agent-agent rollouts. Note that this data can alternatively come from human gameplay [13] [50, or a mix of human and agent data [22]. However, this work focuses on exploring ad hoc collaboration with humans without seeing human data at train time. The dataset, Diraj = { 1. is composed of observation-action pairs for each agent collected from pre-trained set of population agents, (o.") a. " increasing the amount of covered latent space over the set of possible agent types. With this dataset, we subsequently train a VAE to approximate the posterior distribution of the trajectories. The VAE architecture consists of an encoder network (=<=)/) that maps trajectories to distribution over latent strategy variables z, and a sequential decoder network po(a fr|2, o.) that predicts the next H actions given the latent variable and current observation, insight that follows from Zintgraf et al. [52). Specifically, the encoder parameterized by ø compresses the trajectory information into a low-dimensional latent space, producing parameters of multivariate Gaussian distribution (T) (T)). By predicting the // future actions of the agent, we learn a representation of the agent's 's long-term intent. Importantly, we will later be able to leverage the decoder method for generating agents of specific behavior types.
( 1T i E f1. This dataset contains a diverse set of agent strategies and behaviors
The objective of the sequential VAE is to maximize the ligeliielihood of the future actions given past observations and actions: tractable as it requires marginalizing over all possible latent variables: log po (a |o,, T 72-he)
log Po (a (0,, T=-x)). However, directly computing this is in-
log J po (Ou:t f1|2, p)) Instead, we optimize the Evidence Lower Bound (ELBO):
C(O, Φ; r) = =E Be [log (Qe:t H| o.)] - 3D (00kmmmmmmmmmm=))ppp3
(5)
Once we have learned our VAF, we cluster the latent space. To this end, we perform K-means clustering with silhouette analysis [32 determine the optimal clusters and the number of clusters in an unsupervised fashion. Each cluster represents unique agent type, with which we utilize to
train our best response cooperator agent.
4.2 Learning a Strateyy-Conditiones Cooperator Agent
To train our agent, we utilize the strategy clusters and leverage the generative capability of our trained VAE. We retrieve actions for each agent type by sampling from the latent mean of each strategy cluster
5

---

and decoding them, conditioned on the observed environment state. For each episode in the training process, we randomly select one such cluster to sample from, using the priority-based sampling technique proposed in Zhao et al. [51] Using categorical variable corresponding to the partner cluster, we learn bias vector corresponding to the cooperator's ' action space. The actor network's output logits are then biased by this vector, explicitly encouraging or dissuading the cooperator from taking certain actions depending on the partner type (see algorithm []. This motivates the cooperator to learn the unique conventions needed to best respond to its partner (Note that we utilize high-level action space in our experiments). We train our cooperator agent using independent PPO 5 35
Algorithm 1
Cooperator Training
Input: Trained VAE encoder (g(2)7), decoder po(0uc+H|2 H12, strategy (,
clusters
.......,.((KK, ok) ), priority distribution p(c)
2: Initialize cooperator policy me(-)o,
3: for each training episode do
4: Sample a strategy cluster e ~ p(c)
5: Sample latent strategy
N (Me,
6: for each timestep / in episode do
7: Observe state On for cooperator agent Compute action bias vector be from cluster embedding matrix E, where be Elc] Compute action logits from actor network 4 fo)
8:
9:
Apply bias logits: = 1 + be
11:
Sample action from biased distribution a ~ softmax(1)
12:
Sample generated action for partner ae ~ Po (. 0z) Execute actions and collect rewards
13:
14:
end for
15:
Update Te using PPO with collected trajectory
16:
Update priority- baaod sampling weights using total episodic return
17:
end for
18:
Return: Strategy cooperator policy
4.3 Online Adaptation to Novel Partners
The trained cooperator agent will be able to best respond to any partner strategy the VAE distribu- tion, provided that it has knowledge on which latent strategy the partner is playing. This adds the additional challenge of needing infer the partner's type the online setting. An approach that has been explored in previous research is to encode the novel partner's 's trajectory segment into the latent space and conditioning the cooperator on that 13 46). but in practice distribution shift between train-time and test-time trajectories can lead to brittleness suboptimal encoded representations, particularly when playing with humans. Instead, we utilize regret minimization and a no-regret algorithm to represent the partner's behavior.
We implement a variant of the fixed-share algorithm infer the partner type, setting each strategy cluster as an "expert". At test time, sample a latent representation from each cluster and decode into the predicted action, conditioned on the current observation. At the next timestep, we calculate the incurred loss by comparing the previous timestep's prediction with the actual action of the partner, then update the weights of each expert accordingly. Importantly, the fixed-share algorithm differs from standard static no-regret methods (such as Hedge [8). FTRL (27), in that it minimizes the regret given that the expert may switch policies m - times during an episode (algorithm) 2). This is to account for the fact that policies at test-time may be non-stationary and might change latent strategy over the course of the episode, due to influence from its partner or continuous adaptation and learning from the task. We examine the effect of fixed-shan versus static-regret minimization in Figure
Fixed-share allows the agent to model these strategy changes and adapt its behavior during the episode. It also provides the regret bound that if the agent encounters a truly novel strategy at test time, the agent will only perform as poorly as the best-response to the closest observed strategy during training.

---

Algorithm 2 Online Adaptation to Novel Partners with Fixed-Share
1: Input: Latent clusters (Win,, N (2) --.-- -. (ux
trained cooperator policy To(: c), VAE decoder po. switching parameter a E (0, 1), learning rate >> 0
2: Initialize weight vector w (1/K .... 1/K) uniformly over K experts
3: for 1,2,... do
4: Observe current state O:
5: for each cluster с 2........ K) do
6: Sample latent strategy 2e (NMe I
7: Predict partner action aj = arg maxa po (alze, 0t)
8:
end for
9:
Compute leading expert c* = arg max, wi
Execute cooperator action at To 1000 c²) Observe partner's 8 actual action at Compute loss for each expert: &t
11:
log po (a 2c 0z)
13:
Update pre-sharing weights: tH = w¹ exp( -npt)
Normalize: w(+1 w Ee= wet1
15:
Apply fixed-share update: w (1 - aa + / /K
/
16:
end for
Forced Coordination
Open
Hallway
Ring
Figure 2: The four Overcooked layouts used in experiments.
5 Experiments
We evaluate our algorithm in a modified version of the Overcooked-ai environment across agent- agent and human- agent gameplay. In this domain, evaluate across four layouts (see Fig. 2 for agent-agent teams, and three for human-agen teams. These layouts are conceptually similar to those proposed in (2), but feature additional task complexity and timing constraints further highlight the need for effective teamwork.
5.1
Agent-Agent Zero-Shot Coordination
We evaluate TALENTS in agent-agent settings using three different agent populations: Fictitious Co-Play (FCP) 39
Maximum Entropy Population (MEP) [51]. and Behavior Preference (BP) Agents [44). For the FCP and MEP populations, eight self-play agents are initialized with different seeds and trained. The MEP population is trained with an additional population entropy loss [51]. The final FCP and MEP populations consist of the final policies as well as two intermediary policy checkpoints per policy. As stated by Wang et al. (44) well-designed population of behavior preference presents behavior diversity that is more similar to that of human teams, compared to other population generation methods. Therefore, we select it as another training population and train 24 behavior preference (BP) agents to cover wide spectrum partner strategies. The BP is represented by a linear combination of event-based shaped reward terms that encourage the agent to learn various behaviors. Using these populations, we compare our method to two baselines: the population-trainn trained best response cooperator (BR), and GAMMA cooperator. Notably, we select GAMMA
7

---

Table Resulting scores of games played with a held-out set of 12 behavior-preferenced SP agents (mean + standard error).
Pop
Agent
Open
Hallway
Forced-Coore
Ring
TALENTS
710.36 88.75
635.59 = 107.54
34.38 6.59
596.19 33.34
FCP
GAMMA
616.67 + 14.99
537.60- = 26.75
38.36 + 6.65
395. 03 = 10.09
BR
427.07 + 14.17
366. 14 = 70.50
56.09 + 12. 33
288.61 = 31.85
TALENTS
720.84 + 72.63
640.51 + 27.36
36.53 + 2.58
568.11 11 = 8.99
MEP
GAMMA
682.38 + 14.99
575.23 + 21.17
31.89 + 4.33
369.15 + 12.47
BR
437.00 + 27.76
440.48 113.03
50.40 14.29
335.61 + 28.48
TALENTS
842.39 + 36.47
642.94 + 81.00
56.55 + 9.09
647 62 + 16.96
BP
GAMMA
573.93 52.69
513.47 - 34.06
47.52 2.61
387.58 1 17.41
BR
469.88 + 52.01
454.26 86.5
78.53 8.55
640.43 + 28.06
INNS
GAMMA
Oyen
Fancel Cancinttinn
1500
Revaad
Rowand
Rovenn
200
800
600
250
200
204
Stop
Skor
Skep
Step
Figure 3: Evaluation performance during training for each layout, averaged across agents trained with FCP, MEP, and BP populations.
baseline due to the method also leveraging a generative model in its cooperator training process. Through our experiments, wish to evaluate if our strategy-specifte agent generation and cooperator conditioning will result a more robust adaptive cooperator agent.
To generate the trajectories for VAE training, we select agents from the population to form team and perform several joint rollouts, repeating until joint trajectories between all the agents in the population is achieved. This set of trajectories is used to train VAEs for both TALENTS and GAMMA.
To compare the performance of the methods with unseen agent partners, we evaluate using held-out set of behavior preference agents representing a diverse span of strategies and competencies (see Tab. D. We find that with the exception of the Forced Coordination layout, TALENTS able to achieve the highest reward out of the evaluated methods, regardless of the population it was trained on. We hypothesize that both TALENTS and GAMMA are less effective than the population BR agent in Forced Coordination due to there being very clear partition of duties between the teammates. GAMMA and TALENTS, which focus on exploring the diverse space of possible partners (many of which exhibit ineffective or uncooperative behavior), receive less reward signal when paired with agents of lower skill and are, as a result of the sparser training, less proficient the overall task than the BR agent.
To evaluate the efficacy of the tracking-regret minimization framework, we ablate TALENTS by replacing the fixed-share algorithm with a static-regre minimizer. We use exponential weights to keep the weight update structure constant, only ablating the weight-sharing component. We then evaluate our cooperator with partner policy randomly selected from a held-out behavior preference set, but replace the partner agent with a policy with a different behavior preference halfway through the episode. We see in Fig. that the static-regret ablation is unable to update its belief to the new partner, and suffers lowered reward in the latter half of the episode as a result.
All models were trained on a server with 2x AMD EPYC 7713 64-core processors, 1.08 TB of system memory, and 5x Nvidia RTX 6000 Ada GPUs. In our experiments, VAE train time is typically 4-6 hours, while a cooperator agent can be trained in approximately 24 hours.
5.2 Human-Agent Zero-Shot Coordination
In this section, we evaluate our proposed method against state-of-the-arr multi multi-agent coordination baselines with real human participants. Data collection is done online crowd-sourcing platforms
8

---

TALENTS
TALENTS-SiaR
Hallway
Forzed Coodination
Cuuntatvv Rerord
303
100
Befor
Figure 4: Accumulated reward with partner policy swap midway through the episode (t 1200). Error bars are standard error from the mean.
Prolific and Cloud Research. Each participant completed three rounds of gameplay, each lasting for four minutes, with different agents as the partner. The entire session took around 30 minutes to complete and participants received $7 base payment plus up to $3 bonus depending on their performance. For these experiments, we limited the action frequency of agents to match that of human capability. Team score and subjective ratings recorded as the performance metrics.
5.2.1 Experiment Design
A mixed experiment design is used for the human-agent evaluation, where agent type is the within subject variable (three levels: TALENTS, GAMMA, BR) and the map layout the between subject variable (three levels: hallway, open, forced coordination). At the end of each round, participants asked to complete a set of questionnaires measuring their subjective preference for the agent partner they just interacted with. The questionnaire consists of five surveys, each measuring their workload (NASA-TLX), perceived team fluency, trust, coordination, and satisfaction on 7-point Likert scales.
5.2.2 Results
We collected data from a total of 119 participants. To control data quality, we filtered out team trajectories from inactive participants and survey responses from those who incorrectly answered the trap questions, As shown in Fig our agent significantly outperforms both baselines in team score and subjective ratings. Mixed ANOVA tests show significant main effects for team score (F(2, 166) 5.76, .003), with our agent achieving significantly higher score than both baselines post-hoc comparisons (p .05). Similarly, our agent receives higher subjective ratings in team fluency (F(2, 122) = 4.31, .02) and perceived trust (F(2, 122) 3.23, = .04) compared to than the BR baseline (ps .05).
Hallway
Eoondration
Scree by Layes!
100
Figure 5: Human agent teamwork evaluation comprises team scores and participants' subjective
ratings of agent teammates. Perceived workload (shaded) is lower if better. Statistically significant differences between agents are marked by asterisks. Error bars are one standard error from the mean.

---

Conclusion
In this paper, we proposed TALENTS, a method for training a strategy-conditiones intra-episodic adaptive cooperator via generative teammates. Our approach combines variational autoencoder to learn a latent strategy space, unsupervised clustering to identify distinct strategy types, and a fixed-share algorithm for online adaptation. The experimental results in the Overcooked environment demonstrate that our strategy-conditionel cooperator effectively identifies and adapts to partner strategies, achieving better performance than naive cooperators and significantly outperforming self-play agents trained with different partners. Our approach addresses key limitation of existing methods by enabling intra-episodic adaptation to previously unseen and continuously changing partner strategies at test time. By representing strategies in a continuous latent space and clustering them into discrete types, we create best-response mapping that allows for flexible adaptation. The fixed-share algorithm's 's ability to account for non non-stationary partner behavior further enhances our method's robustness dynamic collaborative settings.
6.1 Limitations
Our approach shows promising performance in the Overcooked environment but has limitations. It struggles on the Forced Coordination layout due to sparse reward signals when paired with ineffective or low-skill partners, which additionally causes the priority sampling method to bias toward choosing those partners in future episodes. Generalization to novel teammates is limited by the VAF's interpolation capabilities, meaning our agent can only truly generalize new teammates if they exhibit similar behaviors to the training data. However, we demonstrate that our agent can successfully play with humans without having been trained on real human data. Furthermore, to better align with human gameplay, which requires real-time interactions, we artificially slowed down rates. While agents act at every step, with the game agent actions to match human humans typically only take 3-4 actions in the same timeframe. designed for ten steps per second, This slowdown may hinder agents' long-term planning capabilities.
References
[1] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365 (6456):835-890, 2019.
[2] Micah Carroll, Rohin Shah, Mark Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
[3] Georgios Chalkiadakis and Craig Boutilier. Coordination in multiagent reinforcement learn- ing: A bayesian approach. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems, pages 709-716, 2003.
[4] Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. Generating diverse cooperative agents by learning incompatible policies. In The Eleventh International Conference on Learning Representations, 2023.
[5] Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson, Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 2020.
[6] Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with epponent-learning awareness. Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pages 122-130, 2018.
[7] Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian action decoder for deep multi-agent reinforcement learning. International Conference on Machine Learning, pages 1942-192 1951. PMLR, 2019.
10

---

[8] Yoav Freund and Robert E Schapire. A and an application to boosting, Journal of computer and system sciences, 55(1): 1et::199-133 139, 1997.
generalization of on-line learning
[9] Haobo Fu, Ye Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jiechao Xiong, Ying Wen, Kai Li, Junliang Xing, Qiang Fu, et al. Greedy when sure and conservative when uncertain about the opponents. In International Conference on Muchine Learning, pages 6829-6848, PMLR, 2022.
[10] Tobias Gessler, Tin Dizdarevic, Ani Calinescu, Benjamin Ellis, Andrei Lupu, and Jakob Nicolaus Foerster. Overcookedy2: Rethinking overcooked for zero-shot coordination. In The Thirteenth International Conference on Learning Representations.
[11] Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International conference on machine learning, pages 1802-1811, PMLR, 2018.
[12] Mark Herbster and Manfred K Warmuth. Tracking the best expert. Machine learning. 32(2): 151-178, 1998.
[13] Joey Hong, Sergey Levine, and Anca Dragan, Learning to influence human behavior with offline reinforcement learning. Advances in Neural Information Processing Systems, 36:36094-36111 2023.
[14] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In International Conference Machine Learning, pages 4399 4410. PMLR, 2020.
[15] Dana Hughes, Akshat Agarwal, Yue Guo, and Katia Sycara. Inferring non-stationary human preferences for human-agent teams. In 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pages 1178-1185 IEEE, 2020.
[16] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning. pages 3040-3049. PMLR, 2019.
[17] Diederik Kingma, Max Welling, al. Auto-encoding variational bayes, 2013.
[18] Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017.
[19] Huao Li, Tianwei Ni, Siddharth Agrawal, Fan Jia, Suhas Raja, Yikang Gui, Dana Hughes, Michael Lewis, and Katia Sycara. Individualized mutual adaptation in human-agent teams. IEEE Transactions Human-Machine Systems, 51(6):706-714, 2021. doi: 10.1 10.1109TTHMS. 2021.3107675.
[20] Huao Li, Ini Oguntola, Dana Hughes, Michael Lewis, and Katia Sycara. Theory of mind modeling in search and rescue teams. In 2022 31 IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pages 483 489, 2022. doi: 10.1 MMAN553752.0022999000
[21] Huao Li, Yu Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Charles Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pages 180-192, 2023.
[22] Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S Du, and Natasha Jaques. Learning to cooperate with humans using generative agents. Advances in Neural Information Processing Systems,
2024.
[23] Yuntao Liu, Yuan Li, Xinhai Xu, Yong Dou, and Donghong Liu. Heterogeneous skill learning for multi-agent tasks. Advances neural information processing systems, 35:3701 1-37023, 2022.

---

[24] Dylan P Losey, Mengxi Li, Jeannette Bohg, and Dorsa Sadigh. Learning from my partner's s actions: Roles in decentralized robot teams. Conference on robot learning. pages 752-765. PMLR, 2020.
[25] Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International conference on machine learning, pages 7204-7213. PMLR, 2021,
[26] Long Ma, Yuanfei Wang. Fangwei Zhong, Song-Chun Zhu, and Yizhou Wang. Fast peer adaptation with context-aware exploration. In International Conference on Machine Learning. pages 33963-33982. PMIRR 2024.
[27] Brendan McMahan. gollow-the-rggularizee--eee and mirror descent: Equivalence theorems and 11 regularization. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 525-533. JMLR Workshop and Conference Proceedings, 2011.
[28] Stefanos Nikolaidis, Ramya Ramakrishnan, Keren Gu, and Julie Shah. Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In Proceedings of the tenth annual ACM/IEEE international conference on human-robel robot interaction, pages 189-196, 2015.
[29] Georgios Papoudakis and Stefano V Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001.10829 2020.
[30] Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Ef- fective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33: 13:11805-11 10050-18062, 2020.
[31] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference on machine learning, pages 4218-4227, PMLR, 2018
[32] Peter Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
[33] A. L. Samuel. Some studies in machine learning using the game of checkers. IBM Journal of Research and Development, 3(3):200-229, 1959. doi: 10.1 10.1147/rd.33.02100
[34] Bidipta Sarkar, Andy Shih, and Dorsa Sadigh. Diverse conventions for human-ai collaboration. Advances in Neural Information Processing Systems, 36:23115-231333 2023.
[35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[36] Andy Shih, Stefano Ermon, and Dorsa Sadigh. Conditional imitation learning for multi-agent games. In 2022 17th ACM/IEEE International Conference on Tlmman-Rober Interaction (IRRI), pages 175-175. IEEE, 2022.
[37] David Silver, Julian Schritwwisser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
[38] Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1504-1509, 2010.
[39] DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515. 2021.
[40] Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. In International Conference Learning Representations.
12

---

[41] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350-33 354, 2019.
[42] Chen Wang, Claudia Pérez-D' Arpino, Danfei Xu, Li Fei-Fei, Karen Liu, and Silvio Savarese. Co-gail: Learning diverse strategies for human-robot collaboration. In Conference on Robot Learning, pages 1279-1290 PMLR. 2022.
[43] Tonghan Wang. Heng Dong, Victor Lesser, and Chongjie Zhang. ROMA: Multi-agent reinforce- ment learning with emergent roles. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9876-9886. PMLR, 13-18 Jul 2020.
[44] Xihuai Wang. Shao Zhang, Wenhao Zhang, Wentao Dong, Jingxiao Chen, Ying Wen, and Weinan Zhang. Zvc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination. Advances in Neural Information Processing Systems, 37:47344 47377, 2024.
[45] Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cognitive Science, 13(2)411-432, 2021.
[46] Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. In Conference on robot learning, pages 575-588. PMLR, 2021.
[47] Xue Yan, Jiaxian Guo, Xingzhou Lou, Jun Wang, Haifeng Zhang, and Yali Du. An efficient end- to-end training approach for zero-shot human-ai coordination. Advances in Neural Information Processing Systems, 36:2636-2658, 2023.
[48] Chao Yu, Akash Velu, Fugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems, 15:24611-246224 2022.
[49] Ceyao Zhang, Kaijie Yang, Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song- Chun Zhu, et al. Proagent: building proactive cooperative agents with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17591-17599, 2024.
[50] Michelle Zhao, Reid Simmons, and Henny Admoni. Coordination with humans via strategy matching. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9116-9123. IEEE, 2022.
[51] Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6145-6153,
2023.
[52] Luisa Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann. Deep interac- tive bayesian reinforcement learning via meta-learning. In Proceedings of the 20th International Conference Autonomous Agents and MultiAgent Systems, AAMAS '21, page 1712-177 1714, Richland, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 450383033
13

---

A Human Experiment Details
A.1 Experimental Design
Human study participants were initially acquainted with the premise of the domain via series of instruction screens (Figs A.T A.2
A.3). then had the opportunity to test the game mechanics through
single-playe tutorial session (Hig A.4.. In the gameplay phase of the experiment (Fig (A.4b). participants were randomly assigned to one of the three human-study layouts (open, hallway, forced coordination). Each participant then played three games, one with each evaluated agent (TALENTS, GAMMA, and Population Best Response), presented in randomized order to control for sequence effects. Episode length was = 2400 environment timesteps (4 minutes). Many previous works in the Overcooked domain use game length of T 400 or T 600 [4 22 39 2 We selected longer game-length both due to the longer-horizon horizon tasks of the modified Overcooked environment as well as better compare the intra-episodic team adaptation capability of the algorithms and reduce the amount of randomness our results.
Upon completion of each game, participants were asked to complete post-game survey, consisting of NASA-TLX, team fluency, team trust, coordination, and satisfaction surveys, shown in Figure A.5
A.2 Online Data Collection Instructions and Interface
Collaboration User Study
Section Consent
Hole
Focu checos la seficbur this raes reiniil siv cecenndie cotlers the ciftoo came offedt Descodded
sncthe perccaad stat) auches
Stea Khouse Rules
FMOS in reb you wil jau venion Ove sonded, witelnng villo weitheenehrr
surver
tar sen cal hott in destanday Ta un muuktngg the slins batten
reading inttocdary, Craw
This eere rtayy mese mas 3a--i amare compees
You suse E Alt up parto munvir tenus to you ficcealio
souy nomiined to weentienns heard
provectl ig tivitat,, youre
tavv rectien
Pletform information
Part Layout
Wiich platkem de and stade:
what one M9Ve OF WITE RUTTDAT oigects USDNIS nAN:
Priffe
Chaud Tammert
Demographics
Damar
takaatorr Mamaee
Ingredient spensers:
Have gara Ooeccallet
Tools and landac:
Poe Det su
How lo wirk ynu uhy oe intract
up forc fine
Delleary 27a3
(a)
(b)
Figure A.1: Online data collection instructions.
14

---

Part Understanding the Game
Score
stddow neerting &e audie raer tnme pen
ddrers and Delivery
Cerohee dul by peihiiigg and fnfire feyitng t'e dafie the coecille lop de--iiio Iegievix
ahariiin cocato
Araormue Aichhaic
Dumckk
Controllnng Players (MM)
dhen let rAAl the TETRONENHT Ressnie
Aroekem: New you
No WOUITEE baut gaar tano
no ene diglle Fhe depatlc, the deci silower arall yourel Tamal
Space inad wif aoaltss
boflemie soven.
HEEVAAL
aeaccaa buch
taae avainng loi
manng paio cuacta
Part 4: Racipas
lerit prl
Seak Clean stane chopped sbokk belaed
Orders
steak
Mose burrec Clean place clippeed cooted mushrocre bolled
Time Left 105
fillk cillacca crend and dingi offerr Bay cer them coarters peseedd
(a)
(b)
Figure A.2: Online data collection instructions.
15

---

Part Exampli Bubstasko
Vhen
Figure A.3: Online data collection instructions.
fuards
Soore Tine 112
Two-player (agent in blue, human in red) actual data collection session (hallway layout).
(a) Single-player tutorial session.
Figure A.4: Online data collection gameplay interfaces.
16

---

Post-game Survey
NASA-TLX
Crw MON
HoN hard you www.ooooooooeeoooo your nosi mo
Team Fluency
Clergg opent
contriturne on and better performance
The Nonn's Buenry regroved over
Playyl Hpme
12
Slargs
Section 3: Teom Trust
Inemmod Inateattyy
9
Saction St Satisfaction
1. Littared the grenpioy eperteece.
Section Co-ordination
Chregg
spent
berker
Chrough spenn
look war encoainn EREEESV
Sebent Scoee
(b)
(c)
17
Figure A.5: Post-game survey questions.

---

B
Training Details
B.1 Population Training
For the Behavior Preference 44 separated into three distinct categories. Each preference has the option of an "encouraged" reward shaping term well "discouraged" term. For cach category, we train agents with every permutation of the preferences (8 agents per category, 24 agents total), A best response agent is trained in self-play with each of the behavior- behavior-preferencee preferenced agents. We use these agents the base population in human experiments following Wang et al. [44's ' observation that these agents cover a more diverse strategy space than other population methods (MEP, FCP).
agent population, we define a set of nine tunable preferences,
Preference
Reward Shape
Plating Ingredients
30, 20
Washing Plates
-30, 20
Delivering Dishes
-30, 20
Chopping Ingredients
-30, 20
Potting Rice
-30, 20
Grilling Meat/Mushroom
-30, 20
Taking Mushroom From Dispenser
-15, 10
Taking Rice From Dispenser
-15,
Taking Meat From Dispenser
-15, 10
Table A.I: Event-based BP features and corresponding reward design.
For MEP 51 and FCP 33 agent populations, we train eight unique agents, saving two intermediate
policy checkpoints ( of total training iterations) and the final policy form a total population of 24 policies.
B.2 Encoder Training
As mentioned in the main text, we pair agents in the population and perform several joint rollouts form the trajectory dataset. For FCP and MEP populations, we collect trajectories from every combination of policies in the population. For the BP population, we play each preference agent with its self-play best response. In total, each dataset of trajectories contained between 500 and 700 agent games with 2400 timesteps each. Encoder training byperparameters are listed in table and were tuned via simple linesearch. Encoder input observations were in the form of 26-channel egocentric lossless state representation, one of the standard options of the Overcooked domain. KL weight 3 was linearly annealed over the course of training.
B.3 Cooperator Training
Cooperator agents were trained via Independent PPO. though other RL methods would likely also work. Action masking was applied during both train and test time to ensure agents chose valid actions. To encode the cluster information the TALENTS agent, an action biasing approach was used. We experimented with using an embedding layer to encode the cluster parameters and appending that the agent's 's observation, but found that approach be less effective learning distinguishably different behaviors for different types of partners, and as a result, had worse performance. Training test the belief update and expert switching mechanism
hyperparameters can be found table of fixed share, we tracked the distribution of weights over time when switching the cooperator's partner's policy for one with a different strategy during an episode (same premise as the ablation study shown in 4). As shown in A.6 we the best predicted expert change over time.
18

---

Table A.2: Role Encoder Training Parameters and Model Configuration
Parameter
Default Value
Description
Model Architecture
Latent Dimension
8
Dimension of the latent role space Length of trajectory input sequence Future action prediction steps
Window Length
50
Prediction Horizon
50
CNN Feature Extructor
Input Channels
26
Channels in state representation Convolutional layer filter counts Convolution kernel dimensions Convolutional layer activation
Conv Layers
[16, 32, 32]
Kernel Size
3 x 3
Activation
ReLU
Sequence Encoding
Action Embedding Dim
8 GRU 256 CNN + Action
Discrete action embedding size Sequence processing architecture Temporal encoder hidden dimension Combined feature input
Temporal Encoder
GRU Hidden Size
GRU Input Size
Encoder Network
Hidden Layer Size
256 2x Latent Dim ReLU
Encoder hidden dimension
Output Layer
Mean and log-variance outputs Hidden layer activation
Encoder Activation
Decoder Network
Decoder RNN
GRU
Future sequence decoder
Decoder Hidden Size
256 Latent + CNN Linear
Decoder RNN hidden dimension
Decoder Input
Combined latent and observation features
Action Predictor
Final action classification layer
Training Hyperparameters
Batch Size
512
Training batch size
Number of Epochs
100
Total training epochs
Learning Rate
5 x 10-4
Adam optimizer learning rate Initial B-VAE regularization weight
Start
0.0
3 End
0.05
Final 3-VAE regularization weight Data split ratio (%)
Train/Validatio Split
80/20
Saitshing Effaci Clusaan Wasiglai
Varchi Mox of Wagglt 2.2. (--00
6
teighh
12:00
Figure A.6: Cluster Weight Distribution With a Partner Policy Swap Over Time
19

---

Table A.3: Cooperator Policy Training Parameters
Parameter
Value
Optimization Parameters
Total Training Steps
4e7
Training Batch Size
38,400
Parallel Environments
16
Learning Rate Schedule
[10-3 10-1 10-9
GAE Lambda (X)
0.99
KL Coefficient
0.5
PPO Clip Parameter
0.2
Value Function Loss Coefficient
0.5
Entropy Coefficient
0.02
Gradient Clipping
1.0
Discount Factor (7)
0.995
Fixed Share Parameters
Learning Rate
0.2
Sharing Parameter (a)
0.4
Regret Clipping
1.0
Network Architecture
Actor Embedding Dimension
392
Critic Embedding Dimension
392
Cluster Embedding Dimension
32
Action Space Size
27
Observation Channels
12
Action Bias Weight
2.0
Additional Experiments
C.1
Additional Baselines
The primary motivation of this work was to develop state-of-the-art method for training cooperator using simulated data generated by agent populations. As such, the baseline algorithms used for evaluation (GAMMA, BR) were selected as they are representative SOTA methods for population- based training, and enable us isolate the impact of our novel training and adaptation methodology. However, it is still valuable to compare against other leading zero-shot coordination methods, notably ones which also utilize strategy inference or partner modeling modules.
We evaluate against E3T, a self- -lay method that uses partner modeling module to anticipate its teammate's future intentions, and PACE, an adaptation framework that injects peer identification reward term to encourage agents to choose actions which allow them to form more accurate estimates about their teammates' types. We report the results of preliminary experiments in A.4
We find that TALENTS remains the top performer out all the assessed methods, while E3T is able outperform the remaining baselines. PACE underperforms, achieving comparable scores the population best response baseline. This can likely be attributed to the fact that PACE an effective inter-episodic adaptation framework. However, in zero-shot coordination problem setting, PACE unable to form its belief the short time frame necessary for high performance, and therefore achieves similar score as the naive population best response baseline. In contrast, TALENTS is able to perform intra-episodic adaptation due to its tracking regret minimization module.
20

---

Algorithm
Score (# Std. Err.)
E3T
548.66 = 25,44
PACE
313.34 18.48
GAMMA
374.10 = 24.99
BR
343.71 24.27
TALENTS
605.88 + 27.89
Table A.4: Average performance over the evaluated layouts, comparing the additional baselines of E3T and PACE.
C.2 Effeet of Cluster Count
We use silhouette analysis in order to partition the VAE latent space into discrete strategy clusters in an unsupervised manner. However, there no theoretical guarantee that the optimal cluster count as determined by silhouette analysis will result in optimal performance when those clusters used to train a TALENTS cooperator. To empirically evaluate our approach, we conduct a sensitivity analysis over cluster count, training and evaluating cooperator agents using various numbers of latent clusters. The results of the sensitivity analysis are shown in A.5 while the corresponding silhouette scores shown in A.7 We find that the silhouette optimal number of clusters achieves the highest score, suggesting that silhouette scores are an appropriate metric to use for this application.
Silhouette by Number Clusters
0.24
0.23
0.22
0.21
0.20
Siloouite
0.19
0.18
0.17
0.16
0.15
2.5
12.5
15.0
17.5
20.0
Number of Clusters
Figure A.7: Silhouette analysis of a VAE latent space in open layout. Higher silhouette score indicates more optimal clustering, with 11 being the optimal number for this latent space.
Number of Clusters
Score (# Std. Err.)
650.83 28.00
2
701.68 # 11.93
4
620.35 39,01
8
740.93 # 79.15
11
865. 15.11
14
724.94 # 40.60
16
648.57 26.26
Table A.5: Effect of different numbers of clusters overall performance of TALENTS cooperator, evaluated the open layout.
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We make the following claims:
develop a new method that adapts to agent teammates in Overcooked and achieves SOTA performance. This is shown in Table and holds true for three out of four tested environments.
We claim that our agent can successfully adapt to human players, despite being trained only on agent-agent data. This is shown in our human subject study shown in Figure 5 demonstrating that our agent performs the best (score wise) and is preferred by users (according to the four NASA TLX scores)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss our limitations in Section 6.1 Fundamentally, approach is limited by the generalization capability of the VAE. Furthermore, additional investigations into the coordiation failure in the "Forced Coordination" layout is required.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approxinations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
22

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not provide any theoretical results or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The answer this question nuanced, but largely [Yes]:
We provide the code for our proposed agent, as well as instructions of how to re-train our TALENTS agent
We do not release our offline dataset train our VAE, however, code outlines how it can be regenerated
Due to anonymity and size constraints, instructions the code should be sufficient to reproduce our agents
provide pretrained model, however,
Due to IRB constraints, we are unable to provide the collected human-agent games
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
23

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This work does not provide a dataset. The dataset used to train our VAE can be re-generated using our provided code. The anonymously released code for review contains instructions how to re-rum experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https www.bbmiissiooooooooioo for more details.
//nips htipss.c//uulic//uiiice httpa///gguideee/deeubbbbiiiii
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Data splits, hyperparameters, and training details are disclosed the supple- mental material and anonymous code release.
Guidelines:
The answer NA means that the paper does not include experiments.

---

The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide error and evaluations with respect to the statistical significance of our results in Table I Figure 4 and Figure 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details of the utilized compute hardware in section 6.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////eeurip....c//lle//////
Answer: [Yes]
Justification: Our human-subject study IRB approved and conforms to the NeurlPS Code of Ethics. We obtained consent and compensated participatns for their time and effort.
25

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The research conducted in this work is fundamental research that has no direct societal impact, positive or negative.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not provide high-risk model, nor is any data released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
26

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We the original owners of our method, TALENTS, and have cited the modified Overcooked [? benchmark utilized in this work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: We anonymously release our code for review (see abstract for anonymous GitHub code).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [Yes]
Justification: We have provided details about the human subject study conducted in this
work in Section 5.2 The conducted surveys are provided in Section 5.2.1 5.2..
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
27

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: The human experiments conducted in support of Section ?? are IRB approved and follow the NeurIPS Ethics Code.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is required.
Answer: [NA]
Justification: LLMs have been utilized for light editing and formatting.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLMM policy for what should or should not be described.
28


Reviews:
[{"review_id": "RfhgVFGEz3", "reviewer": "Reviewer_mm8i", "summary": "The paper introduces **TALENTS**, a framework for zero-shot coordination in cooperative multi-agent settings by learning a latent strategy space via a variational autoencoder, clustering behaviors into discrete types, and training a strategy-conditioned cooperator agent. Online adaptation to novel partners is achieved through a fixed-share regret-minimization algorithm that dynamically infers and switches between latent strategy experts during interaction. \n\nHowever, **the core innovation**—combining a VAE-based embedding with clustering and regret minimization—**offers only an incremental advance** over existing latent-embedding and online adaptation methods, and **the experimental design** (single benchmark domain, limited baseline set, and hand-tuned cluster counts) **is insufficiently comprehensive to meet NeurIPS standards**.", "questions": "**Questions for the Authors**\n\n1. **Cluster sensitivity**: How does performance vary with different choices of latent dimensionality and cluster count? Can you provide an ablation?\n\n2. **Cross-domain generality**: Can TALENTS be applied to other cooperative tasks (e.g., Hanabi, pursuit–evasion)? If not, what modifications would be needed?\n\n3. **Real-time feasibility**: What is the per-step inference time for strategy inference and cooperator action selection, and how does this scale with the number of clusters?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. **Clear technical exposition**: The methodology (VAE training, K-means clustering, strategy-conditioned PPO, fixed-share adaptation) is described in a coherent, step-by-step manner.\n\n2. **Human-agent evaluation**: Inclusion of a human subject study with statistical tests (mixed ANOVA, p-values) adds practical validation beyond purely simulated benchmarks.\n\n3. **Implementation details**: The manuscript provides computational resource specifications and training times, aiding reproducibility.\n\nWeaknesses:\n1. **Incremental novelty**: Key components (latent strategy embeddings, clustering, expert weighting) closely mirror prior work (e.g., GAMMA, MeLIBA) without a fundamentally new algorithmic insight.\n\n2. **Limited baselines**: Absence of comparisons to stronger or more diverse adaptation frameworks (e.g., Bayesian meta-task reuse) weakens claims of state-of-the-art performance.\n\n3. **Single-domain focus**: All experiments are confined to Overcooked; generalization to other collaborative tasks is untested.\n\n4. **Cluster selection**: Reliance on K-means with silhouette analysis lacks robustness; no sensitivity analysis over the number of clusters or VAE latent size is provided.\n\n5. **Computational cost**: While training times are given, there is no discussion of inference latency, which is critical for real-time human–agent interaction.", "was_revised": true, "final_justification": "After discussion with the authors, most of my concerns have been addressed, but I still believe the paper lacks sufficient novelty.", "rebuttal": "We thank the reviewer for the thoughtful and detailed review. We appreciate the informed feedback and suggested areas for improvement. We appreciate the positive remarks regarding our __clear methodology__ discussion and the __strength of our human subject study__ which lends credibility to the efficacy of the TALENTS method. Furthermore, we are encouraged by the reviewer’s assessment that our specification of compute requirements for training __aid the reproducibility of our method__. Below, we will address the stated weaknesses and questions. \n\n## Incremental novelty: \nOur work is inspired by the idea of latent policy representations and online partner adaptation. However, there are two primary novelties of TALENTS which significantly differentiate the method from existing state-of-the-art methods:\n\n1. Learning a _clustered latent embedding of agent behaviors_ from offline game trajectories to both condition a cooperator agent and serve as a generative model to produce strategy-specific partner agents during training. GAMMA [1] introduced the method of using a variational autoencoder to create a latent representation of agents from offline trajectories. However, we extend GAMMA by adding a recurrent element to the decoder architecture, enabling the latent to encode a _longer-term behavioral representation of agents_. Furthermore, we propose silhouette analysis as a method of partitioning the learned latent into individual clusters, enabling our agent to more efficiently learn the best response behavior to rarer agent strategies (in GAMMA, these strategies are sampled much less frequently compared to the predominant strategy). Lastly, the TALENTS cooperator is _explicitly conditioned on these clusters_ during training, which is a novel contribution with respect to generative agent-trained cooperators. \n\n2. TALENTS enables __intra__-episodic adaptation to zero-shot coordination problems through a tracking regret minimization framework. Prior work in the agent modeling and online adaptation setting (MeLIBA, GSCU, etc) define the adaptation objective as identifying and optimizing strategies over multiple episodes. However, in the zero-shot coordination setting, successful agents must be able to adapt to unseen partner agents within one episode. TALENTS proposes fixed-share tracking regret minimization as an effective method for accomplishing this, and demonstrates in an ablation (Fig 3) the necessity of optimizing tracking regret as opposed to static regret. This novel contribution results in a SOTA adaptive zero-shot coordination agent.\n\n## Limited Baselines: \nWe appreciate the reviewer for pointing out the need for additional baselines and are excited to report that we have conducted additional experiments with two more baselines, namely E3T and PACE. We have added these baselines in addition to our prior evaluation of TALENTS against several leading zero-shot coordination baselines (GAMMA, MEP, FCP, BP). E3T [2] utilizes a modified self-play objective alongside a partner modeling module to perform ZSC while PACE [3] adapts its behavior based on a context encoder. Preliminary results are as follows:\n\n| Model   | Score (± Std. Err.)     |\n|---------|-------------------------|\n| E3T     | 548.66 ± 25.44          |\n| PACE    | 313.34 ± 18.48          |\n| GAMMA   | 374.10 ± 24.99          |\n| BR      | 343.71 ± 24.27          |\n| TALENTS | 605.88 ± 27.89          |\n\nTALENTS remains the top performer out of all the assessed methods, while E3T is able to outperform all the other baselines. PACE underperforms, achieving comparable scores to the population best response baseline. This can be attributed to the fact that PACE, alongside similar methods like MeLIBA [4], are effective __inter__-episodic adaptation frameworks. However, in this zero-shot coordination problem setting, PACE is unable to form its belief in the short time frame necessary for high-performance, and therefore achieves a similar score as the naive population best response baseline. This highlights a key differentiator and source of novelty of TALENTS, which is its ability to perform __intra__-episodic adaptation.\n\nWe also appreciate the suggestion to consider “Bayesian meta-task reuse”. Unfortunately, it is unclear which specific method the reviewer is referring to. Thus, we would be grateful if the reviewer could kindly point us to a relevant reference or paper, so that we can incorporate that work.\n\n## Cluster Selection and Sensitivity: \nWe appreciate the request for further insights into the number of clusters as well as the sensitivity of TALENTS with respect to the number of chosen clusters. To clarify, we utilize silhouette analysis to cluster our latent embedding space with the goal of finding a suitable representation of discrete strategy types. There is precedence of this method as an effective way of partitioning a latent embedding space [5,6]. To the reviewer’s point, prior work in opponent modeling and latent encoding methods have explored modifications to the VAE objective in order to learn a clustered embedding space in a more end-to-end fashion [9,4], but these objectives often require privileged information in order to function, such as the identity of the agent from which the trajectory originated from [9]. Unlike these methods, TALENTS does not make such assumptions. To evaluate the efficacy of silhouette analysis for selecting an optimal number of clusters, we perform a sensitivity analysis over a range of clusters, ranging from the naive one-cluster case to 16 clusters. We find that the silhouette analysis optimal number of clusters (11) achieves the highest score, empirically demonstrating the strength of our clustering approach.\n\n| Number of Clusters   | Score (± Std. Err.)     |\n|---------|-------------------------|\n| 1     | 650.83 ± 28.00          |\n| 2    | 701.68 ± 11.93         |\n| 4   | 620.35 ± 39.01          |\n| 8      | 740.93 ± 79.15          |\n| 11 | 865.82 ± 15.11          |\n| 14 | 724.94 ± 40.60          |\n| 16 | 648.57 ± 26.26          |\n\n## Cross-domain generality: \nThank you for requesting additional information with respect to the applicability of TALENTS to additional environments, such as Hanabi or pursuit-evasion tasks. TALENTS, as proposed in this work, is targeting pure collaborative tasks between multiple players in a fully observable environment. Overcooked is a challenging and complex environment for human-agent, as well as agent-agent teamwork, requiring effective collaboration between all players [1,7,8]. As such, Hanabi, which utilizes a partial information setting, and pursuit-evasion games, which utilize an adversarial component, are out of scope for the proposed method, but provide an interesting avenue for TALENTS in potential future work. \n\n## Computational Cost and Real-Time Capabilities\n\nWe acknowledge the need for further clarification of TALENTS’s ability to run in real-time and perform inference with minimal latency to collaborate effectively. As such, we measured the inference time of TALENTS. We tracked the average total time taken from when the policy receives an observation from the environment to when it sends its action back to the environment (expert evaluation and cluster inference occur between those two states). We compare to a policy without any adaptation component as a baseline.\n\n| Method   | Clusters | Mean Latency |\n|----------|----------|---------|\n| TALENTS  | 4        | 8.7 ms  |\n| TALENTS  | 11       | 18.8 ms |\n| TALENTS  | 15       | 23 ms   |\n| Baseline | N/A      | 7.5 ms  |\n\nHuman-agent experiments in the Overcooked environment run at 10 frames per second, so even with a 23ms latency and the extreme case of expert computation every frame, there is still a considerable margin. The inference time roughly scales linearly with respect to the cluster number. If the number of experts was considerably larger, the cluster inference steps could be parallelized across GPUs, leading to decreased latency. However, since this latency was not an issue in our experiments, we saw no need to make this optimization.\n\n[1] Liang, Y., Chen, D., Gupta, A., Du, S. S., & Jaques, N. (2024). Learning to cooperate with humans using generative agents. Advances in Neural Information Processing Systems, 37, 60061-60087.\n\n[2] Yan, X., Guo, J., Lou, X., Wang, J., Zhang, H., & Du, Y. (2023). An efficient end-to-end training approach for zero-shot human-AI coordination. Advances in neural information processing systems, 36, 2636-2658.\n\n[3] Ma, L., Wang, Y., Zhong, F., Zhu, S. C., & Wang, Y. (2024). Fast peer adaptation with context-aware exploration. arXiv preprint arXiv:2402.02468.\n\n[4] Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., & Hofmann, K. (2021). Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864.\n\n[5] Zhao, M., Simmons, R., & Admoni, H. (2022, October). Coordination with humans via strategy matching. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 9116-9123). IEEE.\n\n[6] Pandya, R., Zhao, M., Liu, C., Simmons, R., & Admoni, H. (2024, May). Multi-agent strategy explanations for human-robot collaboration. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 17351-17357). IEEE.\n\n[7] Zhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., ... & Yang, W. (2023, June). Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 6145-6153).\n\n[8] Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., & Dragan, A. (2019). On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32.\n\n[9] Papoudakis, G., & Albrecht, S. V. (2020). Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001.10829."}, {"review_id": "EiNzj62Vjs", "reviewer": "Reviewer_Eztg", "summary": "# Summary\nThe paper proposes a novel method for training an adaptive cooperative agent that explicitly adapt online by adjusting its internal strategy clusters. The method first learns a VAE from offline joint trajectories data. Then apply K-mean clustering on the VAE's latent space. The cooperative agent is then trained by sampling \"partners\" from the VAE clusters. During test-time, the \"Fixed-Share\" is used by to infer the current cluster of the partner, which overall gives the adaptivity to the cooperative agent. The method provide good empirical results across Overcooked layouts.", "questions": "See weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- The paper proposes a sophisticated and effective algorithm for online adaptation of a cooperative agent.\n- The method is well motivated.\n- Good empirical results.\n\n\n# Weaknesses\n- A baseline where the cooperative agent training does not require a VAE, e.g., MeLIBA, is not included. The gap between VAE- and non VAE-based (or even learned vs explicit) strategy inference methods would give a good sense of improvement the proposed method provides.\n- I find the analysis to be lacking. It is very interesting that the proposed algorithm optimizes the experts per timestep. However, There is no analysis how the experts provide specialization or how often the switch happens internally. This analysis would still be useful even if the partner is static throughout an episode.\n- No experiment on the impact of the offline data. Having this experiment would provide a useful observation whether the proposed method would still operate given different quantities and qualities of the offline data.\n- Another ablation that is not included is training an RNN policy with algorithm 1 without using algorithm 2 during test-time. This baseline would provide information on how algorithm 2 performs compared to a simple RNN policy.\n\n## Minor comments\n- Figure 1 caption is not complete\n- Line 68: incomplete sentence\n\nI'm willing to increase the score if all the Weaknesses are properly addressed.", "was_revised": true, "final_justification": "I've increased my score to 4 given the current scope and impact of the paper. The evaluation environments are currently limited environments and marginal performance improvement.", "rebuttal": "We thank the reviewer for their thorough and insightful review. We appreciate that the reviewer found the proposed TALENTS method to be well-motivated, sophisticated, and effective, as well as being well-supported by empirical results. Below, we will address the critiques and weaknesses:\n\n## Additional Baselines (MeLIBA, non-VAE methods):\nWe thank the reviewer for their suggestion for additional baselines and evaluating the impact of VAE and non-VAE based methods. We agree that this is an important to evaluation. To this end, we currently evaluate against GAMMA [1] (a SOTA method which __uses a VAE__ for partner generation), as well as BP [2], FCP [3], and MEP [4] (SOTA population-trained methods which __do not use a VAE__ at all). The requested MeLIBA baseline falls into the same category as the GAMMA agents, using a VAE as part of its methodology [5]. However, to further extend our experiments, we have implemented E3T [6] as an additional baseline (a SOTA __self-play__ method which __does not use a VAE__ but has a __partner modeling module__ for strategy inference), as well as PACE [7] (a SOTA __population-trained__ method which uses a __context encoder__ to explore and adapt to partners). Overall results, including the new baselines, are as follows, showing strong performance of TALENTS as compared to other baselines on the open and hallway layouts.\n\n| Model   | Score (± Std. Err.)     |\n|---------|-------------------------|\n| E3T     | 548.66 ± 25.44          |\n| PACE    | 313.34 ± 18.48          |\n| GAMMA   | 374.10 ± 24.99          |\n| BR      | 343.71 ± 24.27          |\n| TALENTS | 605.88 ± 27.89          |\n\nWith the inclusion of new baselines, TALENTS still maintains the highest performance out of all assessed methods, while E3T is able to outperform all the other baselines. PACE underperforms, achieving comparable scores to the population best response baseline. PACE, similar to other bayesian adaptation methods like GSCU [8] and MeLIBA [4], are effective in the __inter__-episodic setting, where the agent is able to form beliefs over __several episodes of play__ with the same agent, transitioning from exploratory to exploitative behaviors over time. In our zero-shot coordination setting, in which agents must immediately adapt to unseen partners, __PACE is unable to form its belief in the short time frame necessary for high-performance__, and therefore achieves a similar score as the naive population best response baseline. This highlights a key differentiator and source of novelty of TALENTS, which is its ability to perform __intra__-episodic adaptation. This capability is in part made possible by the fixed share tracking regret minimization method (Algorithm 2), and is a reason why TALENTS achieves SOTA performance as a zero-shot coordinator.\n\n## Switching Between Experts\nThank you for the question about how TALENTS switches between various experts when observing the player and how experts represent specialization. Fundamentally, our VAE (section 4.1) is used for strategy learning by encoding a set of trajectories from our population agents into a latent space. The optimization target for the VAE is, intuitively, to take a sequence of prior states and actions in order to predict the next actions (e.g. “chop meat”) over some horizon. Given this objective, the latent space can be interpreted as a representation of agent behavior. Clustering this latent space results in a set of experts that exhibit certain behavior patterns, e.g., facilitating a strong preference for cooking rice. We can track the high-level action distribution of each expert to evaluate how specialized each best response behavior is. The mean Jensen-Shannon divergence in the hallway layout across all the action distributions with respect to each other is 0.483 +/- 0.125, indicating moderate behavioral differences. A near-zero divergence would indicate the cooperator has not learned significantly different responses to different partner strategies, while an entirely disjoint near-one divergence would suggest that the cooperator has overfit to each partner type, losing some ability to generalize to out-of-distribution partners.\n\nExactly quantifying how often TALENTS switches its internal estimate of which expert it is playing with depends on various factors, e.g., how skilled the partner is or how often they change their action patterns. To assess TALENTS’s ability to correctly identify player behaviors, we have conducted additional experiments in which TALENTS is paired with static expert agents generated from its own VAE, which allowed us to know the ground truth clusters the partners are supposed to map to, allowing us to evaluate the player behavior categorization method in isolation. Through this experiment, __TALENTS converged to identifying the correct expert in 91.3% of episodes__. In this setting, it typically took the fixed-share mechanism approximately 30 timesteps to converge to an expert it maintained confidence about for the rest of the episode, seldom changing cluster beliefs partway through an episode. Intuitively, this is not true for experiments with switching policies or dynamic human partners, as we discuss next. \n\nFigure A.6 in the appendix (see supplemental material) provides a figure demonstrating a policy switch, dynamically changing which partner TALENTS is paired with during a game. The experiment shows a partner switch one minute into the game (at time-step 600). As shown in the figure, TALENTS changes the estimate of which partner it is playing with within ~200 timesteps (about 20 seconds), underlining its capability to adjust to changing partners.\n\n## Impact of Offline Data\nWe appreciate the request for analyzing the impact of offline data and agree that this analysis provides valuable insights. To clarify, we generate a dataset from our pretrained behavior policies in order to train the VAE (see Section 4.1). With this data, we ensure that the VAE training is comprehensively covering the possible space of behaviors that potential partners could have. As a preliminary experiment to examine the effect of data quantity, we compare agents trained on a large dataset (550 joint trajectories, 2.64 million total timesteps) with those trained on a smaller dataset (200 joint trajectories, 960,000 total timesteps). We find the large dataset agent to significantly outperform the small dataset agent. Intuitively, the VAE is able to learn a richer latent space with the larger dataset and reduces the risk of overfitting to predominant behaviors. \n\n\n| Model         | Score (± Std. Err.)     |\n|---------------|-------------------------|\n| TALENTS-Large | 834.83 +/- 26.04        |\n| TALENTS-Small | 578.04 +/- 31.78        |\n\n## RNN Policy:\n\nThank you for suggesting an additional baseline RNN policy. We agree that such results would be valuable and are providing them below. _TALENTS-Ablated_ refers specifically to a TALENTS agent trained with clustered generative VAE agents (Algorithm 1) but __does not__ employ the fixed-share adaptation mechanism (Algorithm 2), while _TALENTS-Full_ refers to our full method. As can be seen in the following results, the addition of the fixed-share adaptation mechanism (Algorithm 2) provides a significant improvement:\n\n| Model           | Score    |\n|-----------------|----------|\n| TALENTS-Full    | 605.88   |\n| TALENTS-Ablated | 539.71   |\n\n## Missing caption and incomplete sentence\nThank you for identifying these issues, we have amended the draft to fix them.\n\n[1] Liang, Y., Chen, D., Gupta, A., Du, S. S., & Jaques, N. (2024). Learning to cooperate with humans using generative agents. Advances in Neural Information Processing Systems, 37, 60061-60087.\n\n[2] Wang, X., Zhang, S., Zhang, W., Dong, W., Chen, J., Wen, Y., & Zhang, W. (2024). Zsc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination. Advances in Neural Information Processing Systems, 37, 47344-47377.\n\n[3] Strouse, D. J., McKee, K., Botvinick, M., Hughes, E., & Everett, R. (2021). Collaborating with humans without human data. Advances in neural information processing systems, 34, 14502-14515.\n\n[4] Zhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., ... & Yang, W. (2023, June). Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 6145-6153).\n\n[5] Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., & Hofmann, K. (2021). Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864.\n\n[6] Yan, X., Guo, J., Lou, X., Wang, J., Zhang, H., & Du, Y. (2023). An efficient end-to-end training approach for zero-shot human-AI coordination. Advances in neural information processing systems, 36, 2636-2658.\n\n[7] Ma, L., Wang, Y., Zhong, F., Zhu, S. C., & Wang, Y. (2024). Fast peer adaptation with context-aware exploration. arXiv preprint arXiv:2402.02468.\n\n[8] Fu, Haobo, et al. \"Greedy when sure and conservative when uncertain about the opponents.\" International Conference on Machine Learning. PMLR, 2022."}, {"review_id": "aB7zOPx4cw", "reviewer": "Reviewer_xno3", "summary": "This paper introduces TALENTS, a novel framework for zero-shot human-agent and agent-agent coordination based on learned latent strategy embeddings. It employs a variational autoencoder to model teammate strategies, clusters these strategies in latent space, and trains a cooperator agent conditioned on the resulting clusters. Online adaptation is achieved through a fixed-share regret minimization algorithm. The method is evaluated on an extended version of the Overcooked environment and tested with both synthetic agents and human subjects.", "questions": "1. Including results from at least one additional environment—even preliminary—would strengthen the paper’s claim of broad relevance in human-AI coordination. \n\n2. Performing a sensitivity analysis over the key hyperparameters such as the number of the clusters would enhance confidence in the robustness and reliability of this component.\n\n3. Including an ablation where the system encounters humans with atypical strategies would demonstrate robustness and validate assumptions about the latent space. \n\n4. Adding more Ablation Studies and comparisons with alternative approaches would situate the proposed method more clearly in the literature.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. **Novelty and Motivation**: The approach of clustering latent behaviors and adapting via fixed-share regret minimization is original and well-motivated within the context of ad hoc teamwork and zero-shot coordination.\n\n2. **Technical Soundness**: The combination of VAE-based behavior modeling with a regret-minimization framework is technically coherent. The fixed-share approach to handle intra-episodic partner behavior shifts is well-justified.\n\n3. **Empirical Results**: TALENTS achieves strong performance across most Overcooked layouts, particularly outperforming GAMMA and best-response agents in most settings. The human-agent evaluation demonstrates promising real-world applicability, with higher subjective ratings on fluency, trust, and satisfaction.\n\n4. **Inclusion of Human-Agent Study**: The paper includes an online user study with human participants, which is crucial for validating coordination methods in realistic settings. This adds significant value and addresses a critical aspect of human-AI interaction.\n\n---\n\n**Weaknesses and Points for Improvement:**\n\n1. **Limited Generalization Beyond Overcooked**: All experiments are confined to a single domain - Overcooked. While this is a standard testbed, it is unclear whether the method would generalize to other complex multi-agent settings. A discussion of this limitation or preliminary results on a second environment would improve the work’s impact.\n\n2. **Clustering Sensitivity**: Details on Strategy Clustering: While K-Means with silhouette analysis is used to determine optimal clusters, there is no guarantee that the identified clusters correspond to meaningful or semantically coherent behavioral categories. The sensitivity of the overall system's performance to the chosen number of clusters (K) and whether these discrete clusters adequately represent the continuous spectrum of human strategies could be further investigated. The implications of misclassifying a human's strategy are not explicitly discussed.\n\n3. **Latent Space Generalization**: While the VAE is trained on diverse agent behaviors, its ability to encode novel human-like strategies not seen during training is questionable. The paper assumes that the VAE latent space covers the full spectrum of possible strategies, but this is not empirically validated. This assumption may fail in real-world cases (e.g., humans with unique strategies), and the paper does not quantify how performance degrades in such cases, especially from humans who may behave in fundamentally different ways than the baseline agents.\n\n4. **Fixed-Share Regret Minimization**: While the fixed-share algorithm allows for tracking non-stationary strategies, its effectiveness is only demonstrated in one synthetic experiment (partner policy swap mid-episode). It would be helpful to see more extensive evaluation of this component, especially in human-agent interactions where partner policies are inherently non-stationary.\n\n5. **Ablation Studies and Comparisons**: While the paper includes an ablation of the fixed-share component, there is limited analysis of other key design choices: 1) What is the performance when conditioning directly on continuous latents instead of discrete clusters? 2) How does the system behave without priority-based sampling or latent-space biasing? 3) GAMMA is a strong baseline, but including some other advanced related works, or simpler latent-based adaptation methods would strengthen the comparative picture.", "was_revised": true, "final_justification": "Thanks for the responses! I maintain my score.", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and suggestions. We are encouraged to hear that our method, TALENTS, has been perceived as __novel, original, and well motivated__. Additionally, we are excited to hear that our empirical results, particularly TALENTS' performance across Overcooked layouts and the human-agent evaluation, were seen as __strong and indicative of real-world applicability__. We also appreciate the recognition of the __technical soundness of our approach__, especially the integration of VAE-based behavior modeling with fixed-share regret minimization. In the following, we will address the weaknesses and questions of the reviewer:\n\n## Limited Evaluation on Overcooked\nThank you for requesting further information about the evaluation domain. Overcooked is a challenging domain for AI agents [1,2,3], as well as humans, providing a rich testbed for agent-agent and human-agent teaming capabilities due to various factors, such as time pressure, multiple cooking stations, multi-step recipes, and a reward structure that encourages fast deliveries.  While we agree that evaluating TALENTS in additional domains such as Hanabi (which introduces partial observability) or cooperative pursuit-evasion (which adds adversarial dynamics) would be valuable, incorporating such experiments is beyond the scope of this submission. We consider these exciting directions for future work and plan to explore them in follow-up studies.\n\n## Cluster Sensitivity\nWe appreciate the request for further insights into the number of clusters as well as the sensitivity of TALENTS with respect to the number of chosen clusters. To clarify, we utilize silhouette analysis to cluster our latent embedding space with the goal of finding a suitable representation of discrete strategy types. There is precedence of this method as an effective way of partitioning a latent embedding space [7,8]. To the reviewer’s point, prior work in opponent modeling and latent encoding methods have explored modifications to the VAE objective in order to learn a clustered embedding space in a more end-to-end fashion [10,9,6], but these objectives often require privileged information such as the identity of the agent from which the trajectory originated from in order to function [10,9]. __Unlike these methods, TALENTS does not make such assumptions__. To evaluate the efficacy of silhouette analysis for selecting an optimal number of clusters, we perform a __sensitivity analysis over a number of cluster partitions__, ranging from the naive one-cluster case to 16 clusters. __We find that the silhouette analysis optimal number of clusters (11) achieves the highest score, empirically demonstrating the strength of our clustering approach.__\n\n| Number of Clusters   | Score (± Std. Err.)     |\n|---------|-------------------------|\n| 1     | 650.83 ± 28.00          |\n| 2    | 701.68 ± 11.93         |\n| 4   | 620.35 ± 39.01          |\n| 8      | 740.93 ± 79.15          |\n| 11 | 865.82 ± 15.11          |\n| 14 | 724.94 ± 40.60          |\n| 16 | 648.57 ± 26.26          |\n\nThe issue of potentially misclassifying a partner’s behavior and selecting an inappropriate cluster is a valid concern. To this end, we assess TALENTS’s ability to correctly identify player behaviors. We have conducted additional experiments in which TALENTS is paired with all static experts, __correctly identifying the correct expert in 91.3% of evaluations__.\n\n## Generalization and Unseed Human Data\nThank you for requesting further insights on TALENTS’s ability to handle novel human behavior. It is indeed correct that one assumption of TALENTS is that the VAE’s latent space sufficiently covers novel human behavior. The latent space is trained via a dataset of 11 reward-shaped policies. Evaluation is done using a population of 10 diverse unseen policies for all experiments to ensure meaningful generalization of our agent. There is no guarantee that these evaluation policies are necessarily represented by any particular cluster in the latent space, but the rationale of using a regret minimization framework for partner inference is the guarantee that __out-of-distribution policies will not cause catastrophic performance degradation due to the framework's theoretical bounds__. Furthermore, we analyzed the VAE’s ability, which is only trained on our population agents, to encode the data from our human experiments and found that human data covers a similar distribution in our VAE in terms of cluster representation as the behavior preference agents used to train the model. Calculating the Jensen-Shannon divergence of the agent and human data distributions yields 0.1516, __indicating a high degree of similarity between the two__. This value suggests that the __agent behavior closely approximates human behavior, with only minor differences__.\n\n## Fixed-Share Regret Minimization\nWe appreciate the request for a more in-depth evaluation of non-stationary partners and agree that this is an important area of analysis. As shown in Figure A.6, TALENTS is capable of adapting to novel partners within just a few seconds. Additionally, we show that TALENTS successfully identifies the correct expert in 91.3% of evaluation episodes. Given that the VAE effectively captures the range of human behaviors observed in our human-agent experiments, and that our human subject study reports high ratings for fluency, trust, and overall performance relative to other baselines (see Figure 4), we conclude that TALENTS performs robustly in settings with non-stationary partners (such as humans). \n\n## Additional Baselines \nWe appreciate the reviewer for pointing out the need for additional baselines and are excited to report that we have conducted additional experiments with two more baselines, namely E3T [4] and PACE [5]. We have added these baselines in addition to our prior evaluation of TALENTS against several leading zero-shot coordination baselines (GAMMA, MEP, FCP, BP). E3T [4] utilizes a modified self-play objective alongside a partner modeling module to perform ZSC, and PACE [5] adapts its behavior via a learned context encoder. Preliminary results are as follows:\n\n\n| Model   | Score (± Std. Err.)     |\n|---------|-------------------------|\n| E3T     | 548.66 ± 25.44          |\n| PACE    | 313.34 ± 18.48          |\n| GAMMA   | 374.10 ± 24.99          |\n| BR      | 343.71 ± 24.27          |\n| TALENTS | 605.88 ± 27.89          |\n\n__With these results, TALENTS demonstrates that it can, consistently, outperform other baselines, providing a significant improvement over existing literature.__\n\n[1] Liang, Y., Chen, D., Gupta, A., Du, S. S., & Jaques, N. (2024). Learning to cooperate with humans using generative agents. Advances in Neural Information Processing Systems, 37, 60061-60087.\n\n[2] Zhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., ... & Yang, W. (2023, June). Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 6145-6153).\n\n[3] Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., & Dragan, A. (2019). On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32.\n\n[4] Yan, X., Guo, J., Lou, X., Wang, J., Zhang, H., & Du, Y. (2023). An efficient end-to-end training approach for zero-shot human-AI coordination. Advances in neural information processing systems, 36, 2636-2658.\n\n[5] Ma, L., Wang, Y., Zhong, F., Zhu, S. C., & Wang, Y. (2024). Fast peer adaptation with context-aware exploration. arXiv preprint arXiv:2402.02468.\n\n[6] Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., & Hofmann, K. (2021). Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864.\n\n[7] Zhao, M., Simmons, R., & Admoni, H. (2022, October). Coordination with humans via strategy matching. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 9116-9123). IEEE.\n\n[8] Pandya, R., Zhao, M., Liu, C., Simmons, R., & Admoni, H. (2024, May). Multi-agent strategy explanations for human-robot collaboration. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 17351-17357). IEEE.\n\n[9] Papoudakis, G., & Albrecht, S. V. (2020). Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001.10829.\n\n[10] Aditya Grover, Maruan Al-Shedivat, Jayesh K Gupta, Yura Burda, and Harrison Edwards. Learning policy representations in multiagent systems. International Conference on Machine learning, 2018a."}, {"review_id": "pQzc4bNqQv", "reviewer": "Reviewer_LHkq", "summary": "This paper tackles the challenge of adaptation in human-agent collaboration, where agents must adjust to diverse and dynamically changing partner strategies. The authors propose a strategy-conditioned cooperator framework that uses a variational autoencoder to learn a latent strategy space from partner trajectories. Strategies are categorized via clustering, and the agent is trained to cooperate with different strategy types. During interaction with a novel partner, the agent adapts online using a fixed-share regret minimization algorithm that continuously updates its estimate of the partner’s strategy. The method is evaluated in a modified Overcooked environment and a human user study, showing improved performance over existing baselines when cooperating with both artificial and human partners.", "questions": "1. Could the authors provide a more comprehensive discussion of the related work, particularly the relevant studies mentioned above (e.g., E3T, PACE, GSCU, ProAgent)? In addition, an empirical comparison with these closely related baselines, especially those also evaluated in the Overcooked domain, would significantly clarify the contribution and positioning of this work.\n\n2. Could the authors consider extending their framework to competitive settings, as explored in works like PACE and GSCU? The current structure, in which the ego agent infers the partner's behavior and computes a best response, appears well-suited for adaptation in competitive scenarios, such as poker or adversarial games. Furthermore, demonstrating the method's effectiveness in more complex environments with image-based observations and higher-dimensional action spaces would considerably strengthen the paper.\n\n3. In Figure 4, TALENTS shows consistently higher scores in fluency and trust, but does not significantly outperform baselines in coordination and satisfaction. Could the authors elaborate on the reasons behind this discrepancy?", "limitations": "See Weaknesses and Questions.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper is well-written and easy to follow. The figures and pseudocode are generally helpful in conveying the methodology and implementation details.\n\n2. The problem of real-time adaptation in human-AI collaboration is important and well-motivated. The proposed approach, which enables an agent to infer its partner’s strategy and adapt its policy accordingly, addresses a key challenge in this domain. The integration of the fixed-share regret minimization algorithm within TALENTS to reduce regret is both principled and effective.\n\n3. The inclusion of experiments with human participants strengthens the paper by demonstrating the practical utility of the method in real-world human-ai collaborative scenarios.\n\n\n**Weaknesses**\n1. The paper lacks a thorough discussion of related work. Teammate modeling for zero-shot coordination and real-time adaptation has been extensively studied. For example, E3T [1] introduces an end-to-end approach with a partner modeling module to predict the partner’s actions based on historical context. PACE [2] explicitly addresses both teammate and opponent adaptation and similarly encodes behavioral context to update the ego agent’s policy. GSCU [3] also tackles adaptation via a CVAE-based latent policy encoder and online Bayesian updates, using an EXP3 algorithm to manage regret—an approach conceptually close to this work. More recently, ProAgent [4] incorporates LLMs with belief inference for zero-shot coordination. Notably, PACE and ProAgent both evaluate their methods in the Overcooked domain, making them particularly relevant baselines. The omission of these works from the related work and experimental comparisons limits the clarity of this paper’s contribution and novelty.\n\n2. The empirical evaluation is limited to a single environment (Overcooked), which is a grid-based domain with relatively low-dimensional state and action spaces. Additional experiments in more complex or diverse collaborative environments would help demonstrate the generality of the approach.\n\n3. The proposed method is evaluated only in cooperative settings. In contrast, related works such as PACE and GSCU consider both cooperative and competitive dynamics. Addressing broader multi-agent interactions would enhance the applicability of the framework.\n\n[1] Yan, Xue, et al. \"An efficient end-to-end training approach for zero-shot human-AI coordination.\" Advances in Neural Information Processing Systems 36 (2023): 2636-2658.\n\n[2] Ma, Long, et al. \"Fast peer adaptation with context-aware exploration.\" arXiv preprint arXiv:2402.02468 (2024).\n\n[3] Fu, Haobo, et al. \"Greedy when sure and conservative when uncertain about the opponents.\" International Conference on Machine Learning. PMLR, 2022.\n\n[4] Zhang, Ceyao, et al. \"Proagent: building proactive cooperative agents with large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 16. 2024.", "was_revised": true, "final_justification": "The additional discussions and baseline results help address the initial lack of a thorough literature review and the unclear contribution. However, the extension to competitive settings remains an open direction for future work. A rating of 4: Borderline Accept reflects the current quality of the paper.", "rebuttal": "We thank the reviewer for their comprehensive feedback of our work. We greatly appreciate the reviewer for highlighting some of the strengths of our work, including that our method is __well-motivated, effective, and grounded in principled approaches__. We are encouraged that our human-agent user study lends credibility to the real-world applicability of the research, and that the paper is well-written and easy to follow. Below, we address the weaknesses and questions raised by the reviewer. \n\n## Discussion on Related Work\nWe greatly appreciate the reviewer’s suggestions with respect to the discussion of related work. We have expanded our discussion, particularly in the Strategy Inference section to highlight E3T, the bayesian adaptation framework of GSCU, the contextual encoder and exploration reward of PACE, as well as the LLM-based belief inference method of ProAgent. We hope that this additional discussion will provide a clearer understanding of the body of work that has informed and contributed to the development of TALENTS. In addition, we hope that this discussion will provide a clearer understanding as to the novelty and contributions of our method.\n\nFurthermore, in addition to discussing them in the related work section, __we have prepared E3T and PACE as additional baselines in our experiment section__ (see next response for details)\n\n## Additional Baselines\nIn relation to the previous comment, we recognize that evaluating our method against the aforementioned methods is important to provide better context and positioning of our novel TALENTS agent. Based on the reviewer’s recommendation, we have decided to implement __E3T__ and __PACE__ as additional baselines. We chose not to evaluate GSCU, as it was included in the PACE paper as a baseline and was significantly outperformed by PACE in the Overcooked setting [1]. The following are our preliminary results from the new baselines:\n\n| Model   | Score (± Std. Err.)     |\n|---------|-------------------------|\n| E3T     | 548.66 ± 25.44          |\n| PACE    | 313.34 ± 18.48          |\n| GAMMA   | 374.10 ± 24.99          |\n| BR      | 343.71 ± 24.27          |\n| TALENTS | 605.88 ± 27.89          |\n\n__TALENTS remains the top performer out of all the assessed methods__, while E3T is able to outperform all the other baselines. PACE underperforms, achieving comparable scores to the population best response baseline. PACE, similar to other bayesian adaptation methods like GSCU and MeLIBA [2], are effective in the __inter__-episodic setting, where the agent is able to form beliefs over __several episodes of play with the same agent__, transitioning from exploratory to exploitative behaviors over time. In our zero-shot coordination setting, in which agents must immediately adapt to unseen partners, PACE is unable to form its belief in the short time frame necessary for high-performance, and therefore achieves a similar score as the naive population best response baseline. This highlights a key differentiator and source of novelty of TALENTS, which is its ability to perform __intra__-episodic adaptation. __This capability is largely made possible by the fixed-share tracking regret minimization method__ (Algorithm 2), and is a reason why TALENTS achieves SOTA performance as a zero-shot coordinator.\n\n## Extension to Competitive Settings\nWe appreciate the reviewer’s suggestion to explore competitive or adversarial settings. While we agree these are valuable directions, our work is focused on the __zero-shot coordination / ad-hoc teaming problem in cooperative environments__, where the goal is rapid adaptation to unfamiliar partners. Introducing competition would shift the scope to a different class of problems involving adversarial intent and asymmetric payoffs, requiring distinct modeling and evaluation. That said, we agree that extending TALENTS to mixed-motive or adversarial domains is a promising avenue for future research, and __we now note this in the conclusion__.\nWe also agree with the reviewer’s point on environment complexity. This was a key motivation for utilizing a modified version of Overcooked, as __the original setting [4] only had limited opportunities for behavioral diversity__. While recent extensions [3, 1] introduced more difficulty, __our modified environment introduces considerable complexity__ through the inclusion of larger maps, longer episodes (T=240s rather than the typical T=40s [9,4] or T=60s [3,8]), temporal constraints, and multi-step tasks, enabling a more intricate evaluation of agent robustness in highly complex teamwork tasks.\nWhile using image-based observations is feasible, we chose the grid-based representation of Overcooked to enable a smaller model footprint, requiring fewer parameters and less data. Moreover, the structured representation focuses the learning signal, making it easier to analyze and ablate the key factors that contribute to our method’s success. That said, we agree this raises an interesting direction for future work and would be more akin to real-world applications.\n\n## User Study Survey Scores\n\nThis is a great question, we thank the reviewer for raising this discrepancy. The survey responses in the trust and fluency category are typically indicative of the human’s perception of their agent partner’s capability and proficiency [6,7], whereas the coordination and satisfaction metrics tend to be more indicative of the human’s perception of the joint teamwork [5]. We hypothesize that the lack of statistically significant ratings in these regards may be due to the __distribution shift between agent-agent games and human-agent games__. Agents tend to move differently in the Overcooked environment compared to human players, and may perform actions perceived as unpredictable or erratic to their human teammates. Since our cooperator agents were only paired with other agents during training, __there was no mechanism to incentivize them to behave in a “human-interpretable” way__. __This shift is equally present among the three methods assessed in the user study__. To further assess this hypothesis, we would need to ask more post-game questions (particularly open-ended questions) to participants, which was beyond the scope of the initial human subject study.\n\n[1] Ma, L., Wang, Y., Zhong, F., Zhu, S. C., & Wang, Y. (2024). Fast peer adaptation with context-aware exploration. arXiv preprint arXiv:2402.02468.\n\n[2] Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., & Hofmann, K. (2021). Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864.\n\n[3] Liang, Y., Chen, D., Gupta, A., Du, S. S., & Jaques, N. (2024). Learning to cooperate with humans using generative agents. Advances in Neural Information Processing Systems, 37, 60061-60087.\n\n[4] Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., & Dragan, A. (2019). On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32.\n\n[5] Richard M Ryan and Edward L Deci. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, 55(1):68, 2000\n\n[6] Guy Hoffman. Evaluating fluency in human–robot collaboration. IEEE Transactions on Human-Machine Systems, 49(3):209–218, 2019\n\n[7] Richard M Ryan and Edward L Deci. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, 55(1):68, 2000\n\n[8] Strouse, D. J., McKee, K., Botvinick, M., Hughes, E., & Everett, R. (2021). Collaborating with humans without human data. Advances in neural information processing systems, 34, 14502-14515.\n\n[9] Zhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., ... & Yang, W. (2023, June). Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 6145-6153)."}]

Author rebuttal:

Meta-review: The paper proposes a novel method for training an adaptive cooperative agent that explicitly adapt online by adjusting its internal strategy clusters. The method first learns a VAE from offline joint trajectories data. Then apply K-mean clustering on the VAE's latent space. The cooperative agent is then trained by sampling "partners" from the VAE clusters. During test-time, the "Fixed-Share" is used by to infer the current cluster of the partner, which overall gives the adaptivity to the cooperative agent. The method provide good empirical results across Overcooked layouts.

Most reviewers appreciate the novelty of the new method and human studies. While there are some concerns about sensitivity of clustering and insufficient baselines. During the rebuttal, the authors addressed most concerns which are also acknowledged by the reviewers. The AC believes this paper makes a valuable contribution to the community and thus recommends acceptance.
Decision: Accept (poster)