Learning Preferences without Interaction for Cooperative AI: A Hybrid Offline-Online Approach
Authors: Haitong Ma
Haoran Yu
Haobo Fu
Shuai Li
Venue: NeurIPS 2025 poster ()
Keywords: Cooperative AI
offline preference learning
online adaptation
Primary area: reinforcement_learning

Abstract: Reinforcement learning (RL) for collaborative agents capable of cooperating with humans to accomplish tasks has long been a central goal in the RL community. While prior approaches have made progress in adapting collaborative agents to diverse human partners, they often focus solely on optimizing task performance and overlook human preferences—despite the fact that such preferences often diverge from the reward-maximization objective of the environment.
Addressing this discrepancy poses significant challenges: humans typically provide only a small amount of offline, preference-related feedback and are unable to engage in online interactions, resulting in a distributional mismatch between the agent’s online learning process and the offline human data. To tackle this, we formulate the problem as an online&offline reinforcement learning problem that jointly integrates online generalization and offline preference learning, entirely under an offline training regime.
We propose a simple yet effective training framework built upon existing RL algorithms that alternates between offline preference learning and online generalization recovery, ensuring the stability and alignment of both learning objectives.
We evaluate our approach on a benchmark built upon the Overcooked environment—a standard environment  for human-agent collaboration—and demonstrate remarkable performance across diverse preference styles and cooperative scenarios.

Learning Human Preferences without Interaction for Cooperative AI: A Hybrid Offline-nnline Approach
Haitong Ma
Haoran Yu Shanghai Jiao Tong University Shanghai, China harayn4@gmail.com com com
Shanghai Jiao Tong University Shanghai Innovation Institute
Shanghai, China mahaitong@s ju. edu. cn
Haobo Fu Tencent AI Lab Shenzhen, China haobofu@tencent.ccom
Shuai Li" Shanghai Jiao Tong University Shanghai Innovation Institute Shanghai, China shuail180sjtu edu, cn
Abstract
Reinforcement learning (RL) for collaborative agents capable of cooperating with humans accomplish tasks has long been a central goal in the RL community. While prior approaches have made progress adapting collaborative agents diverse human partners, they often focus solely optimizing task performance and overlook human preferences-deste the fact that such preferences often diverge from the this discrepancy poses significant challenges: humans typically provide only a small amount of offlin,, preference-related feedback and are unable engage in online interactions, resulting in a distributional mismatch between the agent's s online learning process and the offline human data. To tackle this, we formulate the problem as an onl in flie reinforcement learning problem that jointly integrates online generalization and offline preference learning, entirely under an offline training regime. We propose simple yet effective training framework built upon existing RL algorithms that alternates between offline preference learning and online generalization recovery, ensuring the stability and alignment of both learning objectives. We evaluate our approach on benchmark built upon the Overcooked environment- standard environment for human-agent collabora- tion-and demonstrate remarkable performance across diverse preference styles and cooperative scenarios.
objective of the environment. Addressing
Introduction
Humans often exhibit diverse behaviors when performing the same task, driven by individual pref-
erences. Prior work refers to these variations preference styles (Pugh al.) 2016 Cully and
Demiris 2017 Mao et al. 2024). In multi-agent cooperative
multiplayer
games designing agents that can adapt to wide range of human preferences has become growing
focus in reinforcement learning (RL) [Klien et al. 2004]] Yuan et al. [023 Dafoe et al. [2221. To
avoid the cost human-in-the-loon training, recent research frames this challenge as an ad hoc
team play [Stone et al., 2010) or Zero-Shot Coordination (ZSC) problem [Hu et al.) 2020 Mirsky
al. 2022]. However, 7SC methods typically optimize only for environment rewards, ignoring
"corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

user-specific preferences, and thus often produce behaviors misaligned with individual styles Yu al. 2023 Liu al. 2025]. For instance. in Overcooked [Carroll et al. 2019). human player may prefer cooking potatoes, while an agent trained solely for reward maximization might independently cook -esuulinns in poor coordination despite high task performance. Moreover, ZSC assumes no prior knowledge of human behavior, limiting adaptation to preference-driven dynamics. With the rise of multiplayer gaming, the assumption that human users are fully unobservable is becoming outdated. Many games now allow players express stylistic preferences through post-hoc feedback-mos notably the "like" mechanism, where users endorse teammates after a match. These liked trajectories implicitly reflect human preferences and are widely available in real-world games such as League of Legends, Honor of Kings, and Brawl Stars, offering a scalable and lightweight source of preference-aligne aligned data. This enables the training of cooperative agents that not only perform effectively but also adapt to diverse human styles.
Importantly, in many real-world games such as 5v5 matches in League of Legends, it is impractical to model or simulate the behavior of teammates and opponents. Therefore, our setting focuses on learning solely from the decisions of cooperative agents recorded in offline trajectories, without relying on access to or simulation of other players' strategies. We formulate this as a novel online&o f RL problem under three assumptions: (1) access to environment that supports online interaction, (2) availability of policy pool for training the cooperative agent, and (3) a small set of offline trajectories reflecting human preferences. The objective is to train cooperative agent that both generalizes to unseen human partners and aligns with preferences expressed in the offline data. Unlike prior RL problems that combine online and offline learning, our setting introduces a new challenge: balancing zero-shot generalization with preference alignment. Learning proceeds in two distinct interaction for generalization and fully offlin optimization for preference adaptation. The o fline RL problem introduces several unique difficulties:
forreeoofinne fline
Challenge 1: Limited preference data and no human interaction. The number of labeled preference trajectories is small, and the agent cannot interact with the target humans online. humans are modeled as part of the environment, this leads to unknown transition dynamics. Existing methods typically learn a reward model from limited data and optimize it online [Wirth et al., 2017 Arora and Doshi 2021], which is infeasible here. Fully offline methods also struggle in this low-data regime- direct imitation on a few labeled trajectories often overfits and fails.
Challenge 2: Objective mismatch between generalization and preference alignment. Offline preference data reflect distributions that differ from those of for alignment can therefore harm generalization across diverse human partners. Existing approaches, including offline preference methods [Lee et al. 2021 Liu et al. 2025 Abdelkareem et al., [2022]
agents. Optimizing
and ZSC methods like HSP YYu et al. 2023 and MEP Zhao et al. 2023 do not explicitly address this trade-off.
To address the aforementioned challenges, we draw inspiration from the pretraining and supervised fine-tuning (SFT) paradigm widely adopted in natural language processing (NLP) [Devlin et al.. 2019 Radford et al. 2019 Raffel et al. 2020). We propose three-stage hybrid framework that integrates online generalization training with offline imitation on preference-labes labeled trajectories. Conceptually, the online training phase mirrors language model pretraining, while offline imitation corresponds to downstream SFT. We observe that applying online generalization recovery after offline imitation allows the pretrained agent to retain part of the learned preferences. Building upon this insight, we introduce a training strategy-Epoocd Epoch-wise Alternation Recovery (EAR)-which alternates between 1-epoch lightweight behavior cloning and generalization recovery. This strategy enables cooperative agents to maintain strong generalization capabilities while effectively incorporating human preferences. To evaluate the ability for the online&oo fline fline RL problem, we further develop a dedicated benchmark based on the Overcooked environment. Extensive experiments demonstrate the effecctivnness of our approach, showing consistent improvements over competitive basclines across multiple evaluation metrics.
2 Related work
2.1 Online generalization ability learning
In game AT, effective cooperation requires agents to generalize across diverse human partners, adapting to various strategics while completing tasks. This challenge mirrors that of language models,
2

---

which achieve adaptability through large-scale pretraining on diverse data (Devlin et al. 2019 Radford al. 2019 Raffel et al. 2020]. reinforcement learning. this generalization challenge is formalized as the ZSC problem, where agents aim to maximize returns when paired with unseen partners. The FCP framework [Strouse et al. [02T] uses policy pool to expose agents to diverse behaviors during training. while MEP (Zhao et al. 2023] extends this through entropy-maximise
policy diversification.
HSP YYu et al., 2023] further enhances robustness by modeling human preferences as biased. While these ZSC methods improve generalization, they largely overlook preference learning the ability to adapt to user-specific preferences. As a result, agents trained only for ZSC problem may struggle in personalized human-agent interactions.
2.2 Offline preference learning
Offline preference learning has gained increasing attention, particularly in natural language pro- cessing (NLP). Direct Preference Optimization (DPO) Raffel ct al. [2020] reframes reinforcement learning with human feedback (RLHF) (Achiam al. 2023 Wang et al. 2024b] as fully offline supervised objective, encouraging models to prefer higher-qualij responses. Kullback-Leibler Target Optimization (KTO) (Hu and Hong 2013) further advanced this by removing the need for explicit preference pairs, while Inverse Preference Optimization (IPO) [Azar et al.) 2024] enhanced DPO's stability by refining its objective to mitigate overfitting. These methods are well-suited for large language models, which retain strong capabilities even after offline fine-tuning. In traditional RL,, preference alignment has focused on two offline paradigms: offline imitation learning [Torabi et al. 2018) Kumar et al.) 2020 Prudencio et al. 2023). which directly learns from offline trajectories, and preference-basee approaches like offline PBRL [Wirth et al. 2017 Lee et al. 2021 Abdelkareem al.) [022] and IRL (Ho and Ermon 2016 Fu et al. [017 Arora and Doshi 2021 which aim to train a reward function from labeled data. However, both methods require large amounts of offline data, making them unsuitable for online&cof f fline problem with limited data.
2.3 Online RL combined with offline data
The proposed problem requires combining online generalization with offline preference learning setting only partially addressed in prior RL research. Existing approaches can be grouped into three categories. The first incorporates offline data into online training via shared replay buffers to improve
sample efficiency and reduce interaction costs [Song al. [022 Ball et al.) 2023 The second,
oflline-to-online RL [Nakamoto et al. 2023 Zhang et al. [023]] Wang et al. 2023 pretrains on
large offline datasets and then fine-tunes online using techniques such as balanced sampling [Lee et al. 2022 Guo et al. 2023] and adaptive conservatism Kostrikov et al. 2021| Nakamoto et al.
2023 to mitigate distributional shift (Prudencio et al.) 2023 Levine al. [020 Andres et al. |025 The third, online-to-ood tofoffine RL [Liu et al.] 2025], begins with online trainnng for task performance and then uses limited human feedback to fit reward models for alignment. However, these methods typically assume access to either large-scale offline data or online human interaction-assumpiiinn that do not hold in setting. In contrast, our multi-agent setting prohibits online interaction with users and offers only small set of positive-only preference data, necessitating new solutions tailored to this constrained regime.
3 Preliminaries
3.1 Decentralized Markoy Decision Process
general n-player multi-agent game can be modeled as a Decentralized Markov Decision Process (Dec-MDP) |Bernstein et al. 2002 Wang et al. 2024a], defined by the tuple (S. A, p. T, r. 7). Here, is the state space, The transition function T(s' a) defines the probability of reaching s' from (s, a), and r(s, a) the shared reward. The discount factor 7 [0, determines the weight of future rewards. Let (TI,...,, . denote the joint policy, where each (Taa" | 8) specifies the action distribution for agent i. At each timestep I, agents act independently: at ~ .T(e 8t), yielding a joint action at = =at,...,,a?"
A
X A, is the joint action space, and p the initial state distribution.
3.1.1 Cooperative agent generalization and policy pool
In cooperative multi-agent settings, agents must coordinate their behaviors through a joint policy to maximize the cumulative return. In the ZSC problem, given a set of previously unseen human
3

---

policies n" = collaborate with these human partners to complete tasks and optimize the overall reward. This cooperative objective is referred to as the generalization larget.
the objective is to learn cooperative agent π that can effectively
max J ) max E
R(s a,
(I)
x
where sy denotes the environment state at timestep t, and a = {at ...... af } is the joint action taken by all n agents. The subset o {at oz. .......,.. , } represents the actions taken by the n-1 unknown human policies, and at is the action taken by the cooperative agent x A at timestep t.
Since directly obtaining the true distribution of human policies Pи is often infeasible, prior works [Strouse al., 2021] Yu et al. 2023, ZZhao et al., 2023 typically resort to constructing policy pool-a a collection of agents with diverse behaviors to approximate this distribution, denoted as
PH. The cooperative agent is then trained to collaborate with agents sampled from this policy pool, enabling it to generalize to unseen human partners drawn from the true distribution. Under this approximation, the generalization objective can be rewritten as:
max Jy 74
max F ~M~P [J(T H 7.
where the policy pool acts surrogate for diverse human partners. During training. follow the setting introduced in HSP (Yu et al. 2023). J. where the cooperative agent interacts online with agents sampled from the policy pool. The collected trajectories are then used to update the agent using MAPPO Yu et al. 2022] as the underlying reinforcement learning algorithm.
3.2 Post-match "like" assumption and preference learning objective
We assume a post-match "like" " mechanism, where after each game episode, human user may provide positive feedback (a "like") one of the cooperative agents if its behavior aligns with their preferences. Accordingly, for each user h, we can collect a small set of positively labeled trajectories, No.
denoted as {{(s), aa) ]==)] == where T' is the trajectory length and N₂ is the number of liked trajectories. Each (sf, at) pair represents an action at state of that was endorsed by h.
Given only these sparse, user-specific demonstrations, we define the preference learning objective as the task of imitating the behaviors favored by the user. Formally, for a cooperative policy и,, the goal is to maximize the likelihood of the user-approved behavior:
NA T
max
Jp( max オ
log (a; s
which corresponds to standard behavioral cloning over limited set of positively labeled trajectories.
Crucially, although these trajectories are obtained through direct interaction with user h, the user is not available during training. Thus, the learning process must proceed entirely offline, relying solely on this small collection of feedback for preference learning.
4 Method
Our approach is inspired by the pretraining-SFF paradigm widely used in NLP, which bears structural resemblance to our cooperative RI. setting: generalization with a policy pool parallels large- scale self-supervised pretraining, while offline preference learning from user-approved trajectories resembles SFT on task-specific data.
Building on this insight, propose three-stage framework (Figure T (1) Pretraining with policy pool to acquire generalizable cooperative behavior; (2) Preference learning with recovery, where offline behavior cloning on human- prefereeeefrred trajectories is followed by generalization recovery to mitigate performance degradation; (3) Epoch-wise alternation recovery, which incrementally refines preference alignment by interleaving imitation and recovery in successive epochs.
This framework enables the agent internalize user preferences without compromising its ability to coordinate with unseen partners. We describe each stage in detail below.

---

Policy pool
Pretrain initialization
Cooperative agent
Online recovery
Offline BC
Epoch-wise Alternation
Generalization
Trajs for evaluation
'like' trajs
Trade-off
Biased style
Post-match
'ikee' rating
Online interaction
User
Normal scenarios
Figure 1: The online&cof fline RL problem and three-stage approach
4.1 Pretrain: generalization prior for cooperative agents
Our pretraining stage corresponds to the online interaction phase described in Section 3.1.1 Specifi- cally, the cooperative agent interacts with a diverse policy pool consisting of previously deployed agents and is optimized using the MAPPO algorithm. This process aims enable the agent to adapt to various human proxies while ensuring task completion. While conceptually similar to the pretrain-and-SFF paradigm in NLP, our setting in Game AI differs several key aspects. Pretraining aims to enhance the model's generalization ability, but cooperative agents typically have significantly fewer parameters than language models, making their training less stable. Moreover, since the objective of preference learning is often misaligned with that of environment reward maximization, offline preference learning frequently causes notable drop in generalization performance referred to as "unlearning" [ummar et al. 2020 Kostrikov et al. 2022
Despite these challenges, pretraining crucial. Applying preference learning to randomly ini- tialized agent with limited user data leads to severe overfitting. To analyze this empirically, we compare pretrained agent and randomly initialized agent by performing behavior cloning to convergence on small number of offline trajectories from three distinct preference styles. After that, both agents undergo generalization recovery phase via online interaction with the policy pool. As shown in Figure? 2 the blue curve represents the agent's average environment return when cooperating with partners from the policy pool. The results indicate that achieving comparable performance without pretraining requires substantially more training. Furthermore, when evaluated under specific preference styles, agents trained from scratch often fail to recover the generalization performance of their pretrained counterparts, even after extensive optimization. This suggests that, for smaller models, pretraining not only enhances data efficiency but also prevents irreversible loss of generalization. Overfitting to user preferences from scratch may impair the agent's ability to exhibit broadly cooperative behavior.
4.2 Preference learning with recovery: empirical analysis of the conflict between offline behavioral cloning and online generalization
To simultaneously pursue preference learning convergence and preserve generalization ability (i.e., maintaining performance comparable to the agent initialized from pretraining), as shown in Algo- rithm the second stage of our algorithm first performs offline behavior cloning on human- -imaan-reeerred trajectories, followed by a generalization recovery phase to mitigate potential performance degrada- tion. Although the optimization objectives of these two procedures are not entirely consistent, our experiments this section demonstrate that such two-step design helps achieve better balance between preference learning and generalization preservation.
5

---

Return, wio Pretrah
Returt Pretish
Acorocy ANO Pretran
Accuncy, Fretral
Oridin Fressh Stye
Style
Sryle
Rebuth
200
AuCes dode THa
SUACY of griat
1597 1950 1995 InKes 32503 19t6e 17595 20000
2006 9900
7506 1n80 18990 1500
19993 20000
5001 7006 1906 32360
1996 17900 2000
sodees
Exhodes
Edlondes
Figure 2: Recovery dynamics after behavior cloning. Dashed lines indicate pretrained agents, while solid lines indicate agents trained from scratch (no pretraining). Blue curves represent the average enviromment return over the policy pool, and orange curves represent the action accuracy of the same pretrained agent evaluated on three distinct preference datasets ("like" datasets). Horizontal dotted lines denote the initial performance of the pretrained agent.
To better understand the optimization dynamics between preference learning and generalization, we further analyze their interaction using the same experimental setup described earlier. In addition to environment return, we also track the agent's ' action agreement accuracy on the "liked" trajectories, which indicates how well the agent maintains aligmment with user preferences throughout training.
As illustrated in Figure we visualize the trade-off using both metrics. We observe that when a well- generalized cooperative agent is trained to convergence on the preference dataset, its environment return can drop by one-third or more when evaluated with the policy pool. In contrast, recovering generalization is considerably slower, the agent must relearn how to cooperate with diverse partners across a range of game situations.
Empirically, fully recovering generalization often requires tens of thousands episodes of online training. Moreover, this recovery process can partially degrade the agent's preference learning performance. Interestingly, after the recovery phase, the agent does not entirely forget its prior aligmment. Its accuracy on the "liked" trajectories remains higher than that of the initial pretrained agent, indicating that some preference information is retained. We observe a similar phenomenon under lightweight preference learning: even with only one epoch of behavioral cloning, the agent retains partial preference learning after the recovery phase. Detailed results are presented in appendix
A This observation motivates the third stage our approach-an alternating training strategy that consistently interleaves preference learning with generalization recovery to achieve stable and effective integration both objectives.
4.3 Epoch- alternation recovery
Based on the above observations, propose epoch-wise alternation recovery, where each round consists of one epoch of BC followed by generalization recovery. This strategy allows for further improving preference alignment while preserving generalization.
Building on the three stages of our approach, we outline practical training algorithm (see Algo- rithm J. Starting from pretrained agent that has converged on the policy pool (Lines 3). we first record its average environment return as the generalization bascline. In the second stage, we perform sufficient behavior cloning the preference trajectories until convergence and record the correspond- ing accuracy on the offline dataset (Line 5 We then recover the agent's generalization ability to match that of the original pretrained agent. In the third stage, apply epoch-wise alternation one epoch of BC followed by recovery- until the preference accuracy reaches the level obtained after full convergence in the second stage (Lines 10)
This epoch-wise alternation cycle repeated until either the preference accuracy recovers to the level achieved in the second stage or predefined maximum number of iterations is reached. By adopting this shallow and incremental training scheme, we enable gradual improvement in both generalization and preference alignment, while maintaining overall training stability.

---

Algorithm Epoch-wise alternation recovery
M 1-=1- offline preference data D, environment &, initial agent x; 2: Output: Well-trained cooperative agent x;
1: Input: Policy pool {
3: Train т with { }; record average return r gegnn # Stage 1
4: BC train on D until convergence; record accuracy yi; # Stage 2
5: Recover π with policy pool on & until avg. return 2 Tgen or budget exhausted;
6: repeat # Stage 3: epoch-wise alternation recovery
7: Train on D for I epoch BC;
8: Recover и with policy pool on & until avg. return Ugen budget exhausted;
9: Evaluate accuracy Neurr on D;
until Meurrr 2 1 or budget exhausted;
11:
return Final policy x
5
Experiments
In this section, we aim to answer the following research questions through the experimental evalua- tions: (1) Can the 3-stage approach effectively align the cooperative agent's behavior with that of target human proxy, producing the desired interaction style? (2) Beyond the specific human proxy used for preference learning. does the aligned agent retain its ability to collaborate effectively with other unseen human proxies? (3) Is the epoch-wise alternation recovery necessary, and how does it influence the agent? Can the this stage be completed within limited training budget?
5.1 Environment and human proxy with different preferences
Following prior works [Strouse et al.
2021 Yu et al. 2023 Zhao et al. 2023 we adopt the 2019 to simulate two-player cooperation. this setting,
Overcooked environment [Carroll et al. cooperative agent and human proxy must coordinate to complete cooking tasks within time limit. Agents perform actions such as moving, picking up, and delivering ingredients to prepare and serve soups. Success depends well in self elay may still fail to collaborate effectively with a human proxy. More details about the environment will be shown in appendix B
coordination rather than individual skill skill-agents that perform
To simulate user preferences, we follow the framework used in HSP and ZSC-Eval, modeling preferences as latent preference reward functions that deviate from the environment reward. As human preferences are typically event-centric, we induce stylistic diversity by assigning biased weights to specific gameplay events. Formally, we define the reward class as:
Tw, where dos, a1 a2) denotes features of state-action pairs and encodes preference bias. A sampled reward R., induces a human proxy policy by reaching a Nash equilibrium via self-play with a standard cooperative agent, yielding partner policy with consistent, biased preferences.
R (Rw Ra (s, a1, a2) o(s, 01, a2) w. |w/co S (maa),
5.2 Benchmark, baselines, evaluation protocols
Benchmark. Following the assumptions in HSP Yu al. 2023 we construct a set of N biased reward functions [RE 1 For each R train policy pair (Tf. =t) via self-play reach Nash equilibrium. Each pair is used to generate two trajectories, yielding a 2N-sized offline dataset. To define human preferences, we randomly sample three biased reward functions from {Pt,) ) and use each to evaluate all trajectories. For each sampled function, we retain the trajectories whose biased return reaches at least 80% of the maximum return under that preference, and treat them as liked trajectories. The corresponding human-side policies {Tí in these trajectories serve as test-time collaborators for evaluating the cooperative agent. To better simulate the diversity of human preferences, we employ multiple stylistically similar human proxies to introduce behavioral variability, For both pretraining and recovery, we adopt the same policy pool as used in the FCP [Strouse et al.. 2021], which comprises suboptimal checkpoints from self-play under neutral reward. This ensures that preference adaptation arises purely from the offline trajectories. We conduct experiments on two Overcooked layouts (Coordination Ring, Many Orders), with detailed layout configurations provided
7

---

in Appendix B For each setting, results are averaged over three runs with different random seeds to ensure robustness. Hyperparameter details are provided in Appendix C
Baselines We denote the benchmark-proyeee golden trajectories as Offline Trajs. We compare
the proposed method against the following baselines commonly used in preference-aware multi- aggett reinforcement learning: (1) Behavior Cloning (BC) [Torabi et al., 2018), which directly trains an agent via behavior cloning on the offline preference trajectories without any pretraining: (2) ZSC (Pretrain) [Strouse et al.) [202]. where a generalizable cooperative agent is trained using FCP, a representative ZSC method. To control for the effect of implicit preference alignment during pretraining, we intentionally select sub-optimal checkpoint; (3) BC on Pretrained Agent (Pretrain + BC), which initializes the agent with the FCP-pre PCP-pretrained agent and further mne-tunes using behavior cloning the offline preference data; (4) BC and Recovery on Pretrained Agent (Pretrain + BC + Recover), which first applies behavior cloning on the FCP-pretrained agent and subsequently performs online generalization recovery using interaction with a diverse policy pool.
Evaluation protocols. The evaluation focuses on the cooperative agent's performance when paired with human proxies in rollouts. We propose two metrics aligned with our dual objectives of task competence and preference learning:
Environment Return.
This metric evaluates the agent's task completion capability in the Over- cooked environment, quantified by the total number of successfully delivered soups. Specifically, EnvReturn = 20 Naelivered. where Naelivered denotes the number of completed deliveries.
Preference Score. This metric measures the agent's alignment with human preferences, modeled as event- rewards. Each preference model assigns biased weights to key gameplay events such as picking up ingredients, cooking, or delivering dishes. The preference score is computed as:
PreferenceScore
Σ freq(e:) score(e:),
where e, denotes the i-th event type, freq(e:) its observed frequency, and score(e,) the corre- sponding reward weight determined by the preference function Rm, Detailed event definitions and scoring schemes are provided in Appendix B
5.3 Preference learning performance (RQ1)
The main experimental results are summarized in Table The two metrics respectively evaluate the agent's general- ization capability and its alignment with user preferences given limited supervision. The results highlight the im- portance of pretraining: agents trained purely via BC on small offfine dataset often fail to complete tasks effectively. Likewise, ZSC-based methods trained without access to preference-specific data tend to exhibit generic behavior styles and struggle to coordinate efficiently with human agents whose actions deviate from those seen in the policy pool. Simply fine-tune tne-ttning pretrained agent via BC results in unstable performance, with some preference styles lead- ing to degraded task execution. This further emphasizes the necessity of incorporating recovery phase to restore generalization. In addition, we conducted supplemen- tary experiments with diversity-oriented ZSC approaches (MEP [Zhao et al. 2023 and TrajDi [Lupu et al.) (202T))
Seraccia petrocer
preference hearning
Figure 3: UMAP visualization for event- count vectors from Style A. The centers of three clusters are denoted as black edges.
which primarily enhance policy pool diversity without leveraging offline preference data. Their relatively weaker performance on our benchmark further highlights the necessity of incorporating offline preference learning. Detailed results are provided in Appendix
At the behavioral level, we further analyze the agent's stylistic alignment by computing event-count vectors from trajectories generated by the offline dataset, the pretrained cooperative agent, and our method. We then apply UMAP for dimensionality reduction and visualization. As shown in Figure 3 trajectories generated by our method (orange points) are closer to those in the offline dataset (red
8

---

Table 1: Performance across different preference styles. Each method is evaluated using Preference Score (f) and Environment Return (f). Results are reported on the layout Coordination Ring; results on Many Orders are provided in Appendix D
Style
Style B
Style
Method
Pref. Score
Env. Return
Pref. Score
Return
Pref. Score
Env. Retur
Offline Trajs
670.0
225.0
235
121
552
224
Pretrain
111.3 = 6.6
47.6
129.3
60.9 = 2.6
188.7 = 6.2
84.0
Pretrain
203.5 39.7
53.2 1 12.8
178.0
33.7
82.5
20.8
120.7 5.00
34.6 L 15.8
BC Only
95.4 34.9
13.7 7.1
97.7
20.2
31.9 +: 12.5
44.2 4..3
4.6 6.5
Pretrain BC Recover
224.3 66.0
82.0 + 24.2
187.7 + 18.5
96.3 + 10.9
252.0+99.4
109.7 -
EAR
274.7 74.0
38.6:31..0
200.3# 17.0
103.6 9.4
259.0 = 9.9
110.3 3.3
points), indicating better alignment with user preferences. Moreover, by comparing the centroids of each distribution, we observe clear shift from the pretrained agent's behavior (blue points) toward that of the offline datasets. This directional shift provides additional evidence that our method effectively integrates stylistic preferences into the cooperative agent's behavior. More detailed case analyses are provided in the Appendix
5.4 Generalization ability (RQ2)
Beyond preference learning, we evaluate the agent's task performance by measuring its environ- ment return when cooperating with various style- specific proxy agents. As shown in Table com- pared to the agent trained with standard pretraining alone, our method achieves strong performance when cooperating with the proxy agents used to generate the offline trajectories. In certain cases, such style B, applying behavior cloning directly on the pretrained agent also yields competitive per- formance. To verify the necessity of the recovery phase and to more comprehensively evaluate the agent's generalization ability, we further test the co- operative agents against benchmark proxy agents that were not involved in the offline preference
Table 2: Average environment returns (f) across all human proxy agents πj, (the N biased human proxies defined in the benchmark) under differ- ent methods and preference styles. As the pre- trained agent is style-agnostic, its performance is reported only once.
Preference Style A
BC
Pretrain BC
EAR
5.1 2.3
14.6 1 3.6
86.2 + 12.6
Style B
12.1 + 5.0
31.3 9.7
77.1 + 2.5
Style C
4.2
28.6 + 19.8
73.9 = 8.5
Pretrain
72.7 0.1
data. The results in Table show that our method maintains generalization ability comparable the original ZSC-trained agents, despite being fine-tuned for specific human preferences. In contrast, the pretraining followed by supervised fine-tuning variants suffer sharp drop in generalization perfor- mance. This highlights that our method effectively balances preference alignment and generalization, and further underscores the necessity of the recovery phase.
5.5 Impact and feasibility of epoch-wise alternation recovery (RQ3)
In this section, we validate the necessity of the epoch-wise alternation recovery. As shown in T compared to the conventional two-stage approach-pretrainigg followed by behavioral cloning and generalization recovery-ov method achieves higher preference scores. To further ex- amine the learning dynamics under different train- ing paradigms, we compare the action accuracy on the offline "liked" trajectories after recovery. The comparative results are presented in Table 3 as consistent with Section 4.2 observe that
Table 3: Action accuracy (1) of behavior cloning under different training methods and preference styles.
Method
Style A
Style B
Style C
Pretrain BC 0.73 0.02 0.64 + 0.01 0.64 0.01
Recover
0.66 + 0.03
0.59 + 0.05
0.63 + 0.01
EAR
0.73 0.02 0.64 + 0.01 0.64 0.01
applying generalization recovery after fully converged BC can significantly degrade action accuracy on the "liked" trajectories- by up to 10% under certain preference styles (e.g., and B). In contrast, epoch-wise alternation recovery, which interleaves one epoch of BC with recovery phase, enables the agent to incrementally restore action accuracy while maintaining generalization capability. No- tably, for preference styles that suffer greater accuracy drop, this strategy leads to more substantial

---

improvements in preference scores, highlighting the necessity of epoch-wise alternation recovery in such scenarios. These results further demonstrate that even garameter-efficienn agents can effectively acquire and retain human preferences when trained through a stable and iterative alternation process.
Table 4: Number of rollouts performed when the EAR algorithm converges under different preference styles in the Coordination Ring layout (maximum rollout batch 250).
Max=250
Style A
Style B
Style C
EAR
204.3 + 30.0
188.3 26.4
205.7 + 24.6
Regarding algorithmic feasibility, our design ensures that the EAR algorithm exhibits stable conver- gence in practice. During the epoch-wise alternation recovery phase, training proceeds iteratively, where each rollout batch contains 200 trajectories, and each trajectory spans 400 environment steps. This configuration yields approximately 200 400 8 10 steps per batch, with a total environment interaction budget of 2 x 10 steps throughout the entire training process. At the end of each rollout batch, we evaluate the cooperative agent's environment return over the policy pool; if the criterion is met, preference accuracy is assessed (see Algorithm []. We record the number of rollout batches required for convergence under three distinct preference styles in the Coordination Ring layout, as summarized in Table 4 The results confirm that the EAR algorithm converges reliably within reasonable training budget, demonstrating its practical feasibility.
Conclusion and future work
In this paper, motivated by practical demands in the gaming industry, we propose a novel reinforce- ment learning problem that jointly considers generalization ability and preference aligmment. Drawing inspiration from the pretraining followed by supervised fine-tuning paradigm in NLP, we design 3-stage training framework that integrates online generalization learning with offline preference modeling. Furthermore, we introduce an alternating optimization strategy to stabilize the training process. Our method is empirically validated through extensive experiments across diverse scenarios, demonstrating its effectiveness. While current work focuses the Overcooked environment-wwhid is representative yet relatively simple-we plan to extend our approach to more complex environments in the future. Additionally, we aim to explore applications beyond the gaming domain, broadening the impact and applicability of the online&tof fline RL problem.
Acknowledgments
The corresponding author Shuai Li is sponsored by CCF-Tencent Open Research Fund.
References
Youssef Abdelkarcem, Shady Shehata, and Fakhri Karray. Advances in preference- reinforce- ment learning: A review. In 2022 IEEE international conference on systems, man, and cybernetics (SMC), pages 2527-2532. IEEE, 2022.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Alain Andres, Lukas Schäfer, Stefano V Albrecht, and Javier Del Ser. Using offline data to speed up reinforcement learning in procedurally generated environments. Neurocompuiing, 29079,,, 2025.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
10

---

Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577-1594. PMLR, 2023.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27): 819-840, 2002.
Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-Al coordination. Curran Associates Inc., Red Hook, NY, USA, 2019.
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245-22) 259, 2017.
Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Graepel. Cooperative ai: machines must learn to find common ground. Nature, 593(7857):333336 2021.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforee- 2017.
ment learming. arXiv preprint
Siyuan Guo, Lixin Zou, Hechang Chen, Bohao Qu, Haotian Chi, Philip S Yu, and Yi Chang. Sample efficient offline-to-online reinforcement learning. IEEE Trunsactions on Knowledge and Data Engineering, (29)::1299-130,, 2023.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
Zhaolin Hu and I. Jeff Hong. Kullback-leibler divergence constrained distributionally robust opti- mization. Available at Optimization Online, (2)):9 2013.
Glen Klien, David D Woods, Jeffrey M Bradshaw, Robert R Hoffman, and Paul J Feltovich. Ten challenges for making automation a" team player" in joint human-agent activity. IEEE Intelligent Systems, 1966)91195, 2004.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 11X:v:21100061109, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit
q-learning. In International Conference on Learning Representations, 2022. URL ht openrevieww www.reviieinee/oorurr2...
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-laaniig for offline reinforcement learning, Advances in neural information processing systems, 33:1 13:1119-119,, 2020.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-effiient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106600001, 2021.
Scunghyun Lee, Younggyo Sco, Kimin Lee, Picter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1712. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review. and Perspectives on Open Problems, 5, 2020.

---

Xu Liu, Haobo Fu, Stefano V Albrecht, QIANG FU, and Shuai Li. Online-to to to-ofiee RL for agent alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL
httis://openrrviier wwt.://wwweeeterooe//ddd////eae
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International conference machine learning, pages 7204-7213, PMLR, 2021.
Yihuan Mao, Chengjie Wu, Xi Chen, Hao Hu, Ji Jiang, Tianze Zhou, Tangjee Lv, Changjie Fan, Zhipeng Hu, Yi Wu, Yujing Hu, and Chongjie Zhang. Stylized offline reinforcement learning: Extracting diverse high-quality behaviors from heterogeneous datasets. In The Twelfth International Conference on Learning Representations, 2024. URL rnHNDihrIT
net/ fer/ffrum?idd
Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V. Albrecht. A survey ofnbsp;ad hoc teamwork research. In Multi- Agent Systems: 19th European Conference, EUMAS 2022, Düssseldorf, Germany, September 14-16, 2022, Proceedings, page 275-293, Berlin, Heidelberg, 2022. Springer- Verlag. ISBN 978-3-031--0613-9, doi: 0000007977883001-206661 16. UtI.httss:// //op:://doi... org 278-3-031-22014-663
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning, Advances in Neural Information Processing Systems, 36:62244 62269, 2023.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions Neural Networks and Learning Systems, 2023.
Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality diversity: new frontier for evolu- tionary computation. Frontiers Robotics AI, 3:40, 2016. Utt/ttps://aan semanticschol.a
org/CorpusID: :1111377
21713708
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 2020.
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.067//6 2022.
Peter Stone. Gal A. Kaminka, Sarit Kraus, and Jeffrey S. Rosenschein. Ad hoc autonomous agent teams: collaboration without pre-coordination. In Proceedings the Firett-Fourrth AAAI Confer- ence on Artificial Intelligence, AAAI 10, page 1 1504-1509. AAAI Press, 2010.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-145155 2021.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint 108.0:1805.01954. 2018
Shenzhi Wang, Qisen Yang. Jiawei Gao, Matthieu Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song. and Gao Huang. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning, Advances Neural Information Processing Systems, 36:4708 1770088-4711 1-47104, 2023.
Xihuai Wang, Shao Zhang, Wenhao Zhang, Wentao Dong, Jingxiao Chen, Ying Wen, and Weinan Zhang. 7sc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination. Advances in Neural Information Processing Systems, 37:47344 47377, 2024a.
12

---

Zhichao Wang. Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mchrotra, Xiang- Mao, Sitaram Asur, et al. A comprehensive survey llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16211 2024b.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46, 2017.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Thirly-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https //ooenrevvew.neeee wttt//orrumttttWaaatB........
Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning zero-shot cooperation with humans, assuming humans are biased. arXiv preprint arXiv:2302.0166) 00000. 2023.
Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress cooperative multi-agent reinforcement learning in open environment. CoRR, 2023,
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00335 2023.
Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-baseet training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6145-6153, 2023.
13

---

A Lightweight behavior cloning and online recovery analysis
We further analyze the interplay between lightweight behavior cloning and online recovery. On the randoml layout, we start with a pretrained agent and conduct 1-epoch behavior cloning on the "liked" trajectories for each of the three preference styles. This is followed by online recovery using the policy pool. The corresponding training curves are shown in Figure
Rlturnie Original Prefrain
220
302
339
0355
1.5
359
342
120
130
023
Actuncc
Natam ACTURO
1000 2000
3000 420
1200
Figure 4: Training curve online recovery after one epoch of preference behavior cloning. The blue horizontal line indicates the environment return of the pretrained agent on the policy pool. The yellow curve shows the agent's action accuracy on the offline "liked" trajectories during recovery.
Our results reveal that even minimal application offline preference caamiing--amllyy a single round of behavior cloning on a small dataset-can lead to substantial degradation in generalization performance, with environment returns dropping by more than one-third. This highlights clear inefficiency mismatch between behavior cloning and recovery, where the latter requires thousands of online interaction episodes to restore the agent's original performance level.
Interestingly, we observe that after recovery. the agent not only regains its generalization ability but also retains partial alignment with the offline "liked"" trajectories. This indicates that preference information is not entirely forgotten during recovery, providing empirical support for the effective- ness of our Epoch-wise Alternation Recovery reinforcing preference learning while maintaining generalization.
B Overcooked benchmark details
The Overcooked environment Carroll et al. 2019 is a widely adopted benchmark for evaluating cooperative behaviors in reinforcement learning. Inspired by the multiplayer video game Overcooked!, simulates a kitchen where two players, each controlling a chef, must coordinate within a confined space and under time constraints prepare and deliver soups. Upon successful delivery, both players receive a shared reward. The primary objective is to maximize the cumulative team reward through effective collaboration.
To prepare a soup, agents typically follow a multi-step process: (1) collect and place the correct ingredients into pot according the recipe; (2) wait for the soup cook for fixed duration; and (3) serve the cooked soup on a dish and deliver it to the designated location. Each agent operates in a discrete action space: up, down, left, right, interact, no-op. Beyond learning the mechanics of soup preparation, agents must also adapt to the behavioral tendencies and preferences their partners. High team performance requires precise coordination and mutual understanding between agents.
B.1 Layouts
Our work builds upon the Overcooked version used the HSP benchmark Yu et al. 2023 primarily focusing two layouts: Coordination Ring and Many Orders.
As shown in the left part of Figure 5 Coordination Ring features a ring-shaped kitchen layout. The tightly arranged space facilitates faster soup preparation, with ingredients, serving station, and plates all located in the bottom-left corner, and two pots positioned in the top-right. The key coordination challenge in this layout lies in the agents' movement directions to avoid collisions. The recipe is
14

---

Coordination Ring
Many Orders
$ 2 20 B
20
20 20
10 10
Figure 5: Two layouts in experiments.
relatively simple-only onion soup is required- and each soup takes 20 time steps to cook, yielding reward of 20.
Many Orders, contrast, adopts a square-shaped kitchen and introduces greater recipe complexity, It includes three types of orders: onion, tomato, and mixed recipe requiring onion and 2 tomatoes. Each recipe provides a different reward. In this layout, the central coordination challenge recipe selection. If the human partner shows preference for particular recipe, the cooperative agent is expected to align with that preference to achieve satisfactory collaboration.
B.2 Events
Following the settings established in HSP, we adopt the same event taxonomy define key sub-events within each layout. For both layouts (Coordination Ring and Many Orders), the primary sub-events and their corresponding event scores are listed as following Table 126
Table 5: Event catagories and responding reward weights of Layout Coordination Ring
Event (e,)
Event reward weight (score(e;))
Picking up an onion from the onion dispenser
-10, 0, 10
Picking up dish from the dish dispenser
0, 10
Picking ready soup from the pot with dish Placing an onion into the pot Delivering a soup to the serving area
-10, 0, 10
-10, 0, 10
-10, 0
Table Event catagories and responding reward weights Layout Many Orders
Event (e:)
Event reward weight (score(e:))
Picking up onion from the onion dispenser
-10, 0, 10
Picking up dish from the dish dispenser Picking up tomato from tomato dispenser
10
0, 10, 20
Picking up a soup
-5,
Viable placement
-10, 0, 10
Optimal placement
-10,0
Catastrophic placement
10
Placing an onion pot
-3,0,3
Placing tomato into the pot
-3,0,3
Delivering soup the serving area
-10,0
15

---

B.3 Offline "like" trajectories
For the offline datasets, we collect trajectories across two layouts and three preference styles, with the number of trajectories for each combination summarized in Table 7 For each layout-style pair, we split the trajectories into training and test using 4:1 ratio. In our reported results, action accuracy refers to performance evaluated on the test set.
Table Number of trajectories in the offline datasets across two layouts and three preference styles. Each set is split into training and test sets with 4:1 ratio.
Layout
Style A
Style B
Style C
total
Coordination Ring
24
20
Many Orders
38
14
12
72
C Hyper parameters
n this section, we present the hyperparameters used and compute resources in our experiments.
C.1 Agent architecture
Following the experimental settings in HSP, adopt CNN-MLP based architecture for all experi-
ments. The convolutional module consists of three layers with output channels {32, 64, 32). Each layer kernel size of 3, padding of 1, and stride of 1. A pooling layer is applied after the convolutional block, followed by two-layer MLP that outputs the final action logits.
C.2 FCP policy pool
To construct the policy pool, perform self-play training with the objective of maximizing vironment rewards. A total 6 self-play runs are conducted, and for each run, we select three checkpoints-iiiiie middle, and final rinal-resulting in policy pool containing 18 diverse agents. The hyperparameters used for the self-play procedure are listed in Table 8
Table 8: Hyperparameters of self-play process
Hyperparameters
Values
Entropy coef'
0.01
Gradient clip norm
10.0
GAE lambda
0.95
Gamma
0.99
Value loss
huber loss
Huber delta
10.0
Mini hatch size
batch size / mini-batch
Optimizer
Adam
Optimizer epsilon
le-5
Weight decay
Network initialization
Orthogonal
Use reward normalization
True
Use feature normalization
True
Learning rate
5e-4
Parallel environment threads
100
Ppo epoch
Environment steps
10M
Episode length
400
Reward shaping horizon
100M
16

---

C.3 Pretraining, recovery, behavior cloning
Since pretraining and online generalization recovery share the same underlying procedure, adopt identical hyperparameters for both stages. Additionally, the hyperparameters used for behavior cloning during preference learning are also summarized in Table 9
Table 9: Hyperparameters of pretraining, online recovery, behavior cloning.
Hyperparameters
Values
BC batch size
32
BC epoch stage 2
16
Hyperparameters
Values
N training threads
N rollout threads
200
Entropy coef
Gradient clip norm
10.0
GAE lambda
0.95
Gamma
0.99
Value loss
huber loss
Huber delta
10.0
Mini batch size
batch size / mini-batch
Optimizer
Adam
Optimizer epsilon
lc-5
Weight decay
Network initialization
Orthogonal
Use reward normalization
True
Use feature normalization
True
Learning rate
Se-4
Parallel environment threads
100
Ppo epoch
Environment steps
20M
Episode length
400
Reward shaping horizon
100M
D Results on the Many Orders Layout
Table 10: Performance across different preference styles. Each method evaluated using Preference Score (t) and Environment Return (1). Results are reported on the layout Many Orders
Style A
Style
Style C
Method
Pref. Score
Env. Return
Pref. Score
Fav. Return
Pref. Score
Fnv. Retur
Offline Trajs
244.7
203.4
669.3
336.4
1002.5
378.3
Pretrain
194.3 18.0
166.7 = 16.0
124.2 = 17.4
125.0
29.7
227.3 + 92.1
120.2 4 63.0
Pretrain + BC
145.0 = 16.8
116.4 = 138
334,0 +
340.0 2.2
616.3 27.1
337.3 4 15.2
BC Only
51.9
62.8 41.5
314.0
316.3
11.0
661.31 13.5
1: 54.3
Pretrain BC Recover
213.7 5.7
4.6
228.3 1 44.1
230.3
59.9
390.7
102.9
217.0
EAR
212.7 4.5
178.3 4.2
239.0 +: 53.8
239.3
68.8
395.3 = 117.1
222.3 4: 65.8

---

Table 11: Average environment returns (1) across all benchmark human proxy agents TH under different methods and preference styles in layout Many Orders. Since the pretrained agent is style- agnostic, its performance is reported only once.
Preference
BC
Pretrain + BC
EAR
Style A
33.6 21.6
14.6 3.6
146.3 7.9
Style B Style C
68.2 1 4.5
86.5 + 3.5
172.3 1 26.0
65.0 1 2.9
67.9 1: 2.6
195.3 1 21.1
Pretrain
142.0 + 22.9
In this section, we present the experimental results on the Many Orders layout. All results are averaged over three runs with different random seeds. As shown Table TO, our proposed method, EAR, achieves consistent improvements across both metrics compared to standard BC and Recovery on Pretrained Agent. However, we observe that for certain styles (e.g.. Style and C), simple behavior cloning can already achieve excellent performance. This is largely due to the fact that partners in this benchmark exhibit strong adaptability- favorable condition that is not commonly observed and has been refuted by results in Style A and the Coordination Ring layout. Overall, our method demonstrates robust and competitive performance across diverse settings, highlighting its effectiveness under more realistic and challenging scenarios.
Furthermore, when cooperating with proxy agents that are distinct from the "like" trajectories in the benchmark, we investigate the generalization capability of collaborative agents. Table IT reveals that while standard BC leads overfitting on offline trajectories resulting degraded performance on out-of cuttof-diisribbutoo scenarios our method maintains its effectiveness. This finding underscores the importance of the recovery in enabling generalization.
E
Case analysis
In this section, we provide a behavioral analysis the agent trained with EAR, focusing on its coordination patterns under Layout Coordination Ring with Style A, which corresponds to Figure 3 We compare three types of trajectories: (1) the human-preferred offline trajectory, (2) the generalizable pretrained agent, and (3) the EAR-trained agent that balances generalization and preference alignment.
The preference vector for Style A is summarized in Table [2 emphasizing actions such as "picking up dish from the dish dispenser" and "picking up an onion from the onion dispenser." In essence, the human proxy tends to favor trajectories that include these two operations while completing the cooperative cooking task.
For each method, we select a representative trajectory and visualize frames sampled every 40 timesteps from the first 200 steps. As shown in Figure 6 the offline human proxy (top row) primarily stays near the dish and onion dispensers, focusing on delivering items to the teammate- demonstrating a clear division of labor consistent with the stated preferences. In contrast, the pretrained agent (middle row), trained solely for reward maximization, fails to account for such stylistic tendencies. When the pot nearly ready, it prematurely moves pick up dish for serving, thereby blocking its partner and disrupting coordination. The EAR-trained agent (bottom row) achieves a more balanced behavior: respects the human proxy's preferred roles while maintaining adaptive flexibility, taking initiative fetch dishes when appropriate. This illustrates how EAR effectively reconciles the trade-off between preference alignment and generalization.
Table 12: Event catagories and responding reward weights of Layout Coordination Ring
Event (e.)
Event reward weight (score(e:))
Picking an onion from the onion dispenser
10
Picking up a dish from the dish dispenser
Picking up ready soup from the pot with dish Placing an onion into the pot Delivering soup to the serving area
-10
0
-10
18

---

Offline trajs
score
score:
score: 20
score:
score: 80
score: 100
Pretrain
score:
score:
score:
score: 20
score:
score: 20
EAR
Score:
score:
score: 20
score: 60
score: 80
score: 100
Figure 6: Representative trajectory segments from three agents (offline dataset, pretrained agent, and EAR EAattraieed agent) under style A in the Coordination Ring layout
Table 13: Performance across different preference styles. Each method is evaluated using Preference
Score (f) and Environment Return (1). Results are reported on the layout Coordination Ring.
Style A
Style B
Style C
Method
Pref. Score
Env. Return
Pref. Score
Env. Return
Pref. Score
Env. Return
Offline Trajs
670.0
225.0
235
121
552
224
MEP
118.1 16.6
35.6 6,2
114,3 3.4
49.9 3.7
224.0 12.0
85.5 5.3
TrajDi
107.2 + 13.3
31.1 = 5.5
103.5
13.5
39.4
1 8.1
202.3 + 17.2
75.5 + 8.3
EAR
274.7 + 74.0
88.6 + 31.0
200.3 17.0
103.6 9.4
259.0 9.9
110.3 + 3.3
EAR Compared with Pretraining Methods Focused on Enhancing Diversity
Our approach specifically targets scenarios where a pretrained agent's generalization fails to adapt to user's unique preferences-2 situation commonly observed in both real-world and experimental settings, especially as environments become more complex (e.g., League of Legends, Honor Kings, and Brawl Stars).
We acknowledge that when biased human agent is unavailable, partner diversity becomes an important consideration and has been explored in several ZSC studies. However, the key insight of these diversity-oriented methods lies in maximizing environmental rewards through coordination with diverse yet representative set of partners, rather than aligning with seer-preference-relaae rewards. Consequently, when the biased human proxy's preferences fall outside the distribution covered by the pretrained policy pool, simply enlarging the diversity of policies during pretraining does not effectively resolve the misalignment issue.
To empirically validate this claim, conducted experiments using two representative ZSC methods (MEP [Zhao al. 2023) and TrajDi (Lupu al.) (021]) that explicitly emphasize policy pool diversity. We selected pretrained checkpoints from ZvC-eval [Wang et al. 2024a] (built upon the HSP Yu al. 2023) framework), using three random seeds and a policy pool size of 36- 36-larger than the pool used in our main experiments. When evaluated against the biased-
biased-preference preference human proxies our benchmark. As shown Table these methods exhibited significantly weaker performance compared with EAR.
19

---

This finding supports our hypothesis that, when user preferences extend beyond the coverage of the pretrained policy pool, developing methods that leverage small number of liked trajectories is both necessary and effective.
G
Limitations
While our work demonstrates strong empirical performance, it has some natural limitations. Our method is relatively simple and could be further enhanced with more sophisticated designs. Addition- ally, we rely on biased preference proxies to simulate human preferences, which inevitably differ from real human behavior. We also do not fully exploit the abundant unlabeled trajectories left by human players, which could provide additional suboptimal data to further improve preference alignment. Despite these limitations, approach establishes solid foundation for learning cooperative agents that generalize to diverse partners while aligning with human preferences, and it opens promising directions for future exploration.
H
Compute resources
All experiments were conducted on single NVIDIA GeForce RTX 2080 Ti GPU. Table reports the per-run execution time with the number of rollouts set to 200. We focus on the results for the Many Orders layout under Style A.
Table 14: Compute resources
Methods
Execution time (s)
1 1-epoch Behavior Cloning (BC)
27.1
One round of environment rollouts
5.7
PPO (15 epochs) after one rollout
10.5
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented in the abstract and introduction accurately reflect our contributions and scope, as claborated in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
The effectiveness of our method is demonstrated through comprehensive experiments and analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All hyperparameters are provided Appendix Cooo support the reproducibility of our results. We also include the detailed algorithm in Algorithm
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will consider releasing our code with the camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/publiccguu.eee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings are described in Section 5.2 and hyperparameter details are provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct experiments with three random seeds and report the averaged results in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed information about the computing resources used is provided in Appendix HH
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research presented fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed in Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve high-risk data models that could be easily misused.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All external resources used are properly cited and credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any new datasets, models, or other research assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The use of LLMs in this project does not affect the core methodology. scientific rigor, or originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
27


Reviews:
[{"review_id": "OeB3KI1O3m", "reviewer": "Reviewer_YpWX", "summary": "The paper addresses the conflict between generalizing to unseen partners and aligning with limited offline preference data in cooperative reinforcement learning. It introduces a training strategy named Epoch-wise Alternation Recovery, which alternates between one epoch of lightweight behavior cloning and generalization recovery. This strategy enables cooperative agents to maintain strong generalization capabilities while effectively incorporating human preferences. Experiments in the Overcooked environment show that the new strategy achieves both high preference scores and high environment returns, demonstrating that it effectively balances the two objectives.", "questions": "The experimental comparison would be more convincing if it included at least one method from section 2.3 that jointly addresses generalisation and preference alignment.\nThe exposition in Section 4 breaks the announced flow: subsection 4.2 inserts design rationale before stage 2 is introduced. Re-ordering the material so that Stages 1–3 are described sequentially, followed by the design discussion, would improve readability and align with the section’s opening statement.\nSeveral table annotations need clarification. Please specify what the “Offline Trajs” row in Table 1 represents, and ensure the description of Table 2 is consistent between the caption and the main text. Resolving these discrepancies would remove residual ambiguity in the presentation.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Pros.\n\nTackling distribution shift in human‑cooperative reinforcement learning is a timely and underexplored problem. This fresh solution is poised to spark follow‑up studies on generalization and preference alignment under non‑stationary conditions.\nI appreciate that the paper’s core claims rest on a solid foundation: the methods are both principled and well matched to the problem, and the extensive suite of experiments convincingly demonstrates their effectiveness. 3. The three‑stage training pipeline is logically coherent and neatly justified. The paper articulates the rationale for this integration clearly, and the differentiation from prior work is precise, with all key citations in place.\nThe paper is well written and supplies enough implementation detail to reproduce results.\nCons.\n\nThe promise to “describe each stage in detail below” is undermined when subsection 4.2 veers into design rationale instead of covering the second stage. This break in flow makes it harder to follow the pipeline in the order the authors intended.\nThere’s confusion around Table 2: the main text calls out “benchmark proxy agents not involved in the offline preference data,” yet the caption refers to “all benchmark human proxy agents.” Aligning these descriptions is crucial for readers to understand exactly what’s being evaluated.\nAlthough the experiments are thorough, the baseline set skips over promising alternatives—such as methods leveraging successor features or contrastive learning for joint generalization and preference alignment. Including those would strengthen the comparative evaluation and better position the contribution relative to state‑of‑the‑art approaches.", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer YpWX,\n\nWe sincerely thank you for the time and effort spent reviewing our paper and for providing valuable suggestions for improvement. We address each of the comments point by point below.\n\n**Clarification on Section 4.2**  \nWe apologize for the confusion caused by unclear descriptions in Section 4.2 and the corresponding tables. The sentence “describe each stage in detail below” was intended to introduce the methodological description of the three stages of our approach. Specifically, the second stage involves a single round of online generalization recovery after offline behavior cloning. In Section 4.2, our goal was not only to present the detailed implementation of this second stage, but also to analyze its limitations through experiments—thus motivating the necessity of the third stage: Epoch-wise Alternation Recovery. We acknowledge that the current title of Section 4.2 may have been misleading, and we commit to revising it in the camera-ready version to better reflect the section’s content.\n\n**Clarification on Table Descriptions**\nIn Table 1, offline trajs refer to the limited number of offline expert trajectories available during training. These serve as golden trajectories and are generated by preference-aligned agent pairs that have reached a Nash equilibrium through self-play within the benchmark environment. We also apologize for the ambiguity in our phrasing of “agents that were not involved in the offline preference data” and “all benchmark human proxy agents.” Our intention was to convey that, in addition to evaluating on the agents that were involved in the offline preference data, we also conducted evaluations on agents that were not involved in the offline data, in order to assess the generalization ability of the cooperative agent. The combination of results from both sets of agents constitutes what we refer to as “all benchmark human proxy agents,” which is exactly what is reported in Table 2. We will unify and clarify the terminology in the table caption in the camera-ready version to avoid ambiguity.\n\n**On the Choice and Applicability of Baselines**\nThe methods discussed in Section 2.3 are designed for different RL settings. In our proposed setup, the task is cooperative in nature, where each partner induces a different environment dynamic. In contrast, most baselines in Section 2.3 assume that the online environment shares the same dynamics as the offline trajectories—an assumption that does not hold in our case, and which limits their direct applicability. Nevertheless, we made efforts to reproduce ALIGN-GAP [1] (published at ICLR 2025) within our setting.  For reward model learning, we used the ground-truth reward function directly. Specifically, when the preference-based reward of a trajectory exceeds the threshold observed in the offline trajectories, the reward model outputs a “like” signal for that trajectory. The reproduced results under the **random1** layout are as follows: \n\n| Method | Style A (Pref. Score, Env. Return)| Style B (Pref. Score, Env. Return) |  Style C (Pref. Score, Env. Return) |\n|-----------|----------|----------|----------|\n| EAR      | **(274.7 ,  88.6)**    |  **(200.3 , 103.6)**  | **(259.0, 110.3)**\n| ALIGN-GAP | (145.0 , 50.0)     | (131.0 , 58.8)     | (235.0, 100.0)\n\nIt is important to note that the ALIGN-GAP method was originally designed for single-agent settings, where the dynamics of the offline trajectories and the online environment are assumed to be the same. This key difference from our cooperative multi-agent setting, where the dynamics vary depending on the partner, makes the direct application of ALIGN-GAP ineffective. The experimental results further highlight the necessity of adopting our proposed approach.\n\nIf you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.\n\n[1] ONLINE-TO-OFFLINE RL FOR AGENT ALIGNMENT.  (ICLR 2025)"}, {"review_id": "VCYAP6muxd", "reviewer": "Reviewer_7shS", "summary": "At a high level, this work addresses the problem of incorporating human preference data into the training of policies for AI agents that can cooperate with previously unseen partners.  They first pre-train a cooperative policy against a pool of human proxy policies learned using the previously described Fictitious Co-Play framework.  They then allow (simulated) humans to interact with this policy, and \"like\" the joint trajectories that they found to be most successful.  Finally, they alternate between training (under a behavioral cloning loss) on these \"liked\" trajectories, and training against the human proxy pool.  The purpose of this alternating training scheme is to avoid catastrophic overfitting to the small set of human-annotated trajectories.  Experimental results on a subset of the Overcooked zero-shot coordination benchmark show that this method significantly outperforms the FCP baseline against \"stylistically biased\" human partners.  Ablation experiments also show that removing the alternating training setup significantly degrades this performance.", "questions": "1. For the HSP-based policies used to generate the trajectories for behavioral cloning, to what extent were the learned policies \"sub-optimal\" with respect to the base Overcooked reward function?  Ideally, methods like FCP would cover the space of near-optimal joint strategies, so perhaps relatively large reward pertubations could explain its poor performance with the \"biased\" partners?\n2. How well did the joint strategies trained with FCP, and the pre-trained policy trained against them, perform under the base Overcooked reward (without perturbations)?\n3. Was the architecture of the pre-trained policy recurrent (allowing for adaptation)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. The proposed EAR framework proved to substantially outperform the FCP baseline, using only a small set of style-specific trajectories.\n2. Possibly the most interesting result from this work is in Table 1, showing that alternating between BC and \"recovery\" training against the FCP pool leads to significantly improved performance over a single round of BC + recovery.\n\n**Weaknesses:**\n1. HSP is used to generate stylistically biased human trajectories, but is not used as a baseline method itself.  It would seem likely that a policy pre-trained against a pool of agents generated through HSP (trained on perturbed reward functions) would outperform FCP in this setting, even without BC finetuning or alternating recovery.\n2. There are a number of other ZSC methods that might been good baselines here, for example TrajeDi (Lupu et al. 2021) or LIPO (Charakorn et al., 2023).  The poor performance of FCP pre-training in these experiments likely has to do with the diversity of the human partners the method is evaluated against.  Compared to FCP, both TrajeDi and LIPO (along with several other baselines) have mechanisms to encourage diversity within the pool of target agents, and might be expected to perform well even without fine-tuning on human preferences.", "was_revised": true, "final_justification": "The additional experiments the authors discuss, and their further clarifications about the method, have addressed most of my concerns, and I have increased my score accordingly.", "rebuttal": "Dear reviewer 7shS:\n\nWe sincerely thank you for the time and effort dedicated to reviewing our work. Your comments are highly appreciated and help us further improve the quality and clarity of our paper.\n\nWe address each of your points in detail below:\n\n**W1: HSP as a Baseline**\n\nOur work aims to simulate a realistic setting where we are given an available policy pool and environment, along with only a very limited number of biased human trajectories (possibly as few as a dozen). The key goal of our method, EAR, is to effectively infer the preferences of a biased human agent—even when this agent is not part of the policy pool—using only a small amount of offline data. This setup closely mirrors practical constraints in real-world game industry scenarios. In contrast, in the Overcooked environment, if the biased human agent is also available (i.e., present in the policy pool), then the online recovery phase would encounter the same dynamics as the offline trajectories. This setting is already well addressed by existing work such as ALIGN-GAP [1]. Our work instead tackles a more challenging and realistic setting that has not been adequately addressed by prior methods.\n\n| Random1 | Style A (Pref. Score, Env. Return)| Style B (Pref. Score, Env. Return) |  Style C (Pref. Score, Env. Return) |\n|-----------|----------|----------|----------|\n| EAR      | **(274.7 ,  88.6)**    |  **(200.3 , 103.6)**  | **(259.0, 110.3)**\n| ALIGN-GAP | (145.0 , 50.0)     | (131.0 , 58.8)     | (235.0, 100.0)\n\n**W2: ZSC Methods:**\n\nWe agree that in cases where the biased human agent is unavailable, partner diversity becomes important and has been considered in some ZSC works. However, it is important to note that these methods typically aim to maximize environment reward, not to align with human preferences—which is the central goal of our preference learning setup.\n\nTo provide a fair comparison, we reproduced TrajeDi [3] and MEP [2], two representative ZSC methods. For evaluation, we directly used publicly released checkpoints from ZSC-eval [4]. These checkpoints were trained with a policy pool of size 36 (agents with standard preferences), which is larger than ours (18), and thus exhibits stronger generalization ability. The experimental results are summarized below:\n\n| Random1 | Style A (Pref. Score, Env. Return)| Style B (Pref. Score, Env. Return) |  Style C (Pref. Score, Env. Return) |\n|-----------|----------|----------|----------|\n| EAR      | **(274.7 ,  88.6)**    |  **(200.3 , 103.6)**  | **(259.0, 110.3)**\n| TrajDi | (100.0 , 32.5)     | (115.0 , 48.8)     | (190.0, 73.2)\n| Mep | (105 .0, 29.8)     | (112.0 , 45.3)     | (224.0, 82.4)\n\nThese results further demonstrate the necessity of explicitly modeling preference learning, beyond general reward maximization.\n\n**Response to Q1**:\nIn the main text, the term “sub-optimal” is used in two respects. First, in the context of FCP, the biased human agent is unavailable, and its performance in the ORACLE setting is close to that of the offline trajectories—indicating limited improvement. Second, we acknowledge that in the absence of the biased human, methods such as TrajeDi and MEP from the ZSC literature may perform better than FCP in terms of coordination. However, their performance does not show significant advantages over FCP when evaluated under the preference learning objective. \n\nThe relatively poor performance of FCP in our benchmark is indeed due to its inability to cover the space of the biased human, which is a well-known challenge in real-world game industry applications. Human preferences are inherently diverse and difficult to fully anticipate or enumerate. Regarding the “large reward perturbations”: the implicit reward induced by human preferences can deviate substantially from the environment reward. More fundamentally, since the biased human is unavailable, the dynamics encountered during online recovery differ significantly from those observed in the offline trajectories. This distribution shift makes it difficult for agents trained via FCP to adapt effectively to the biased human’s behavior.\n\n**Response to Q2**:\nBelow, we provide the average environment return of the agent trained using FCP across its policy pool, as well as its scores under the three different human preference styles:\n\n| Random1 | Style A (Pref. Score)| Style B (Pref. Score) |  Style C (Pref. Score) |Env. Return|\n|-----------|----------|----------|----------|----------|\n|FCP |209.0|-55.0|244.0|142.0|\n\n**Response to Q3**:\nIn principle, the pre-trained policy can be recurrent. However, in our experiments, we opted for a non-recurrent architecture due to implementation considerations when applying Behavior Cloning. Training with recurrent policies requires handling memory, which involves either storing hidden states for each timestep or adopting a trajectory-wise training scheme. We appreciate your thoughtful question, and we will consider extending our experiments to include recurrent architectures in future work.\n\nIf you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.\n\n[1] ONLINE-TO-OFFLINE RL FOR AGENT ALIGNMENT. (ICLR 2025)\n\n[2] Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination (NeurIPS Cooperative AI Workshop 2021)\n\n[3] Trajectory Diversity for Zero-Shot Coordination. (PMLR 2021)\n\n[4] ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination (NeurIPS 2024)"}, {"review_id": "99u5WaoFJi", "reviewer": "Reviewer_7F41", "summary": "This work proposes a hybrid RL training framework that alternates between offline preference learning and online generalization recovery for cooperative agents. The cooperative AI problem setup assumes pretrained policies for cooperative agent training, and availability of offline preference data. The paper attempts to simultaneously enhance the zero-shot generalization and preference alignment capabilities of the cooperative agent. The training regime and metrics are validated in the Overcooked environment against a number of relevant baselines.", "questions": "1) What are the conditions on the set of pretrained policies available to the agent? How much should policies be reflective of/adhere to the preferences in the offline data?\n2) What exactly does lightweight behavior cloning mean in the training regimen?", "limitations": "The limitations of the training paradigm and dataset collection need to be stated.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1) The paper is largely clear and well-structured.\n2) The work proposes an interesting new tweak of epoch-wise alternation recovery, and provides empirical experimental evidence of its success.\n\n### Weaknesses\n1) The post-match \"like\" assumption is informative at the trajectory level, but may not be so at the granular action level. The trajectory is endorsed by the human, but the human may still have corrections or different preferences for some individual actions at specific states. This may affect zero-shot coordination.\n\n2) The training method of epoch-wise alternation recovery proposed involves performing generalization recovery with the policy pool until either the average return is greater than the original value or the budget is exhausted. This can be computationally expensive, considering recovery is generally slow. It would be useful to get information on the budget limit used, the number of times the budget was exhausted, and the average time taken for recovery, etc. In addition, how would these change for a more diverse policy pool, etc.\n\n3) Section 5.3 needs to be explained with respect to preference styles. Currently, it's too generic and not well-grounded in the experimental results.", "was_revised": true, "final_justification": "Most of my concerns and questions have been addressed by the rebuttal clarifications. I believe the work presents an interesting tweak with useful empirical results. However, I still have reservations about the impact of this work. So, I decided to maintain my borderline accept rating, but with improved confidence.", "rebuttal": "Dear reviewer 7F41:\n\nWe sincerely thank you for the time and effort dedicated to reviewing our work. Your comments are highly appreciated and help us further improve the quality and clarity of our paper.\n\nWe address each of your points in detail below:\n\n\n\n**Response to W1 (“human may still have corrections or different preferences”):**\nWe agree with the reviewer’s observation. Indeed, the fact that “a trajectory is endorsed by the human” does not imply that the human will always make identical decisions in the future. We have taken this uncertainty into account when designing our benchmark. Specifically:\n\n1.\tTo encourage stochasticity and avoid overfitting to specific behaviors, we generate and evaluate offline trajectories by sampling from parameterized agents, rather than using deterministic policies.\n\n2.\tAs stated in Lines 267–269 of the paper: “we retain the trajectories whose biased return reaches at least 80% of the maximum return under that preference, and treat them as liked trajectories. The corresponding human-side policies in these trajectories serve as test-time collaborators for evaluating the cooperative agent.” Each human preference is associated with a well-defined reward function, consistent with the assumption made in ZSC-eval [1]. Multiple human proxy agents may be used to evaluate under the same preference, further capturing the diversity and variability of human decision-making in real-world settings.\n\n**Response to W2 (Concern about computational cost):**\nWe appreciate the reviewer’s concern regarding computational efficiency. In our experiments, we set the rollout size to 200 and the trajectory length to 400, with a total environment step budget of 20 million. This implies that EAR performs at most 250 rollouts throughout training.\n\nAll experiments were conducted on a single NVIDIA RTX 2080 Ti GPU. Under the random1 layout, style A, the average training time per step is as follows:\n\n| Random1 (methods in EAR)| Execution time (s) |\n|-----------|----------|\n| 1-epoch Behavior Cloning (BC)    | 22.3   \n| One round of environment rollouts (rollout size 200)| 27.2\n| PPO (15 epochs) after one rollout | 9.4\n\n| Random1| The number times rollouts was exhausted (max 250)|\n|-----------|----------|\n|Style A|204.3 |\n|Style B|188.3|\n|Style C|205.6|\n\nAs shown in the table above, EAR requires approximately 2 to 4 hours of execution time per style on a single RTX 2080 Ti GPU.\n\nRegarding the question “how would these change for a more diverse policy pool”, we sincerely thank the reviewer for raising this important point. Exploring the scalability of our method under a more diverse policy pool is a valuable direction, and we consider it an important aspect for future work.\n\n**Response to W3 (Unclear description in Section 5.3):**\nWe apologize for the unclear description in Section 5.3. Our intent was to first present the overall performance of our method, and then to provide a deeper understanding of the learned preferences by visualizing event-count vectors (one per trajectory) using UMAP. The resulting clusters reflect the impact of preference learning on the cooperative agent’s behavior.\nWe appreciate the reviewer’s feedback, and we will improve this section by including more fine-grained, preference-specific analyses in the camera-ready version.\n\n**Response to Q1:**\nThe pretrained policies available to the agent—i.e., the policies in the policy pool—are not subject to any explicit conditioning or constraints. In our experimental setup, these policies are trained via self-play under the standard environment reward function, meaning their behavior is solely driven by reward maximization. This design choice ensures that no information about human preferences is leaked into the policy pool, maintaining a clear separation between environment reward and human preference.\nRegarding “how much should policies be reflective of / adhere to the preferences in the offline data,” we determine whether a policy aligns with a given preference by computing the preference score of the trajectories it generates. This allows us to assess how well a given policy adheres to a particular reward function without requiring explicit conditioning.\n\n**Response to Q2:**\nBy lightweight behavior cloning, we refer to the fact that in our method, the behavior cloning (BC) phase requires significantly less time and computational resources compared to the online recovery stage.\n\nIf you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.\n\n**Response to Paper Formatting Concerns:**\nThank you for pointing out the formatting concerns. We will carefully review all figures and thoroughly check the entire paper to ensure that all formatting issues are resolved in the camera-ready version.\n\nIf you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.\n\n\n[1] ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination (NeurIPS 2024)"}, {"review_id": "zHJSlrJ2iv", "reviewer": "Reviewer_5sd7", "summary": "This work proposes a method for integrating offline human data into RL agent training. They first pretrain using RL on a pool of existing policies before iterating a loop of single-epoch BC and online RL recovery (with existing policies). Their ablations demonstrate the importance of their method on balancing alignment with environment rewards.", "questions": "- In section 3.2, it seems like only \"liked\" trajectories are favored, but there is still critical information one can find from \"unliked\" trajectories. Why weren't negative trajectories included in your dataset and method?\n- Why was BC chosen as the key method of alignment instead of inverse RL?\n- Why are the bounds in algorithm 1 based on the original average returns and accuracy? It feels a bit arbitrary relative to a more formal function that takes average returns and alignment into account. For instance, I would consider KL-constrained RL to be a more formal way of combining RL with existing demonstrations.\n- In lines 281-282, you say that a sub-optimal checkpoint is chosen to control for implicit preference alignment. How exactly is this checkpoint chosen then?", "limitations": "Yes (appendix E)", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The approach proposed in this paper is quite general and easy to implement.\n- To my understanding, the proposed pipeline is novel.\n- The experimental results support the claim that their technique balances environment reward with co-agent preferences.\n\nWeaknesses:\n- This work does not mention the recent body of work on generating diverse policies/conventions using cross-play minimization (LIPO/ADVERSITY/BRDiv/CoMeDi/L-BRDiv + others), which have been very successful at creating high-quality and diverse populations with no human data while still having strong ad-hoc teamwork with real humans because they try to be as robust as possible against the broadest set of policies. I expect comparisons against a representative subset of these methods in your experiments.\n  - Additionally, it does not compare against existing RLHF work, like KL-constrained PPO (Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog), which operates on a very similar problem setting. It also does not compare against $PPO_{BC}$ from the original Overcooked work, which is also applicable to this problem setting.\n- This work only conducts evaluations against human proxies, which are trained with RL from biased reward functions, instead of conducting a real human user study. This makes it hard to determine how well the proposed approach would work with real humans.\n\n\nOther notes:\n- The definition of ZSC used in this work is more aligned to the concept of ad-hoc teamwork (AHT) than ZSC. In particular, the original ZSC work (Hu et al., 2020) defines the goal of ZSC as finding a learning rule such that independent instances have high cross-play instead of defining based on an unknown set of human policies (which falls more in line with AHT).\n- I think some of the equations have typos. In particular, I think there's an equal sign missing in equation 3 (before the second max), and there is a second max missing in equation 1.\n- The text in figure 2 and 3 is too small relative to the text of the body, making it hard to read.\n- In section 4.1, it is unclear what is meant by \"pretraining\" (Best reponse RL against the policy pool? BC on the policy pool? Some combination of both?) and where the policy pool comes from to begin with. I think it is important to explicitly say RL (and the specific algorithm) when training is mentioned, since pretraining in the context of LLMs typically refers to self-supervised learning.\n- In table 3, I think all the results need to be multiplied by 100 (because you mean 73%, not 0.73%)", "was_revised": true, "final_justification": "This work seems to stand in the middle ground between two popular lines of MARL work. On one side, we have zero-shot/ad-hoc coordination, where we assume no access to human data but instead must generalize to new partners. On another side, we have a large number of human demonstrations, which we can start from to do behavior cloning or do KL-constrained RL. This work instead imagines a setting where we have a limited amount of convention data, and instead wants to work with that specific convention at test time, which is more akin to few-shot coordination.\n\nIn this framing, the authors demonstrate the importance of few-shot coordination in the rebuttal by comparing against existing popular ZSC works, which their method outperforms. They do not explicitly compare against works that typically require large amounts of human demonstrations, which I feel would've neatly tied up loose ends for this work (though I would expect those methods to fall behind their approach). Overall, this work focuses on an understudied direction of MARL and provides a simple, yet effective method to use prior human data.", "rebuttal": "Dear reviewer 5sd7:\n\nWe sincerely thank you for the time and effort dedicated to reviewing our work. Your comments are highly appreciated and help us further improve the quality and clarity of our paper.\n\nWe address each of your points in detail below:\n\n**Response to W1 (“work on generating diverse policies/conventions”):**\nWe thank the reviewer for raising this important point. We will include a discussion of relevant methods on generating diverse policies and conventions in the related work section of the camera-ready version. Regarding the recent body of work on generating diversity through cross-play minimization, we selected two representative and validated methods from ZSC-eval [1]: TrajeDi [2] and MEP [3]. These methods have been shown to be effective in the Overcooked environment. For evaluation, we directly used the released checkpoints from ZSC-eval. Notably, these checkpoints were trained with a policy pool size of 36 (agents with standard preferences), which is larger than ours (18) and thus has stronger generalization potential. The experimental results on layout random1 are as follows:\n\n| Random1 | Style A (Pref. Score, Env. Return)| Style B (Pref. Score, Env. Return) |  Style C (Pref. Score, Env. Return) |\n|-----------|----------|----------|----------|\n| EAR      | **(274.7 ,  88.6)**    |  **(200.3 , 103.6)**  | **(259.0, 110.3)**\n| TrajDi | (100.0 , 32.5)     | (115.0 , 48.8)     | (190.0, 73.2)\n| Mep | (105 .0, 29.8)     | (112.0 , 45.3)     | (224.0, 82.4)\n\nThese comparisons further support the necessity of explicit preference learning, as our method consistently performs well even when evaluated against strong ZSC baselines.\n\n**Further Response to W1 (on KL-constrained PPO and PPObc)**:\nIn our setting, the number of available offline trajectories is extremely limited (around a dozen), making it difficult to estimate KL-constrained rewards reliably. Many states encountered during training are out-of-distribution (i.e., not present in the offline data), especially at the early stage. This leads to large estimation errors in KL-constrained methods due to the mismatch in dynamics between the offline trajectories and the policy pool policies. Our method avoids this issue by decoupling preference learning and generalization: the online recovery stage in EAR focuses solely on regaining generalization ability, without relying on preference alignment.\n\nRegarding PPObc, the core idea is to first fit a human proxy, and then train the cooperative agent via reinforcement learning to coordinate with this proxy. While this may work in simple or two-player settings, modeling the proxy accurately becomes significantly more challenging in multi-agent or more complex environments. In contrast, EAR does not rely on accurate modeling of the human proxy, and is able to simultaneously generalize and align with human preferences even when the dynamics in offline data and online recovery differ.\n\n**Response to W2 (“This work only conducts evaluations against human proxies”)**:\nWe acknowledge this limitation. Due to budget and time constraints, we have not yet conducted experiments involving real human participants. We appreciate the reviewer’s feedback, and we consider this an important direction for future work. \n\n\n**Response to Notation Error (1)**:\nThank you for pointing out the distinction between AHT and ZSC. We will include a discussion of their relationship in the camera-ready version. Our current use of ZSC terminology follows the convention adopted in prior works such as FCP, MEP, HSP, and ZSC-eval. That said, we agree that the definition may require further clarification, and we will revisit it carefully in our revision to ensure conceptual precision.\n\n\n**Response to Notation Errors (2), (3), and (5):**\nWe appreciate the reviewer for identifying these issues. These are either minor typos or instances of unclear writing, and we will conduct a thorough review of the entire paper to eliminate such avoidable errors and improve the overall clarity.\n\n\n**Response to Notation Error (4):**\nThank you for pointing out the imprecise use of the term “pretraining” in our paper. In our setting, pretraining refers to the process where the cooperative agent is trained via RL with policies from the policy pool to improve its generalization ability, i.e., its capacity to collaborate with a variety of agents to complete tasks in the environment. We acknowledge that this use of pretraining may differ from its usage in NLP, where pretraining typically involves learning from large-scale corpora to capture syntax and semantic knowledge. However, we view the intent as analogous—just as NLP models gain basic language competence through pretraining, our cooperative agent is meant to acquire foundational understanding of task structure and partner dynamics in Overcooked via this pretraining process. We will make this distinction clearer in the camera-ready version to avoid potential confusion.\n\n**Response to Q1:**\nThank you for the question. We clarify that we did not use any poor-quality or noisy data in our experiments. However, we acknowledge that our current method does not leverage unlabelled or low-quality preference data, which we regard as a limitation and explicitly discuss in Appendix E. That said, a key contribution of our work is that EAR achieves a strong balance between generalization and preference alignment using only a small number of positive-labeled trajectories. We view this label-efficient approach as a meaningful step toward more practical preference learning in multi-agent settings.\n\n\n**Response to Q2 and Q3:**\nIn our current setup, we do not attempt to model a preference-based reward function. The primary reason is the extremely limited amount of offline preference data—only around 10-20 trajectories. Under such sparse data, reward learning methods (e.g., inverse reinforcement learning) are prone to extrapolation errors due to a large number of out-of-distribution states, which leads to unreliable or unstable reward estimation.\n\nSimilarly, using KL-constrained RL to learn from these offline trajectories faces challenges: the dynamics of the offline trajectories differ significantly from those induced by the policy pool, resulting in large initial KL divergence and unreliable gradients, especially early in training. This can cause performance degradation due to compounding errors. In contrast, our method avoids this issue by separating preference alignment and generalization: EAR’s online recovery phase is used solely to restore generalization ability, while preference information is distilled only from the limited set of labeled trajectories.\n\n\n\n**Response to Q4:**\nIn the main paper, “checkpoint” refers to the best-performing checkpoint of FCP. The term sub-optimal is used in two senses:\n(1) Under the FCP setup, the biased human agent used for supervision is no longer available, and its behavior closely resembles the offline preference trajectories. This suggests that there is limited room for improvement beyond the offline data.\n(2) More broadly, in the absence of the human supervisor, other methods from the ZSC literature such as TrajeDi or MEP may achieve better coordination than FCP when evaluated under standard ZSC metrics. However, when evaluated under our preference alignment objective, these methods do not significantly outperform FCP, suggesting that coordination alone is insufficient for preference learning.\n\nIf you have any additional questions or concerns, we would be happy to provide further clarification and are glad to have the opportunity to elaborate more on our work.\n\n\n[1] ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination (NeurIPS 2024)\n\n[2] Trajectory Diversity for Zero-Shot Coordination. (PMLR 2021)\n\n[3] Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination (NeurIPS Cooperative AI Workshop 2021)"}]

Author rebuttal:

Meta-review: This paper proposes a new framework for cooperative AI that uses a small amount of preference data while adapting online. 
The work is timely and novel, and experiments show that the method is simple yet effective. In particular, the evaluation in the overcooked environment demonstrates clear improvements in both preference scores and environment rewards, which was highlighted as a major strength.

There were concerns about unclear presentation, limited baselines, and the reliance on proxy agents rather than real humans. 
The authors addressed these points with additional experiments and revisions, and showed that the proposed method outperforms existing methods despite its simplicity.

Overall, the paper combines novelty and empirical support, and points to an important direction for cooperative AI research.
Decision: Accept (poster)