Title: Learning Preferences without Interaction for Cooperative AI: A Hybrid Offline-Online Approach
Venue: NeurIPS 2025
Accepted: yes
Keywords: Cooperative AI

offline preference learning

online adaptation
Abstract: Reinforcement learning (RL) for collaborative agents capable of cooperating with humans to accomplish tasks has long been a central goal in the RL community. While prior approaches have made progress in adapting collaborative agents to diverse human partners, they often focus solely on optimizing task performance and overlook human preferences—despite the fact that such preferences often diverge from the reward-maximization objective of the environment.
Addressing this discrepancy poses significant challenges: humans typically provide only a small amount of offline, preference-related feedback and are unable to engage in online interactions, resulting in a distributional mismatch between the agent’s online learning process and the offline human data. To tackle this, we formulate the problem as an online&offline reinforcement learning problem that jointly integrates online generalization and offline preference learning, entirely under an offline training regime.
We propose a simple yet effective training framework built upon existing RL algorithms that alternates between offline preference learning and online generalization recovery, ensuring the stability and alignment of both learning objectives.
We evaluate our approach on a benchmark built upon the Overcooked environment—a standard environment  for human-agent collaboration—and demonstrate remarkable performance across diverse preference styles and cooperative scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method for integrating offline human data into RL agent training. They first pretrain using RL on a pool of existing policies before iterating a loop of single-epoch BC and online RL recovery (with existing policies). Their ablations demonstrate the importance of their method on balancing alignment with environment rewards.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The approach proposed in this paper is quite general and easy to implement.\n- To my understanding, the proposed pipeline is novel.\n- The experimental results support the claim that their technique balances environment reward with co-agent preferences.\n\nWeaknesses:\n- This work does not mention the recent body of work on generating diverse policies/conventions using cross-play minimization (LIPO/ADVERSITY/BRDiv/CoMeDi/L-BRDiv + others), which have been very successful at creating high-quality and diverse populations with no human data while still having strong ad-hoc teamwork with real humans because they try to be as robust as possible against the broadest set of policies. I expect comparisons against a representative subset of these methods in your experiments.\n  - Additionally, it does not compare against existing RLHF work, like KL-constrained PPO (Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog), which operates on a very similar problem setting. It also does not compare against $PPO_{BC}$ from the original Overcooked work, which is also applicable to this problem setting.\n- This work only conducts evaluations against human proxies, which are trained with RL from biased reward functions, instead of conducting a real human user study. This makes it hard to determine how well the proposed approach would work with real humans.\n\n\nOther notes:\n- The definition of ZSC used in this work is more aligned to the concept of ad-hoc teamwork (AHT) than ZSC. In particular, the original ZSC work (Hu et al., 2020) defines the goal of ZSC as finding a learning rule such that independent instances have high cross-play instead of defining based on an unknown set of human policies (which falls more in line with AHT).\n- I think some of the equations have typos. In particular, I think there's an equal sign missing in equation 3 (before the second max), and there is a second max missing in equation 1.\n- The text in figure 2 and 3 is too small relative to the text of the body, making it hard to read.\n- In section 4.1, it is unclear what is meant by \"pretraining\" (Best reponse RL against the policy pool? BC on the policy pool? Some combination of both?) and where the policy pool comes from to begin with. I think it is important to explicitly say RL (and the specific algorithm) when training is mentioned, since pretraining in the context of LLMs typically refers to self-supervised learning.\n- In table 3, I think all the results need to be multiplied by 100 (because you mean 73%, not 0.73%)\n\nQUESTIONS:\n- In section 3.2, it seems like only \"liked\" trajectories are favored, but there is still critical information one can find from \"unliked\" trajectories. Why weren't negative trajectories included in your dataset and method?\n- Why was BC chosen as the key method of alignment instead of inverse RL?\n- Why are the bounds in algorithm 1 based on the original average returns and accuracy? It feels a bit arbitrary relative to a more formal function that takes average returns and alignment into account. For instance, I would consider KL-constrained RL to be a more formal way of combining RL with existing demonstrations.\n- In lines 281-282, you say that a sub-optimal checkpoint is chosen to control for implicit preference alignment. How exactly is this checkpoint chosen then?\n\nLIMITATIONS:\nYes (appendix E)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a hybrid RL training framework that alternates between offline preference learning and online generalization recovery for cooperative agents. The cooperative AI problem setup assumes pretrained policies for cooperative agent training, and availability of offline preference data. The paper attempts to simultaneously enhance the zero-shot generalization and preference alignment capabilities of the cooperative agent. The training regime and metrics are validated in the Overcooked environment against a number of relevant baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1) The paper is largely clear and well-structured.\n2) The work proposes an interesting new tweak of epoch-wise alternation recovery, and provides empirical experimental evidence of its success.\n\n### Weaknesses\n1) The post-match \"like\" assumption is informative at the trajectory level, but may not be so at the granular action level. The trajectory is endorsed by the human, but the human may still have corrections or different preferences for some individual actions at specific states. This may affect zero-shot coordination.\n\n2) The training method of epoch-wise alternation recovery proposed involves performing generalization recovery with the policy pool until either the average return is greater than the original value or the budget is exhausted. This can be computationally expensive, considering recovery is generally slow. It would be useful to get information on the budget limit used, the number of times the budget was exhausted, and the average time taken for recovery, etc. In addition, how would these change for a more diverse policy pool, etc.\n\n3) Section 5.3 needs to be explained with respect to preference styles. Currently, it's too generic and not well-grounded in the experimental results.\n\nQUESTIONS:\n1) What are the conditions on the set of pretrained policies available to the agent? How much should policies be reflective of/adhere to the preferences in the offline data?\n2) What exactly does lightweight behavior cloning mean in the training regimen?\n\nLIMITATIONS:\nThe limitations of the training paradigm and dataset collection need to be stated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAt a high level, this work addresses the problem of incorporating human preference data into the training of policies for AI agents that can cooperate with previously unseen partners.  They first pre-train a cooperative policy against a pool of human proxy policies learned using the previously described Fictitious Co-Play framework.  They then allow (simulated) humans to interact with this policy, and \"like\" the joint trajectories that they found to be most successful.  Finally, they alternate between training (under a behavioral cloning loss) on these \"liked\" trajectories, and training against the human proxy pool.  The purpose of this alternating training scheme is to avoid catastrophic overfitting to the small set of human-annotated trajectories.  Experimental results on a subset of the Overcooked zero-shot coordination benchmark show that this method significantly outperforms the FCP baseline against \"stylistically biased\" human partners.  Ablation experiments also show that removing the alternating training setup significantly degrades this performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The proposed EAR framework proved to substantially outperform the FCP baseline, using only a small set of style-specific trajectories.\n2. Possibly the most interesting result from this work is in Table 1, showing that alternating between BC and \"recovery\" training against the FCP pool leads to significantly improved performance over a single round of BC + recovery.\n\n**Weaknesses:**\n1. HSP is used to generate stylistically biased human trajectories, but is not used as a baseline method itself.  It would seem likely that a policy pre-trained against a pool of agents generated through HSP (trained on perturbed reward functions) would outperform FCP in this setting, even without BC finetuning or alternating recovery.\n2. There are a number of other ZSC methods that might been good baselines here, for example TrajeDi (Lupu et al. 2021) or LIPO (Charakorn et al., 2023).  The poor performance of FCP pre-training in these experiments likely has to do with the diversity of the human partners the method is evaluated against.  Compared to FCP, both TrajeDi and LIPO (along with several other baselines) have mechanisms to encourage diversity within the pool of target agents, and might be expected to perform well even without fine-tuning on human preferences.\n\nQUESTIONS:\n1. For the HSP-based policies used to generate the trajectories for behavioral cloning, to what extent were the learned policies \"sub-optimal\" with respect to the base Overcooked reward function?  Ideally, methods like FCP would cover the space of near-optimal joint strategies, so perhaps relatively large reward pertubations could explain its poor performance with the \"biased\" partners?\n2. How well did the joint strategies trained with FCP, and the pre-trained policy trained against them, perform under the base Overcooked reward (without perturbations)?\n3. Was the architecture of the pre-trained policy recurrent (allowing for adaptation)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the conflict between generalizing to unseen partners and aligning with limited offline preference data in cooperative reinforcement learning. It introduces a training strategy named Epoch-wise Alternation Recovery, which alternates between one epoch of lightweight behavior cloning and generalization recovery. This strategy enables cooperative agents to maintain strong generalization capabilities while effectively incorporating human preferences. Experiments in the Overcooked environment show that the new strategy achieves both high preference scores and high environment returns, demonstrating that it effectively balances the two objectives.\n\nSTRENGTHS AND WEAKNESSES:\nPros.\n\nTackling distribution shift in human‑cooperative reinforcement learning is a timely and underexplored problem. This fresh solution is poised to spark follow‑up studies on generalization and preference alignment under non‑stationary conditions.\nI appreciate that the paper’s core claims rest on a solid foundation: the methods are both principled and well matched to the problem, and the extensive suite of experiments convincingly demonstrates their effectiveness. 3. The three‑stage training pipeline is logically coherent and neatly justified. The paper articulates the rationale for this integration clearly, and the differentiation from prior work is precise, with all key citations in place.\nThe paper is well written and supplies enough implementation detail to reproduce results.\nCons.\n\nThe promise to “describe each stage in detail below” is undermined when subsection 4.2 veers into design rationale instead of covering the second stage. This break in flow makes it harder to follow the pipeline in the order the authors intended.\nThere’s confusion around Table 2: the main text calls out “benchmark proxy agents not involved in the offline preference data,” yet the caption refers to “all benchmark human proxy agents.” Aligning these descriptions is crucial for readers to understand exactly what’s being evaluated.\nAlthough the experiments are thorough, the baseline set skips over promising alternatives—such as methods leveraging successor features or contrastive learning for joint generalization and preference alignment. Including those would strengthen the comparative evaluation and better position the contribution relative to state‑of‑the‑art approaches.\n\nQUESTIONS:\nThe experimental comparison would be more convincing if it included at least one method from section 2.3 that jointly addresses generalisation and preference alignment.\nThe exposition in Section 4 breaks the announced flow: subsection 4.2 inserts design rationale before stage 2 is introduced. Re-ordering the material so that Stages 1–3 are described sequentially, followed by the design discussion, would improve readability and align with the section’s opening statement.\nSeveral table annotations need clarification. Please specify what the “Offline Trajs” row in Table 1 represents, and ensure the description of Table 2 is consistent between the caption and the main text. Resolving these discrepancies would remove residual ambiguity in the presentation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-14"}