Title: Learning from Preferences and Mixed Demonstrations in General Settings
Venue: NeurIPS 2025
Accepted: no
Keywords: reinforcement learning

rl

human feedback

rlhf

modelling

preferences

demonstrations

rankings

machine learning

reward learning
Abstract: Reinforcement learning is a general method for learning in sequential settings, but it can often be difficult to specify a good reward function when the task is complex.
In these cases, preference feedback or expert demonstrations can be used instead.
However, existing approaches utilising both together are either ad-hoc or rely on domain-specific properties.
Building upon previous work, we develop a mathematical framework for learning from human data and based on this we introduce LEOPARD: Learning Estimated Objectives from Preferences And Ranked Demonstrations.
LEOPARD can simultaneously learn from a broad range of data, including negative/failed demonstrations, to effectively learn reward functions in general domains.
It does this by modelling the human feedback as reward-rational partial orderings over available trajectories.
We find that when a limited amount of preference and demonstration feedback is available, LEOPARD outperforms baselines by a significant margin.
Furthermore, we use LEOPARD to investigate learning from many types of feedback compared to just a single one, and find that a combination of feedback types is often beneficial.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a unifying framework called Reward-Rational Partial Orderings (RRPO), which models diverse feedback as Boltzmann-rational orderings over trajectory fragments. Building on RRPO, they propose LEOPARD, which jointly optimizes a reward model and agent policy by iteratively incorporating all available feedback types. In evaluations, LEOPARD consistently outperforms standard baselines that use demonstrations, then preferences, and other combined feedback methods, especially when feedback is scarce or mixed. Moreover, experiments show that leveraging a combination of feedback types—even limited negative demonstrations alongside preferences—yields more robust reward learning than using any single feedback source.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- **Unified Feedback Integration:** Seamlessly combines demonstrations, preferences, and rankings into a single reward-learning framework.\n- **Robustness under Scarcity:** Demonstrates superior performance in low-data regimes and when feedback is heterogeneous.\n\n### Weaknesses\n- **Simple Evaluation Domains:** Tasks are toy-scale with well-known ground-truth rewards; it is unclear whether LEOPARD scales to complex environments that require intricate reward engineering.\n- **Lack of Ablation on Feedback Types:** No quantitative analysis of each feedback modality’s efficiency or relative contribution to the learned reward.\n- **Fragment Independence Assumption:** Assumes trajectory fragments are independent, which may break down in long-horizon tasks (e.g., AntMaze) where small fragments fail to capture causality and long-term dependencies.\n- **Strict Ordering Assumption:** Imposes a rigid ordering among negative demonstrations, agent-generated fragments, and positive demonstrations; since agent behavior quality evolves, injecting mixed orderings could introduce noise into the reward learning process.\n\nQUESTIONS:\n- **Multi-Objective Trade-offs:** As tasks grow more complex, how does LEOPARD’s single loss function handle the inherent trade-offs of optimizing over multiple feedback types simultaneously?\n- **Feedback Budget Selection:** What guided the choice of $n_{\\mathrm{demos}}$ and $n_{\\mathrm{prefs}}$ in your experiments? Are these settings transferable across domains?\n- **Guaranteeing Balanced Mixing:** In Section 4.3, optimal data-mixing strategies differ by domain. How can one ensure that all feedback types are leveraged appropriately without manual tuning?\n- **Baseline Comparison:** How does LEOPARD perform relative to the limited-preference imitation learning [1]?\n\n**References**\\\n[1] Xingchen Cao et al., Limited Preference Aided Imitation Learning from Imperfect Demonstrations, ICML 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to learn from a combination of preferences and mixed demonstrations (positive and negative) by modeling human feedback using theory of reward rational choice ordering. The paper begins with going over modeling human feedback using Boltzmann Rational Choices and using this to learn a reward model. Then the paper extends this to guide policy search by using several forms of feedback such as preferences and demonstrations (positive and negative). The paper follows this up with benchmarking the proposed method and baselines (imitation learning with preferences, and deep IRL then RLHF) on four environments from gymnasium.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Very interesting and well motivated problem, has ample scope to derive new algorithms\n\nWeaknesses:\n- The methodology section appears to be fairly thin in terms of details - in particular, the writing when describing the proposed framework just appears to be described using a few lines of math and not much explanations (see lines 200-210).\n- Limited analysis on when the proposed framework works, and what are the failure modes\n- The experimental simulations with gymnasium shows promise in the realms of continuous control but again this is rather limited when it comes to judging the framework's promise.\n\nQUESTIONS:\n1. Could the authors expand upon their framework to study how the proposed framework converges with transitive, intransitive preferences? Does it recover Nash when we have intransitive preferences? \n2. How does the framework scale up with simple LLM experiments even for a single task such as helpfulness vs harmlessness?\n3. What happens if we employ a simple RLHF adaptation where we use pairwise RMs for preferences and pointwise for positive and negative demonstrations and utilize a combination of these reward models for policy search?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LEOPARD, a reward-learning framework designed for mixed human feedback datasets, which include comparisons as well as positive and negative demonstrations. Motivated by reward-rational choice (RRC), the paper introduces the RRPO loss, which maximizes the log-likelihood of selecting a given trajectory over all trajectories ranked lower in the provided partial orderings. Empirical evaluations demonstrate that LEOPARD outperforms two baseline methods, DeepIRL + RLHF and AILP, across various environments. The results also confirm that incorporating multiple types of feedback data leads to better performance compared to relying on a single type.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The proposed framework is easy to implement, scalable, and generalizable across diverse environments and tasks.\n- The proposed method is generally well-motivated and clearly presented.\n\n**Weaknesses:**\n- The baseline methods used for comparison in experiments are not explicitly designed to address multiple feedback data types. It would have been more interesting to see the comparisons with algorithms specifically developed for this setting.\n- The provided theoretical analysis covers only representation properties and does not address generalization or statistical guarantees.\n\nQUESTIONS:\n- In Section 4.3, it is common for reward-learning algorithms to achieve better performance by combining multiple types of feedback data, as doing so allows reward functions to be learned across a broader state-action space, thereby mitigating reward ambiguity or degeneracy. Therefore, I believe this result is not specifically an advantage of LEOPARD but rather a general phenomenon. Could the authors comment on this?\n- It is unclear why using the RRPO loss would be better than simply aggregating the Bradley-Terry loss across all partial orderings. Could the authors provide more intuition or explanation regarding this?\n- In Figure 2, the performance of the two baseline methods seems too poor in some cases, showing little improvement from the arly training stages. Could the authors clarify why this occurs in the baselines?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of learning policies from a combination of both preference-based feedback and task demonstrations.  It presents LEOPARD, an approach that integrates demonstrations and preferences by converting both into a partial preference ordering over trajectory fragments, and then using a loss inspired by the RRiC framework to learn a reward function maximizing the likelihood of this partial ordering.  There seem to be two key ideas behind this work:\n1. Interpreting demonstrations as \"implicit\" preferences for the demonstrated trajectories over a collection of alternative trajectories (either AI generated or provided by yhe user).\n2. In contrast to existing RLHF methods, the training loss employed by LEOPARD does not assume that individual preference orderings are sampled independently, but instead considers the joint probability of the entire set of preferences.\n\nExperiments in a set of standard RL benchmark tasks (including the MuJoCo Ant and Half-Cheetah) environments, demonstrate that LEOPARD significantly outperforms two baseline approaches to combining preferences and feedback.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The proposed training loss is novel, and may be able to capture information conveyed through the choice of a specific *set* of preferences (or demonstrations) that a loss based on the Bradley-Terry model might ignore.\n2. The experimental results show that LEOPARD outperforms both baseline methods by a wide margin, at least in the two MuJoCo tasks evaluated.\n\n**Weaknesses:**\n1. While this work builds on the RRiC model, it seesm the loss used in LEOPARD does not incorporate one of the key insights of RRiC, namely, the idea that demonstrations (and preferences feedback) are selected by the human teacher with the aim of maximizing the expected reward under the distribution over AI trajectories that this demonstration induces.  The RRPO loss (4) still assumes that choices are made based on the advantage of the preferred trajectory over all other trajectories in the choice set.\n2. While RRPO combines preferences and demonstrations, it is not clear whether it fully incoprorates demonstration data.  The reward model may focus on specific components of the trajectory, and ignore differences between the final policy learned and the action choices observed in the demonstrations themselves.  As demonstrations can often communicate far more information that preferences, this would seem to put the method at a disadvantage to direct imitation learning methods.\n3. The experimental evaluation could have been more extensive, for example, evaluating across more (and more challenging) tasks, and comparing against additional baselines (in particular behavioral cloning from the positive demonstrations alone, and DPO for preferences alone).\n\nQUESTIONS:\n1. How would LEOPARD be expected to compare against a baseline that translates demonstrations to preferences in the same way, but uses a standard RLHF loss (which treats each preference as being independent)?  That is, how important is the novel loss function (Equation 4) itself?\n2. Was there a specific reason why behavioral cloning on the demonstration trajectories was not used as a baseline, with AILP and RLHF being the only baselines?\n3. In Figures 2 and 3, what does one \"iteration\" correspond to?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-07"}