Title: Uncertainty-aware Preference Alignment for Diffusion Policies
Venue: NeurIPS 2025
Accepted: yes
Keywords: Preference-based Reinforcement Learning

Diffusion Model

Uncertainty Awareness
Abstract: Recent advancements in diffusion policies have demonstrated promising performance in decision-making tasks. To align these policies with human preferences, a common approach is incorporating Preference-based Reinforcement Learning (PbRL) into policy tuning. However, since preference data is practically collected from populations with different backgrounds, a key challenge lies in handling the inherent uncertainties in people's preferences during policy updates. To address this challenge, we propose the Diff-UAPA algorithm, designed for uncertainty-aware preference alignment in diffusion policies. Specifically, Diff-UAPA introduces a novel iterative preference alignment framework in which the diffusion policy adapts incrementally to preferences from different user groups. To accommodate this online learning paradigm, Diff-UAPA employs a maximum posterior objective, which aligns the diffusion policy with regret-based preferences under the guidance of an informative Beta prior. This approach enables direct optimization of the diffusion policy without specifying any reward functions, while effectively mitigating the influence of inconsistent preferences across different user groups. We conduct extensive experiments across both simulated and real-world robotics tasks, and diverse human preference configurations, demonstrating the robustness and reliability of Diff-UAPA in achieving effective preference alignment. The project page is available at https://github.com/mr20010112/Diff_UAPA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of sample efficiency and robustness in preference-based reinforcement learning (PbRL), where the learning agent relies on human preference feedback instead of explicit numerical rewards. The authors propose an uncertainty-aware PbRL framework that explicitly models the uncertainty in human feedback and integrates it into both the preference prediction and policy optimization processes. Specifically, the method employs a Bayesian preference model to capture epistemic uncertainty and leverages this information to guide more informative queries and safer policy updates. The paper demonstrates the proposed approach on several benchmark continuous control tasks, showing that the method achieves improved sample efficiency and robustness compared to existing PbRL baselines. The key contributions include the integration of uncertainty estimation into the preference model, the design of an active query strategy based on uncertainty, and an uncertainty-aware policy optimization algorithm.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- This paper accurately identifies a crucial but often overlooked issue in PbRL: the uncertainty of preference data. This is an important topic in current alignment research. The authors propose using a Beta distribution as a prior for trajectory advantage values, combined with regret-based preference modeling and MAP inference, enabling the model to capture preference uncertainty explicitly.\n\n- The paper introduces Diff-UAPA, which is designed for both streaming and batch preference inputs, demonstrating broad applicability.\n\n- The authors tightly integrate the preference alignment problem into the diffusion policy framework, proposing chain reward and advantage modeling based on diffusion trajectories, which addresses limitations faced by DPO/CPL methods in this setting.\n\n- The experimental validation covers diverse scenarios, including synthetic tasks, real-world preference datasets, and real robot control tasks. Moreover, the authors conduct sensitivity analyses under various noise levels and types, along with ablation studies, to demonstrate the method’s advantages in dynamic preference iteration and uncertain data conditions.\n\n------\n\n**Weaknesses:**\n\n- Although the Beta distribution can theoretically represent uncertainty in the strength of preferences, its effectiveness under complex preference distributions (such as asymmetric or multi-modal cases) remains questionable. Additionally, the paper initializes the Beta parameters $\\alpha$ and $\\beta$ to 1. How would varying the initialization affect the model’s performance under different scenarios (e.g., the various noise conditions shown in Table 5)?\n\n- Equation 8 and Equation 17 introduce additional computational overhead. How does this overhead compare to previous methods? It would be helpful if the authors could provide a more detailed comparison of the computational cost relative to prior approaches.\n\n- The comparative experiments only benchmark against diffusion-based methods. Why not compare with non-diffusion baselines (such as INPO, IPO, or Nash-MD-PG)?\n\n**Others:**\n\n- Line 106: “datatset” → “dataset”\n\nQUESTIONS:\n1. In Equation 11, multiple layers of approximation are introduced. Could the accumulation of these approximations affect the final performance?\n2. Proposition 4.1 assumes that trajectories are i.i.d, but in practice, trajectories may be dependent. How can you ensure that this is maintained in MDP?\n3. In Algorithm 1, how is the reference policy $\\pi_{ref}$ initialized? How sensitive is the final performance to the choice of $\\pi_{ref}$?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Diff-UAPA, an uncertainty-aware framework for aligning diffusion policies with human preferences using a Beta prior to model preference inconsistencies. The approach integrates a maximum posterior objective with iterative preference alignment, demonstrating robust performance across simulated and real-world robotics tasks. Extensive experiments validate its effectiveness against baselines, including ablation studies and noise sensitivity tests. However, a critical question arises regarding the originality of the core contributions, as key components appear to build heavily on prior works.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is well-written and clearly organized, with a logical flow from problem statement to experimental validation.\n\n- Important Motivation: The focus on handling uncertainty in preference alignment is timely and relevant, as human preferences from diverse groups inherently contain inconsistencies. The work addresses a critical gap in existing PbRL methods that overlook such uncertainties.\n\n- The empirical evaluation is robust, covering multiple robotic tasks, real human preference datasets, and various noise conditions. The comparison with state-of-the-art baselines (e.g., BET, Diff-CPL, FKPD) and ablation studies (e.g., Beta prior ablation) effectively demonstrate the method’s superiority and validate the contribution of each component.\n\n**Weakness**:\n\n- The paper’s novelty is unclear, as the two key components seem to rely heavily on prior works:\nThe diffusion policy alignment framework in Section 4.1 builds directly on Wallace et al. (2024), which introduces Direct Preference Optimization (DPO) for text-to-image diffusion models.\nThe Beta prior model in Section 4.2 is inspired by Xu et al. (2025), which uses a Beta distribution for uncertainty-aware preference alignment in offline settings.\nWhile the combination of these components is novel, the paper fails to sufficiently demonstrate its unique innovations tailored to the problem at hand, weaken its claim to breakthrough innovation.\n\n- The theoretical contribution is primarily a reapplication of Beta priors (a well-known conjugate prior for Bernoulli distributions) to the diffusion policy context, without substantial new theoretical insights. The proof in Proposition 4.1, while correct, formalizes a standard Bayesian modeling approach rather than introducing a novel framework.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Diff-UAPA, an uncertainty-aware preference alignment algorithm for diffusion policies. The algorithm employs a maximum posterior objective with an informative Beta prior to align policies with regret-based preferences, enabling direct optimization without explicit reward functions while mitigating inconsistent preferences across groups. Extensive experiments on simulated and real-world robotics tasks demonstrate Diff-UAPA's effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method is reasonable, and the paper is clearly written.\n2. Diff-UAPA shows demonstrable effectiveness in simulated and real-world robotics tasks.\n\nWeaknesses:\n\n1.  This method builds upon several existing works [1,2,3], which may limit the novelty of the proposed approach. In my understanding, compared to [3], this paper primarily replaces the reward distribution with an advantage distribution and represents it using a Beta distribution (where the advantage derivation process is similar to [1]), while the optimization objective derivation for Diffusion Policy is largely based on [2]. The paper would benefit from clarifying the distinctions from these prior works to better highlight its contributions.\n\n2. The experimental evaluation mainly employs label flipping in preference datasets, which may not fully replicate the inherent uncertainties characteristic of human preferences.\n3. The necessity of the Beta distribution warrants further justification—whether directly predicting advantage distributions (as in [3]) or employing alternative distributions would be equally effective.  Moreover, the Beta distribution's unimodal nature may limit its ability to capture the diverse human preferences.\n\n[1 ]From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function. COLM 2024.\n\n[2] Diffusion model alignment using direct preference optimization. CVPR 2024.\n\n[3] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR 2025.\n\nQUESTIONS:\n1. In the introduction line 54, the authors mention \n> interpreting preference alignment as a voting process\n\nbut this concept is not elaborated upon or referenced again in the main text.\n\n2. It appears that UA-PbRL[1] and other uncertainty-aware preference alignment algorithms are not compared in the main experimental evaluation.\n\n3. While the paper frequently refers to iterative preference alignment, the model training appears to be offline, which seems contradictory.\n\n[1] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR 2025.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes Diff-UAPA, a direct preference alignment algorithm for diffusion policies, designed to address inherent uncertainties in human preferences across diverse user groups. Diff-UAPA uses a MAP objective, guided by an informative Beta prior for capturing the uncertainty. Experiments on both simulated robotic and real-world tasks demonstrate the robustness of Diff-UAPA for the uncertainty in preference data.

STRENGTHS AND WEAKNESSES:
Strengths.

1.  The proposed method advances beyond traditional MLE objective in PbRL, which implicitly assumes a uniform prior and consequently lacks sensitivity to the inherent uncertainty in preference dataset, by using an informative beta prior and MAP objective.

2. The paper provides comprehensive empirical validation, consistently demonstrating Diff-UAPA's superior performance and robustness across both simulated and real-world robotics tasks, and various human preference configurations including synthesized, realistic, and noisy data.

Weaknesses.

1. Training informative beta prior leads to additional computation costs.

2. It seems that the baseline methods are not iteratively trained to adapt to evolving preference signals in the same manner as Diff-UAPA.

QUESTIONS:
Q1. Can we come up with the proposed method's online counterpart?

Q2. Could the author provide a discussion on the advantages and disadvantages of the proposed method over simple robust methods such as data filtering and label smoothing?

Q3. Could the author consider other iterative method baselines for a fair comparison with Diff-UAPA?

LIMITATIONS:
The authors present a limitation section in Appendix.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03