Uncertainty-aware Preference Alignment for Diffusion Policies
Authors: Runqing Miao
Sheng Xu
Runyi Zhao
Wai Kin Victor Chan
Guiliang Liu
Venue: NeurIPS 2025 poster ()
Keywords: Preference-based Reinforcement Learning
Diffusion Model
Uncertainty Awareness
Primary area: reinforcement_learning
TL;DR: We propose Diff-UAPA, a novel framework that aligns diffusion policies with human preferences by integrating uncertainty-aware objectives and MAP estimation.

Abstract: Recent advancements in diffusion policies have demonstrated promising performance in decision-making tasks. To align these policies with human preferences, a common approach is incorporating Preference-based Reinforcement Learning (PbRL) into policy tuning. However, since preference data is practically collected from populations with different backgrounds, a key challenge lies in handling the inherent uncertainties in people's preferences during policy updates. To address this challenge, we propose the Diff-UAPA algorithm, designed for uncertainty-aware preference alignment in diffusion policies. Specifically, Diff-UAPA introduces a novel iterative preference alignment framework in which the diffusion policy adapts incrementally to preferences from different user groups. To accommodate this online learning paradigm, Diff-UAPA employs a maximum posterior objective, which aligns the diffusion policy with regret-based preferences under the guidance of an informative Beta prior. This approach enables direct optimization of the diffusion policy without specifying any reward functions, while effectively mitigating the influence of inconsistent preferences across different user groups. We conduct extensive experiments across both simulated and real-world robotics tasks, and diverse human preference configurations, demonstrating the robustness and reliability of Diff-UAPA in achieving effective preference alignment. The project page is available at https://github.com/mr20010112/Diff_UAPA.

-ncertainty-aware Preference Alignment for Diffusion Policies
Runqing Miao 12*, Sheng Xu Runyi Zhao', Wai Kin Victor Chan', Guiliang Liu" Schooo of Data Science, The Chinese University of Hong Kong, Shenzhen, Tsinghua Shenzhen International Graduate School, Tsinghua University mrq23mmails.ttiine edu. en, [shengxu1, runyizhao) (aaiiiaaaa@liiii @1 ink. cuck edu. cn chanw@sz. tsinghua. edu. cn, liuguiliang@cunh edu. cn
Abstract
Recent advancements in diffusion policies have demonstrated promising perfor- mance decision-making tasks. To align these policies with human preferences, common approach is incorporating Preference-based Reinforcement Learning (PbRL) into policy tuning. However, since preference data is practically collected from populations with different backgrounds, key challenge lies in handling the inherent uncertainties in people's 's preferences during policy updates. To address this challenge, we propose the Diff-UAPA algorithm, designed for uncertainty-awwe preference alignment in diffusion policies. Specifically, Diff-UAPA introduces novel iterative preference alignment framework in which the diffusion policy adapts incrementally to preferences from different user groups. To accommodate this online learning paradigm. Diff-UAPA employs a maximum posterior objective, which aligns the diffusion policy with regret-based preferences under the guidance of informative Beta prior. This approach enables direct optimization of the diffusion policy without specifying any reward functions, while effectively miti- gating the influence of inconsistent preferences across different user groups. We conduct extensive experiments across both simulated and real- world robotics tasks, and diverse human preference configurations, demonstrating the robustness and reliability of Diff-UAPA in achieving effective preference alignment. The code is available lttts::/ thub. www.mnrr0001112/111......
1 Introduction
Reinforcement Learning (RL) algorithms commonly employ either deterministic or Gaussian policies to tackle sequential decision tasks by optimizing cumulative rewards (Wang et al. 2022). Al- though these RL policies have demonstrated notable success across wide range of applications (Mnih et al., 2015 Silver et al. 2016 Fang et al.) 2019). they may struggle with learning multi-modal polictes, which may hinder their ability generalize effectively and lead to suboptimal performance in complex environments (Zhu et al. (2023). Recently, diffusion models have gained attention due to their strong modeling capabilities (Ho et al. 2020; Song et al. 2020). As a result, more studies have investigated applying diffusion models in RL tasks, particularky lrly in leveraging diffusion models as policies to model complex action distributions (Wang et al., 2023 Chen et al. 2023a Kang et al. 2023al Lu et al. 2023 Chi al. 2023). To learn a diffusion policy that generates desired outputs. recent approaches have leveraged Preference- based Reinforcement Learning (PbRL) (Christiano) et al. 2017) techniques, which address learning-to-rant problem using preference data, enabling
alignment with human intentions |Wallace et al. 2024 Dong et al. 2024 Shan et al.) 2024).
In practice, preferences are typically gathered from a diverse population, encompassing a wide range of expertise, perspectives, and beliefs. This diversity presents significant challenge, as preferences
*Equal contributions. *Corresponding author: Guiliang Liu, Huguiliang@cuuk.ed..
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Human Comparing
(t',z)) (r' r)
Learning Beta Prior
ft
Beta
Bayesian Alignment
Trajectory Dataset
Preference Dataset
Diffusion Model
12
Ppree
ng (als)
Figure I: The framework of DifI-UAPA. Given the potentially inconsistent preference dataset ranked by diverse humans, we first learn Beta prior to capture uncertainties, and then derive Maximum A Posteriori (MAP) objective to align the diffusion policies.
from different user groups may conflict or evolve over time, introducing great uncertainties during policy updates. To ensure more reliable alignment, this necessitates the development of policy that could account for the uncertainty arising from potentially inconsistent preferences. However, PbRL approaches are typically based on the Bradley-Terry model (Bradley & Terry 1952) with maximum
likelihood estimation, which lacks sensitivity to the inherent uncertainties from preference datasets.
To address the uncertainties in preference alignment, several methods (Liang et al. 2022 Shin et al. 2023 Xue et al. 2024) have employed techniques such as ensemble models and Bayesian dropout. However, the underlying mechanism by which the estimated ensembles correlate with uncertainty remains largely unexplained. Motivated by a recent work (Xu al. 2025). which proposes learning a distributional reward model using Maximum A Posteriori (MAP) objective to address epistemic uncertainty from an offline preference dataset, we explore how to bypass the reward learning and develop an uncertainty-aware algorithm beyond the offline setting for aligning diffusion policies.
In this work, introduce Uncertainty-aware Preference Alignment for Diffusion Policies (Diff- UAPA), a novel algorithm designed to align diffusion policies with human preferences using an uncertainty-awe objective, as illustrated in Figure Specifically, we introduce an iterative prefer- ence alignment framework, in which the diffusion policy progressively adapts to the labels coming from different user groups, each of which may have distinct preferences. To address this challenge, Diff-UAPA involves learning an informative Beta prior that captures the uncertainty arising from diverse human preferences. By interpreting preference alignment that the Beta distribution is sensitive to the uncertainty among trajectories, assigning high confidence to trajectories in which the majority of human raters share a common preference and low confidence to those with divergent preferences. To ensure computational tractability, we parameterize the Beta distribution with neural networks and train the model via variational inference.
voting process, we demonstrate
Building on the learned Beta prior model, we integrate it into the alignment process with regret-based preference model, formulating unified Maximum A Posteriori (MAP) objective. This approach allows for direct optimization of the diffusion policy without the need to learn reward function, while effectively capturing uncertainty in noisy preferences across diverse user groups.
To assess the empirical performance of Diff- UAPA, we conduct extensive experiments on two simulated roboties environments and real-world task, comparing it against recent baseline methods. Furthermore, we investigate its effectiveness using diverse human preference data, including synthesized, realistic, and noisy preferences. The results demonstrate the robustness and reliability of Diff-UAPA in handling the uncertainty in preference data.
Related Works
2.1 Preference-basee Reinforcement Learning
Preference-bart Reinforcement Learning (PbRI.) is a pivotal approach for aligning agents with human intent, particularly in scenarios where specifying explicit reward functions challenging (Wr nell et al. 20188) Wirth et al. (011)). Previous works generally adopt a two-step procedure, where an explicit reward model first inferred from human preferences using the Bradley-Terry model (Bradley
& Terry 1952), followed by training an RL agent to optimize the learned reward (Christiano et al.
2017 barz et al., 2018). Building on this framework, several methods (Lee et al.) [201] Park et al. 2022 Liang et al. 2022. Hwang et al., [023] Choi et al.) 2024) have enhanced the learning process, focusing on improving efficiency and capability. While earlier works assume preferences stem from
summed Markovian rewards, recent studies (Kim et al. 2023] Verma & Metcalf 2024) model them
using Markovvnn rewards. Another line of work bypasses reward modeling by directly optimizing
policies or value functions from human preferences (An et al. 2023
Heina et al.) 2024 Hejna &

---

Sadigh 2024). This approach is more straightforward. avoiding the biases and information bottleneck from intermediate reward modeling (Kang et al., 2023b).
2.2 Diffusion Policy for Decision Making
Diffusion models have surpass earlier generative models in both sample quality and training stability, gaining wide attention across domains, including offline RL (Janner al., [002 Ajay et al.] (023). online RL. (Yang et al., 2023] Chen et al.) 2024). and robotics (Sridhar et al.
2024 Chen et al. 2023b Recent advancements have leveraged diffusion models as RL policies capture arbitrary action distributions, improving decision-making capabilities (Zhu et al., 2023). Among these works, Diffusion-QL (Wang al. | (202)), first integrated diffusion policies into the Q-learning framework. Following this, SIBC (Chen et al. 2023a) refined policy learning by decoupling behavior learning from action evaluation, while CEP (Lu et al. 2023) extended this framework to enable sampling from broader energy-guice -egided distributions. In preference-baser tasks, AlignDiff (Dong et al. 2024) utilized diffusion planners to generate trajectories aligned with human preferences through a two-step procedure, while FKPD (Shan et al., (024) introduced one-step framework for direct alignment.
However, these methods fail to account for the uncertainties inherent in preferences. How to handle these uncertainties when aligning diffusion policies remains a critical challenge (Casper et al. (023).
3 Problem Formulation
Preference- based Reinforcement Learning (PbRL). Reinforcement Learning algorithms (Sutton
& Barto 2018) typically consider an episodic Markov Decision Process (MDP), which is formally detined as a tuple M = (S, A. PR. T,, Ho), where: 1) S and A represent the state and action spaces, 2) Pr (rlss a) and pr(''|s,, a) define the (stochastic) reward and transition functions, 3) y E (0, 1] is the discount factor, 4) fun denotes the initial state distribution and 5) T e (0, X)) denotes non-fixed planning horizon, and the games is reset when the agent reaches a terminating or goal state at a time step T. In many applications, the reward function is not directly available, reducing the episodic MDP to a reward-free MDP MMp. To resolve this challenge, PhRI. algorithms Christiano et al. (017)) proposed learning the reward function from human preferences dataset. Specifically. given an unlabeled dataset of trajectory segments D, {r), humans randomly select a pair of trajectories and rank them according to their preferences. By recording these pair-wise comparisons, we create a preference dataset D Ppre = {(T"", r')), where each trajectory segment of length k is 8k. 0x) ), and T" is preferred over r'. Based on this dataset, recent methods Christiano et al. (2017) Ibarz et al. (2018) commonly infer the rewards by employing the Bradley- Terry model Bradley& & Terry 1952 with maximum likelihood estimation (MLE).
defined as r = [$1, a1. $2, 022
Uncertainty Model in Preference Alignment. The Bradley- -Trryy model (Bradley & Terry 1952 can effectively model pairwise comparisons, whether by explicitly inferring a reward function Chris tiano et al. 2017 Lee et al. 2021 Park et al. 2022) or by directly aligning policies with prefer- ences (Hejna et al. 2024] An et al.,
2023). However, this approach fails to account for the inherent uncertainty in human preferences (Nemman] 2023] Xu et al. 2025), , particularly when these pref- erences are collected from a diverse population with varying levels of expertise, perspectives, and beliefs. Notably, another line of research focuses robust preference learning to address noisy labels, employing techniques such as data filtering, label smoothing, and robust loss functions (Cheng et al. 2024| Mandal et al. 2024 Bukharin et al. 2024). However, these approaches often exclude inconsistent data to facilitate training. which risks discarding valuable information if informative data points mistakenly treated outliers. More critically, for continuous learning, the policy must adapt dynamically to preferences from different user groups, which often arrive incrementally over time. To resolve these challenges, we study iterative preference alignment problem:
Definition 3.1 (Iterative Preference Alignment) Let D, {T} denote the trajectory dataset, and let Ppate = (5) represent the pairwise comparisons dataset constructed at the th iteration. These comparisons are generated by 1) sampling pairs of trajectories from DDT and 2) inviting a group of annotators to label them. The algorithm must progressively align the policy with the preference dataset D at each round n E [1, ] in an online manner.
In this setting, different groups of human annotators may provide inconsistent or even conflicting preferences for the same pair of trajectories (Liang et al. 2022 Shin et al.) 2023 Xue et al. 2024). The problem solver must dynamically adapt the policy to iteratively updated preference signals while ensuring the learned policy effectively represents general preferences by performing online updates.
3

---

In this work, we assume D, records offtine trajectories, since interactions with the environments are not always feasible. The primary challenge is to stabilize the policy optimization process and learn a reliable control policy by effectively managing the aleatoric uncertainty inherent in stochastic and potentially inconsistent preference signals on the provided trajectories.
Preference Alignment for Diffusion Policies. Denoising diffusion models (Ho et al., 2020) represent a class of generative models characterized by iterative diffusion and denoising process, which have gained significant attention in decision-making tasks due to their ability to represent complex distributions (Zhu et al., 2023). This capability is crucial for characterizing the policy function Ha (a]s), surpassing previous deterministic or Gaussian- policies (Wang et al. 2023).
Diffusion policies are typically formulated as conditional generative models as follows
To(as(s)) = - N(at; 0, (u(a; (a 81 s,)da; 1:1
(I)
where Tg( -aa,,=11pp lar", (t) is often parameterized as Gaussian with fixed pnnestep-dependentt covariances as M(a "Iua(a) St, i), )²) Although diffusion policies can be trained from expert demonstrations, their performance is often constrained by the size and quality of such dataset. As result, many previous methods have explored aligning diffusion policies with human feedback (Liu et al. 2024). In this setting, recent research (Wallace et al. (2024) proposed leveraging Direct Preference Optimization (DPO) (Rafailvv et al.] 2023) to align text-to-image diffusion models with human preferences. Specifically, DPO algorithms directly optimize policies without learning a reward model as follows:
T0000000
ss" (w))
TT 0.J (s))
(D) = -E
EE log a (Alog Teer(a
A Alg
Frer (a (t)
where ((s", au.r), (st, as), D pref are state-action samples and nee denotes the reference policy. In this study, we investigate the alignment of diffusion policies with human preferences within the context of reinforcement learning, addressing the challenge of iterative preference alignment.
4 Uncertainty-Awar Preference Alignment for Diffusion Policies
In this section, we outline our approach for aligning diffusion policy with human preferences while effectively accounting for uncertainty. Specifically, we present: 1) a Maximum Likelihood Estimation (MLE) objective for diffusion policy alignment in the context of RL, based on maximum entropy framework and direct preference optimization (Section 4.1). 2) a Maximum A Posteriori (MAP) objective that incorporates Beta prior model for capturing the underlying uncertainties (Section 4.2 and 3) the training procedure for the Beta prior model (Section 4.3)
4.1 Maximum Likelihood Diffusion Policy Alignment
Following previous works on preference align- Rafailov et al. 2024). we adopt the Maximum Entropy (MaxEnt) RL framework (Ziebart 2010 The objective is to learn policy me that maximizes both the cumulative discounted rewards and causal entropy, while regularizing the KL-divergence from a reference policy:
MaxEnt A Alignment under Regret Preference.
ment (Hejna al., 2024
T od (r(ss,, a,) - a log x(arls:) )er(a))s:)
Σ
max ES
(3)
Here, a determines the weight of entropy. Upon learning an optimal policy x², we can compute the corresponding optimal state-value function (*,s)), the optimal state-action value function Q*(st, at). and the optimal advantage function A (st,, a.) A Q",(s,,,) - (sx), which measures the relative benefit of taking a specific action a given state compared to the average expected value of the state. More importantly, in the MaxEnt RL setting, the optimal advantage function is proportional to the
logelikelihodd of the optimal and reference policy (Haarnoja et al. 2017 Hejna et al., 2024):
A A* ax) = a log
π
x*(at(8:)
(rer(a) To stabilize the process of preference alignment, we follow Knox et al. (2022) and base the preference
alignment on discounted regrets, defined as - ~² (V(z)) Q( a.)). In this framework, a
In this work, we use superscripts (i E {0, 1,.., 1} to denote diffusion timesteps and subscripts (! (0, ........) Th)) to denote trajectory timesteps.

---

trajectory segment is preferred if it incurs lower regret compared to the intended optimal policy, so that the preference between trajectory segments (r". r') can be modeled as:
exp E A a4) exp L oe (s)" oz") exp exp Ei A (s),
PA PAo(T">N
(5)
By substituting Equation (4) into Equation (5) the advantage function A can be replaced by the optimal policy x² under the MaxEnt framework. The learned policy uy can then be optimized through maximum the likelihood of generating preferences follows (Hejna et al. 2024):
To Ta(af s( Trat oy ((r)
7d log To(a]] s/) Taes (a/((a))s))
CCP (0) = log logoooo (a.
log
Σ
(6)
2-0
Diffusion Policy Alignment. To adapt the previous model to aligning the diffusion policy to(ae(s)) as defined Eq. [ a primary difficulty due to the intractability diffusion policy mo (aa (8))
S aa (a (s))da as it requires marginalizing over all possible diffusion paths (a, at. ......... 0g that
lead to a.. To address it, we propose modeling the chain reward function (Wallace et al. 2024
r(st, a.) B ( af [r(ss,, ).
The optimal chain advantage function can be defined by marginalizing intermediate diffusion path:
z (a. [st))
(s,, a.) = E (a2e))
(s [ at a = E (a (o? a log
(8)
Teer( (a 0:1 (8x))
In principle, we can interpret the latent diffusion actions unified chain action 0r a: despite the final output being determined by ag. This perspective allows us to reformulate Equation 3 in terms of the diffusion policy:
Σ (r(sse (or(se,, σz) - a log
Ty a
max (oelaa)
Wref (e (8))
This objective is defined over the entire diffusion path ar, which aims to maximize the cumulative rewards and the entropy within a trajectory across the reverse process. By paralleling from Equa-
tion (3) to Equation (6). the objective in (2) can be directly optimized with respect to the diffusion policy Te (e(a) St) by maximizing the following likelihood:
((" i,MLE (0) log a (a
(aj (
E
log
(10)
no (a;
Teat a. (s))
r-0
no (oglot)
Σ
E To
log
Teet (of (s/)
where a is the sigmoid function. However, major challenges in optimizing this objective are: I) inefficiency from sequential computations over timesteps, and 2) intractability due evaluating the joint distribution. Following Wallace et al. (2024). we apply Jensen's 's inequality and the convexity of log a to move the expectation outside, boosting efficiency. We also approximate the reverse process no (a. (sx) with the forward process q(a) (82) to enhance tractability.
With some algebra, we derive the following loss function:
C (-",-') 1.MLE
(0) -E -Bg glo o a. .t"), at al"-g((t 1ab .()
log (ogo(-al)
ai
T
7 (1ee - ce(a , i)ll ||e
(a niw st ,1"))))3)
Σ 7 701e' eglatt s/,0 le' Eneraat , i) (0))
2,MLE (0),
(11)
where 1) ((0, is the diffusion timestep, 2) a////l ~ g( a wwll 0,w//1 w//l denotes the action Dawel corrupted with noise "n/! after diffusion steps, as defined in (Ho et al. 2020). this study,
explore addressing the iterative preference alignment problem by aligning human preferences with a diffusion policy model. The detailed deviation is shown in Appendix
5

---

4.2 Bayesian Alignment with Informative Beta Prior
The regret preference model (Eq. (5) represents the likelihood of generating human preferences based on the advantage function. The corresponding maximum likelihood objective implicitly assumes uniform prior over 2
A (s,, az), which does not account for the uncertainty within the
preference dataset (Newman 2023 Xu et al. (025). We next derive more informative prior.
Since human feedback is provided the trajectory level rather than for individual state-action pairs, the strength of a trajectory can be defined by its trajectory-level advantage, computed as the discounted cumulative advantage under the diffusion policy mp:
A (T)
Σ 7' ( ar) = Σ 2-0
7 (a (s ar)
(12)
t-
The average strength of the trajectories under policy Tp then defined as:
A Ao(T) Prref
Σ (T).
(13)
TED
Therefore, we model the binary outcome of whether a trajectory with strength A (T) wins against the average candidate as a Bernoulli variable with success probability ((T) (A (T) AT (0, 1).
By applying the chain rule, the prior on the advantage function can be defined as:
dø(r) d (T) po (o((()))
Po(A" (T)) po(((T))
A1 [Ppert
(14)
This prior reflects our initial belief about the strength of different trajectories within the dataset. Motivated by Xu et al. (2025), we use the Beta distribution po()(T)
Beta( ax as the informative prior, which serves as the probability density function of Φ((). The main benefits of the Beta distribution are: 1) it is the conjugate prior for the Bernoulli distribution ((T), and 2) the parameters a and 3 can intuitively represent the counts of preferred and umpreferred human feedback. By reformulating Eq. (14). we present the following proposition:
Proposition 4.1 Let the informative prior po()(T)) be Beta distribution Beta(o(T); 3). This prior can effectively capture the uncertainty arising from the iterative preference alignment pro- cess (Definition 3.7). Consequently, the prior on the strength of a trajectory is proportional to Beta((((T); a + 1, + 1)), i.e., Po(A"* (eea) Beta(eeeaaa;)))o
1, 3 +
The proof is shown in Appendix C
The corresponding prior loss then be derived in manner similar to the derivation of the maximum likelihood loss (Eq. [O):
pi.ppror( (0)
log Beta( (eet(((())))) (dT); + 1, + 1)
T ( (a-a al. ((ooaccooa co(a),
E
log Beta
Erer(at,, sc, - 11:1))2)
gt
+1]
(/le co(a), st, i]12 - -lalllllll-alahhhhheeoo curlat, : 1,8 1, 1,1811]
D Dprer| p0
2.tioor ( (T)
(15)
Equation 15 can be interpreted as guiding the diffusion policy align its estimate of Φ(7) with the corresponding prior Beta distribution. According to Bayes' rule, the MAP estimate satisfies PMAP (AA)) po (A(T)) (AL)(A(F)). By incorporating the prior into the MLE objective and maximizing the log form of the posterior, we can derive the Diff-UAPA loss:
LIMEUARA (0) = E(T",r")- "Cb""²") (TT) ( + CT (To) Ce (To)
(16)
Maximizing the posterior probability incorporates prior knowledge and regularizes advantage values, preventing divergence. We introduce how to estimate the Beta prior in the following section.
4.3 Training the Beta Prior Model
To learn the Beta prior po(c)r))(Pped (2025), we propose using a variational inference approach to approximate it by estimating the approximate posterior ge (((()) (Pret) i.c., po( (Ppoet) (e(o(T)(Ppet) where is the model
Beta((((); (3) in continuous spaces, following Xu et al.

---

parameters. The objective is to maximize the Evidence Lower Bound between the prior and posterior. This leads the following trajectory- wise objective (Xu et al. 2025):
(17)
max E [Ens(rr ((7)))("))))))) (r" (ee,tr,,")eDpaallp ((+')())))))))))))) -
where 1) ge(((())TT= Beta(a,, B.), where [a,, =-1= f and f denotes neural network, 2) p(((r)) (eta 90), with co, Bo specifying our prior belief (we set 0g 30 in this work), and 3) ((T) represents the Bernoulli probability that T is ranked higher than 7'. The first two terms aim to optimize the parameter & to align with the preference dataset, while the final KL-divergence term ensures the posterior distribution does not deviate too far from the prior belief, which can be optimized using the Dirichlet VAE approach (J0o et al. 2020).
In this work, we implement Algorithm 1 Uncertainty-aware Preference Alignment for Diffu- sion Policies (Diff-UAPA)
(T) using a transformer- based neural network (Vaswani] 2017), where the trajectory т is fed as input and [a,, 3-1 is produced as the output to form the Beta prior distribution. The complete Diff-UAPA algorithm shown in Algorithm| D
1: Input: Preference dataset Ppret
2: Initialize Beta prior model f (T). (r), policy Tee (es(aa)). and diffussion policy To [alss.
3: Learn Tee via behavior cloning on Prref-
4: Update Beta prior fBeta via Eq. (17) until convergence.
5: Update we by minimizing Eq. (T) until convergence.
5 Empirical Evaluation
In this section, we perform empirical evaluations five robot manipulation tasks across two environments (Sec. 5. locomotion tasks with real human preferences (Sec. (52)). and real-world
pick-and-place task (Sec. 5.3) We further conduct more comprehensive analyses, including ablation studies, evaluations under various noise types, and noise sensitivity tests (.44
Experiment Settings. Our experiments consist of four rounds of iterative updates, each with a fixed number of training episodes. To account for potential inconsistencies in human preferences, we introduce a reverse rate. Specifically, in each round, we randomly select 20% of trajectory pairs and apply 50% reversal rate by flipping the labels. Each experiment is repeated using three random seeds. More experimental details can be found in Appendix D.I
Comparison Methods. We utilize two baseline policies: the couussaa--based policy from Behav- ior Transformer (BET) (Shafiullah al., 2022) and the Diffusion Policy (Diff) (Chi al. 2023) Building on BET, we propose the following comparison methods: 1) BET-Direct Preference Op- timization (BET-DPO) and 2) BET-Contrastive Preference Learning (BET-CPL.) which leverage direct preference optimization (Rafailov et al., 2023) and contrastive preference learning (Hejna) al. 2024) align the BET model and 3)
(uu al., 2025), which learns a distributional policy for epistemic uncertainty awareness.. For diffusion- policies, we introduce: 4) Diffuusion Policy-CPL (Diff-CPL) that uses MLE for aligning the diffusion policy (Obj. [1] and 5) FKPD Shan et | (2024) that performs forward KL regularized preference optimization. For our Diff-UAPA algorithm, we explore two strategies for updating the Beta prior model: 6) DifT-UAPA- that trains the Beta model using full preference data across the iterations without updates and 7) DifT-UAPA-I that incrementally updates the Beta model on the current preference data through the iterative process.
5.1 Model Performance in Robot Manipulation Tasks
We evaluate the model's performance across four tasks from Robomimic (Mandlekar et al. 2021 and the Franka Kitchen task introduced in (Gupta et al., 2019). both of which use state-based observations. The Robomimic tasks cover diverse manipulation skills, while the Kitchen task involves long -orizonn multi-step interactions with seven objects, aiming complete many tasks possible in any order. For each task, the reference policy Tiet is trained to reach approximately 40% success, then rolled out to collect 560 trajectories used to build the preference dataset based on their rewards. Please check Appendix D.2 for environmental details and Appendix D.4 for preference dataset construction.
Table I presents the evaluation results. The results indicate that both variants of Diff-UAPA con- sistently outperform other methods across different tasks. This is primarily due to their use of a Beta prior, which effectively captures the uncertainty arising from inconsistent preferences, thereby enhancing the diffusion policy training process. Moreover, the performance gap between DAPP- Cand Diff-UAPA- I is relatively small, suggesting that the Beta prior can be trained effectively in
7

---

Table 1: Success rates (in percentage) across tasks, with each value presented as the mean + std over 3 training seeds and 560 evaluation episodes. The best results for each task are highlighted in bold. For the Kitchen task, pz indicates the frequency of interaction with or more objects.
Robomimic
Kitchen
Lift
Can
Square
Transport
pl
p2
p3
p4
BET
43.6
48.8 3.1
55.1 2.0
43.1 1.9 1.9
96.4
96.2
76.6 1.3
44.6 2,0
BET-CPL
49.2 4.4
42.1= 1.
2.3
4.8
97.0
9.5
88.4 2.3
2,0
BET-DPO
3.3
47.0=10
4..6
4.2
85.5
+ 8.5
8.7
8.99
6,,4
UA-PbRL
50,3
5.34
5..3
53.6
100.0
9.
92.2
6..3
Diff
45.1=
4
52.8299
5..
9.8
9..4
59.0 91.8
Diff-CPL.
48.6=22
4..9=28
55.2
58.1
100.0
9.26
9.2
6.85
FKPD
51.2 0.7
=8.5=2.5
52.3 2.74
9
9..
8.9
64.1
DiS--UAPACC
56.1 0.9
61.3 2.2
68.1 = 0.6
64.0 + 4.0
100.0 +: 0.0
9.2
95
70.9
2,5
Diff-UAPA-I
54.3 L1
59.9
1.7
66.2
= 1.3
61.5
2.6
99.9 = 0.1 99.8 1:
95.7 1.9
71.7 1 4.6
both approaches, depending on the specific practice. Notably, for the long-horizon Kitchen task, Diff-UAPA-I, which trains the Beta model incrementally, slightly outperforms Diff- UAPAAC which pre-trains the Beta model using the complete dataset. This difference be attributed the fact that incremental training allows the model to adapt more dynamically to the changing preferences and environmental conditions over time, whereas pre-training may not fully capture such variability. We also provide the visualization results in Figure in Appendix D.6 and Supplementary Materials.
5.2 Model Performance in Locomotion Tasks with Real Human Preferences
The primary goal of PbRL is to align policies with human preferences. In this section, evaluate on real human preferences from Uni-RLHF (Yuan et al.] (024)) in the HalfCheetah and Hopper tasks from D4RL (Fu trajectory coverage. Please check Appendix D.2 for more environmental details.
al. 2020), using medium-expie expert and medium-replay datasets to ensure diverse
Table 2: Episodic rewards of all methods in the HalfCheetah and Hopper environments with real human preferences. 'M' denotes Medium and 'R' denotes Replay.
BET
BET-CPL
BET-DPO
Diff
Diff-CPL
FKPD
Diff-UAPA-C
Diff-LAPA-I
Hal(ChottabM--E
2577 198
2976 + 66
2948
+
2838 325
3121 148
3060 201
3399 72
3297 101
HalfCheetah-M-Al
1580
1818 201
1659
1691
+
2021
1949
Hopper M-E
1161 1 90
1226 1 85
1129 1 79
1296 1 137
1313 103
1370 1 120
1591 1.51
1499 i: 70
Hopper-M-ER M-R
769
5:
31
7961:20
874 1:39
933
39
The results are presented in Table 2 We find that baseline methods can hardly outperform Diff-UAPA. This occurs because the inconsistent labels increase uncertainty and hinder the policy's ability to identify and imitate better trajectories. Diff-UAPA addresses this by using a prior model to capture uncertainty, enabling more reliable trajectory evaluation and improved performance. We also find that diffusion--asses policies outperform Gaussian ones, due to their stronger modeling capacity.
5.3 Real-world Fick-and-Place Task
To assess the effectiveness of the proposed method in sim-to-real transfer and real-workd deployment, we conducted a real-world nick-and-place experiment using a Rokae SR3 robotic arm. The task involved picking up banana and placing it in a designated area. Over 3000 simulated trajectories were generated via motion planning with visual observations and proprioceptive inputs, encompassing both expert and sub-optimal executions. Preference labels were assigned to favor expert trajectories.
We conducted real-world evaluations, where our method achieved 60% success rate, surpass- ing the 40% achieved by the Diff-CPL baseline. Diff-CPL. suffers from the noise that prefers sub- optimal behaviors like reaching the wrong posi- tion. As shown in Figure 2 the red circle marks an incorrect pick pose by Diff-CPL, while the green circle highlights the accurate pick-and- place behavior generated by our method. These results demonstrate the robustness and effectiveness of our approach in sim-to-real transfer. We also provide the full process visualization in Figure 4 and videos in the Supplemental Materials.
Figure Left: Diff-CPL: Right: Diff-UAPA.
8

---

5.4 More Experiments
Ablation Studies. To assess the contributions of the diffusion policy and Beta prior components, we conducted ablation studies across multiple tasks. The evaluated variants include: 1) BET-CPI, which trains BET using the MIE objective, 2) BET-UAPA, which applies the MAP objective with learned Beta prior, 3) Diff-CPL, which trains the diffusion policy with the MLE objective, and 4) SA--UAAA-Unffoot which applies the MAP objective with uniform Beta prior.
Table 3: Ablation results across Robomimic and Kitchen tasks. v indicates the inclusion of the corresponding module, while A indicates partial inclusion of the module.
Method
Diff
Modules
Beta
Lift
Can
Rohomimic
Square
Transport
pl
Kitchen
pa
p4
BET-CPL.
49.2
42.1
576
45.2
488
97.0
62.6
BET-UAPA
54.3
1:
0.5
48.0
1
28
602
1 25
$2.3
: 09
99.3
1 0.
9933
1
09
94.7
1
1.0
68.0
1.6
Din--CIL
48.6
+
2.2
45.9
28
55.
57
58.1
= 6.2
100.0
: 0.0
9966
02
942
0.2
63.5
088
DIf-UAAPN-Uniorm
48.5
544
2.7
58.2
9.8
99.8
08.3
14
93.5
64.1
Didt UAPA-C
68.1
0.6
64.0
4.0
100.0
0.0
+
70.9
The results are presented in T
Comparing BET-based and Diff-based models highlights the superior performance of the diffusion policy. Additionally, evaluating methods with and without the Beta prior demonstrates the effectiveness of the proposed Beta model.
Various Noise Types. Following Bukharin et al. (2024), we experimented under two additional types of noises beyond the previously used irrational noise, including 1) Stochastic Noise, which generates preference labels by scaling the reward difference between pairs using a sampled tempera- ture T in (1.3) at each round; and 2) Myopic Noise, which derives preference labels from cumulative discounted rewards computed with an random discount factor in (0.5, 0.999) at each round.
We include a broader of baselines for comparison. These include two-step PbRL approaches that first learn a reward function: 1) BET-PbRL, which integrates BET with TD3BC (Fujimoto & Gu 2021), and 2) Diff-PbRL, which employs an efficient diffusion policy (Kang 2023a) Additionally, we consider robust preference alignment methods, including 4) RIME (Cheng et al.)
2024) and 5) R3M-DPO SEM-DPO(Bukharin
al. 2024). We use the Diff-UAPA-I variant for comparison.
Table 4: Performance comparison under different types of noisy preferences.
HaltCheetah-Meellm--eR Expert
MalCCheetah-Medume
Irrational Noise
Stochastic Noise
Myopic Noise
Irrational Noise
Stochastic Noise
Myopic Noise
BET-PbRL
2851 + 45
3218 + 102
2890 81
1687 91
1430 + 90
1209 + 45
Diff-PbRL
3158 : 44
3298 1 86
3002 = =58
1730 + 32
1499 + 133
1389 + 86
UA-PbRL
3291 + 103
3410 72
3198 90
1803 +
1674 +
1503 1
RIME
3200 92
3397 58
3090 101
1891 114
1592 101
1459 120
R3M-DPO
3258 1 80
3505 + 91
3516 68
1914 + 70
1643 4 196
1655 455
Diff-UAPA
3297 + 101
3674 + 169
3458 : 118
1949 5:
1715 57
1707 :: 85
The results shown
demonstrattttt that the effectiveness of Diff-UAPA. This advantage stems from its ability to handle iterative preference alignment and progressively adapts the policy to these evolving signals. In contrast, baseline methods generally assume a static preference dataset, making them less effective in this dynamic setting. Moreover, robust preference learning methods that rely on filtering strategies may discard diverse yet informative preferences, thereby limiting the opportunity for comprehensive learning potentially useful preferences.
Noise Sensitivity. We also evaluate noise sensitivity in the Franka Kitchen environment by varying the reversal rate r from 50% (as used in previous experiments) to 25% and 75% assessing robustness to label inconsistency. For clarity, report only the most challenging p4 metric.
Table presents the evaluation results. As the noise level increases (i.e., the reversal rate), all methods show a decline in performance, highlighting the sig- nificance of uncertainties in the dataset. However, compared to the other methods, Diff-UAPA consis- tently exhibits better performance with the highest success rate regardless of the scale of noise. This un- derscores the effectiveness of incorporating the Beta prior model to handle such uncertainties.
Table 5: Evaluation under different levels of reverse rates in the Kitchen environment.
2=25
r=50%
r=75%
BET-CPL
65.7
1.6
62.6 2.0
55.0
BET-DPO
60.2
6.6
47.2
Diff-CPL
1.0
63.5
2.5
FKPD
21.
64.1
62.3 4.6
Diff-UAPA-C
7..9
70.9
70,5
338
Diff-UAPA-I
7..0
7.6
5.21

---

Reference Policy Sensitivity. To further verify the sensitivity to the success rate of the reference policy, we conducted a sensitivity test the Robomimic environment on the Lift task by introducing reference policies with success rates of 25% and 75%.
From the results in Table 6 the success rate of the reference policy has significant impact on the perfor- mance of the final policy. However, the performance of Diff-UAPA outperform than other algorithms in different success rates of reference policies, further demonstrating the importance of the beta prior com- ponent to the robustness of PbRL algorithms.
Table 6: Evaluation under different success
rates of reference policies in Lift Task.
r=25%
r=50%
r=75%
BET-CPL
4.2
56,3 +
79.0 +
BET-DPO
3:50
50.4
1 3.3
76.3 1 1.3
DIP
40.6
53.7
13.7.221 2.1
80.0
2.0
FKPD
42.0+2200
56.2
Diff-UAPA-C
44.7 1:4.1
58.1 1 1.7
84,0
+ 3,5
DAP-UAPA-I
41.7
56.1 1.3
82.7
1.1
6 Conclusion
In this paper, we present an uncertainty-aware preference alignment approach for diffusion policies using an iteratively updated preference dataset. Building on the maximum likelihood objective for directly aligning diffusion policies without learning reward model, we introduce Maximum Posteriori (MAP) objective with an informative Beta prior, which is capable of capturing the uncertainty arising from potentially inconsistent human preferences. Empirical results across various domains demonstrate the effectiveness of our method. Future work will focus on extending this framework to the online setting, enabling agents to interact with the environment and dynamically adapt evolving human preferences.
Acknowledgments
This work is supported in part by Shenzhen Science and Technology Major Program under grant 00200240901140008000 Shenzhen Fundamental Research Program (General Program) under grant JCYJ20230807/ 14202005, Guangdong-Shenzzeer Joint Research Fund under grant 2023A1515110617
(UHK(S2)-GDSTTC Joint Collaboration Fund No. 2025A0505000053, Guangdong Basic and Applied Basic Research Foundation under grant 2024A1515012103 and Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001).
References
Ajay, A., Du, Y., Gupta, A., Tenenbaum, J., Jaakkola, T., and Agrawal, P. conditional gener- ative modeling all you need for decision-making? In International Conference on Learning Representations, 2023.
An, G., Lee, J., Zuo, X., Kosaka, N., Kim, K.-M., and Song. H. O. Direct cefference-baseed policy optimization without reward modeling. Advances in Neural Information Processing Systems, 36: 70247-70266, 2023.
Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):344-345 1952.
Bukharin, A., Hong, I., Jiang, H., Li, Z., Zhang, Q., Zhang, Z., and Zhao, T. Robust reinforcement learning from corrupted human feedback. Advances in Neural Information Processing Systems, 2024.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T. Lindner, D., Freire, P., et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint urXiv:2307.15211 2023.
Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline reinforcement learning via high-fidelity gener- ative behavior modeling. In The Eleventh International Conference Learning Representations, 2023a.
Chen, I.., Bahl, S.,, and Pathak, D. Playfusion: Skill acquisition via diffusion from language-annotated play. In Conference Robot Learning, pp. 2012-2029, 2023b.
10

---

Chen, Y., Li, H., and Zhao, D. Boosting continuous control with consistency policy. In Autonomous Agents and Multiagent Systems, pp. 535-344, 2024.
Cheng, J. Xiong, G., Dai, X., Miao, Q. Lv, Y., and Wang, F.-Y. Rime: Robust preference-based reinforcement learning with noisy preferences. In International Conference Machine Learning. ICML, 2024.
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. Diffusion poliey: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2023.
Choi, H., Jung, S., Ahn, H., and Moon, T. Listwise reward estimation for offline preference-based reinforcement learning. International Conference on Machine Learning, 2024.
Christiano, P. F, Leike, J., Brown, T. B., Martic, M., Legg. S., and Amodei, D. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp- 4299-4307, 2017.
Dong. 7.,, Yuan, Y., J.O, Ni, F., Mu, Y., ZHENG, Y., Hu, Y., Lv, T., Fan, C., and Hu, Z. Aligndiff: Aligning diverse human preferences via behavior-customisable cuatooi-custooiiablee diffusion model. In International Conference on Learning Representations, 2024.
Fang, B., Jia, S., Guo, D., Xu, M.. Wen, S., and Sun, F. Survey of imitation learning for robotic manipulation. Int. J. Intell. Robotics Appl., 344:3622-69,, 2019.
Fu. J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint 0tti::20040072119 2020.
Fujimoto, S. and Gu, S. S. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-2011 2021.
Gupta, A., Kumar, V., Lynch, C., Levine, S., and Hausman, K. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint 191.v:1910.111 2019.
Haarnoja, T., Tang, H., Abbeel, P., and Levine, Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1352-1361, 2017.
Hejna, J. and Sadigh, D. Inverse preference learning: Reeeeence--bsedd rl without reward function. Advances in Neural Information Processing Systems, 36, 2024.
Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., and Sadigh, D. Contrastive preference learning: Learning from human feedback without rl. In International Conference on Learning Representations, 2024.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in Neural Information Processing systems, 33:6840-6851. 2020.
Hwang, M., Lee, G., Kee, H., Kim, C. W., Lee, K., and Oh, S. Sequential preference ranking for efficient reinforcement learning from human feedback. Advances in Neural Information Processing Systems, 36:49088-49099 2023.
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems, NeurlPS, pp. 8022-8034, 2018.
Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
Joo, W. Lee, W., Park, S., and Moon, I. Dirichlet variational autoencoder. Pattern Recognition, 107: 107514, 2020.
Kang, B., Ma, X., Du, C., Pang, T., and Yan, Efficient diffusion policies for offline reinforcement learning, Advances in Neural Information Processing Systems, 36, 2023a.

---

Kang, Y. Shi, D., Liu, J., He, L., and Wang, D. Beyond reward: Offline preference-guide policy optimization. In International Conference on Machine Learning. 2023b.
Kim, C., Park, J., Shin, J., Lee, H., Abbeel, P., and Lee, K. Preference transformer: Modeling human preferences using transformers for RL. In International Conference on Learning Representations, 2023.
Knox, W. B., Hetsis- S., Booth, S., Niekum, S., Stone, P., and Allievi, A. Models of human preference for learning reward functions. arXiv preprint (rXii:2206.022311 2022.
Lec, K., Smith, L., and Abbeel, Pebble: Feeddbck--effcient interactive reinforcement learning via relabeling experience and unsupervised pre-training, In International Conference on Machine Learning, 2021.
Liang, X., Shu, K., Lee, K., and Abbeel, P. Reward uncertainty for exploration in preference-based reinforcement learning. arXiv preprint arXiv:2205. 12401, 2022.
Liu, B., Shao, S., Li, B., Bai, I.,, Xu, 7.,, Xiong, H., Kwok, J., Helal, S., and Xie, 7. Alignment of diffusion models: Fundamentals, challenges, and future. arXiv preprint vrXiv2240.002533 2024.
Lu, C.,, Chen, H., Chen, J., Su, H., Li, C., and Zhu, J. Contrastive energy prediction for exact evergy-guidee diffusion sampling in offline reinforcement learning, In International Conference on Machine Learning, pp. 22825-22855, 2023.
Mandal, D., Nika, A., Kamalaruban, P.. Singla, A., and Radanović, G. Corruption robust offline reinforcement learning with human feedback. arXiv preprint arXiv:2402.068 067344 2024.
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., and Martín-Martín, R. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
Mnih, V., Kavukcuoglu, K., Silver, D.,, Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Ried- miller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-333, 2015.
Newman, M. E. Efficient computation of rankings from pairwise comparisons. Journal of Machine Learning Research, 24(238):1-25, 2023.
Park, J., Seo, Y., Shin, J., Lee, H., Abbeel, P., and Lee, K. Surf: Semi-supervised reward learning with data augmentation for bredback-efficieen preference-basee based reinforcement learning. In International Conference Learning Representations, 2022.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In Advances in Neural Information Processing Systems, 2023.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From SrS Sqa#$; Your language model is secretly q-function. In First Conference on Language Modeling, 2024.
Shafiullah, N. M. M., Cui, Z. Altanzaya, A., and Pinto, L. Behavior transformers: Cloning SkS modes with one stone. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
Shan, Z., Fan, C., Qiu, S., Shi, J., and Bai, C. Forward kl regularized preference optimization for aligning diffusion policies. arXiv preprint arXiv:2409.05662 2024.
Shin, D., Dragan, A. D., and Brown, D. S. Benchmarks and algorithms for offline preference-based reward learning. Transactions on Machine Learning Research, 2023.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.,, Van Den Driessche, G., Schrittwieser. J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484445) 2016.
12

---

Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arxiv preprint arXiv:2011.13456. 2020.
Sridhar, A., Shah, D., Glossop, C., and Levine. S. Nomad: Goal masked diffusion policies for navigation and exploration. In JEEE International Conference on Robotics and Automation, pp. 63-70, 2024.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Verma, M. and Metcalf, K. Hindsight priors for reward learning from human preferences. In International Conference on Learning Representations, 2024.
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C.,, Joty, S., and Naik, Diffusion model alignment using direct preference optimization. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8228 8238, 2024.
Wang. X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X.,, Dai, B., and Miao, Q. Deep reinforcement learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 35(4):5064 5078, 2022.
Wang. Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations, 2023.
Warnell, G., Waytowich, N., Lawhern, V., and Stone, P. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. A survey of preference- reinforcement learning methods. Journal of Machine Learning Research,
2017.
Xu, S., Yue, B., Zha, H., and Liu, G. A distributional approach to uncertainty-awant preference alignment using offline demonstrations. In International Conference on Learning Representations, 2025.
Xue, W., An, B., Yan, S., and Xu, Z. Reinforcement learning from diverse human preferences. In International Joint Conference on Artificial Intelligence, 2024.
Yang. L., Huang, Z., Lei, F., Zhong, Y., Yang, Y., Fang, C., Wen, S., Zhou, B., and Lin," Z. Pol- icy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13112, 2023.
Yuan, Y., Hao, J., Ma, Y., Dong, Z., Liang. H., Liu, J., Feng, Z., Zhao, K., and Zheng, Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In International Conference on Learning Representations, ICLR, 2024.
Zhu, Z.,, Zhao, H., He, H., Zhong, Y., Zhang, S., Guo, H., Chen, T., and Zhang, W. Diffusion models for reinforcement learning: A survey, arXiv preprint arXiv:3311.01225 2023.
Ziebart, D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
13

---

Contents of Appendix
More Details in Section 4.1
15
More Details in Section 4.2
15
Proof of Proposition 4.1
17
D
More Experimental Details
17
D.1 Experimental Settings
17
D.2
Environmental Details
18
D.3
Computational Overhead
18
D.4
Manipulation Preference Dataset
19
D.5
Hyperparameters
19
D.6
Visualization Results
19
Limitation
20
Social Impact
20
Author Contributions
21
14

---

A
More Details in Section 4.1
We detailed the deviation from Equation (10) Equation 11 here.
2/ (0)
(())
Ta(a
T Σ
x2 log To To(a (s))s))
(re((a)(sf) ))
=-loge =-logo
B no
log
xuto 1: logiafitt 1
Tiat aa)"
To (a (11)
=-logo
B
7 log
To
log
11110 no (0)
TTee (a (st")
Tral Tead (a/11))
t-0
- logo
Σ
T
B
7" log
(a.
1 1s!
- log
(a
(s))
t-0
molos
Teel a
18,00 |s"")
Tar(
1-1 (st))
T
[a(a; --
To (a 11id (s!)
(ai
log a a
(aa Aele Ri)
? log
log
(0 111 1s/)
Tral
To (a= - 11.1 (s?")
to -- st{)
- | )))
E log logo(al.
mogalalll yasta ooe .. of malel oi 2. to (1 (1
i-2010.))
log
nzel (ap 1-111:1110 118, (s7")
log
(a
Since
log (x) is a convex function:
(=loga(z))) = (o(x) 1) (o(x)( axx)) >> 0
According to Jensen's 's inequality:
To (a; 1/100 1x;²
E
t-1(0,1).
log al aaJ-B,y- 1B,--1 xx)(-
7r log
(a -- 18,,00 1
a "imaq(a"""aa .a7). maaa 11. (
Tul x- (1.1119555)))
-7 log
(ag 1/kit 1st)
(18)
T Σ
E
log a al
(7 DKI.
Dic..Taaaa (-111111 117)))77)) I| newr 111100 1
(-1
of ~o(a) (a(afi"ia 10 .2).
>'IDII.[TOOC 1111)))) || Tred (a, (11, (11l9166))))))) [6)1)
According to Formula
T it can be further simplified as:
log a (-al-
T
-E.-u of -g((7]44).s))
Σ
>' (lle" - 7.00 ")"",))11 """"""",,""""" -lle" "a ,,,,, T1)3)
(lle - co(a," sT 7)12 -IlE- =nee sT,
Dprey
where 1)i~ ((0, I) is the diffusion timestep, 2) at dwefl ~ g( a
(a 0.wif swil) denotes the action 1.0f
Daw/l corrupted with noise w/ll after diffusion steps, and 3) Eg ag
will is the noise predictor.
B More Details in Section 4.2
We detailed the deviation of Equation 15 here.
15

---

c(r 1 (8) 1.pter
loo
Beta
100
T
B
To(at
log aa (s
(a (()
ped tmo
og (n(
1 2: ITT 2
36 log
Thar (a (x7)
(a (x5)
:0 + +.1+11
log
Lid 12gi0
72 lox
(a (x"))
and (0 [of) Proel
log
xo(a [o[]
:a + 1, 1.8+11)
ard
sola, 11
log mo 11: 1077
log
7" log Tarla, 116 100)
peet
ro
TC peet
marla [s]]
(a 1 po7)))
log a
B
Ta
Miiii )
Σ
x' log
Taala; 11
Σ
T Ppurl
low
veD find
sota (14
log
al
E i-te(o.
Σ > log
log
Kar 61"(s")
(ar (0-11197777)
re puot 30a
Since - log o (Beta(x; az .)) is a convex function when a + B > 2. Define g(t) = (og() Since
- -log(o(t)) = log +
it suffices to show that log(1 + t) is convex in t. Differentiating, _ Ite et +1
1
log + t
di
and hence
d²
e¹
log + + (et 1)2 (vt R). dt
This shows log( 1 + e-t) is strictly convex in t. Therefore, for the function
f(x) - log (Beta(æ; + 1, 1,,8+11 1))}
the inner part Beta(æ; 1, 1) serves the real argument and the composition preserves
convexity, implying [((x) is convex.
According to Jensen's inequality
[ log (Beta aIl (F " (12.0""))
TED at' >a(a"" "a (airee)) .t()
log (a
(111111,00177 (s7)
Σ
T
log
Tola i- 1/6,7 155] ]l) 11, +1++
[a i- 118,000 137"))
D Drrerl
[ae
112. |s7]
re 1
T
log logo aT.
(al
Σ
(D'DKK.
Tooaa (1 (8(") || m
-- (o(1)
TeDs
a} (g(a)"()) 100 ( .86)
1-1/is7 (8%) || Trer(at 115 115 (o6))))
DKL. (a
D (Prett
T=Dpat.t=0
According to Formula ①1, it can be further simplified as:
T
-E DeDaut aqaa" la! .s.)
log or (-al- Σ (1ee " "o(at"",s""" ,),,i)] -le" Eroo(a ,,,, ())1))
(le" - ee(att",,,, - le - Ger(att ,s7, ())
rEDport (e 1=0 (Drort
16

---

where 1) i ~ u(0,1) I) is the diffusion timestep. 2) a q(a) la," .s) denotes the action a, corrupted with noise after i diffusion steps, and 3) Eg the noise predictor.
C Proof of Proposition 4.1
Proposition 4.1 can be divided into two parts: 1) the uncertainty-awarr property of the Beta prior, and 2) the prior on the strength of a trajectory.
Part 1. We show the uncertainty-awe aware capability of the Beta prior
during the
iterative preference alignment process outlined in Definition 3.T as follows.
The probability density function (PDF) of the Beta distribution Beta ((T); () is given by:
f(o(T); o(T) ((()))(((1 (())) o ((T)) &
(19)
B(a, B)
where B(a, B) (1- di is the Beta function, serving as a normalizing constant. The variance of a Beta distribution Beta( (cta()(T)::s) is given by the following formula:
o:3
Var(Beta(a, 8)) (a + ()
(20)
In the process described in Definition 3.1 the uncertainty arises from the varying preferences of different human raters for a given trajectory pair (r', r3), Without loss of generality, assuming an initial belief of Beta(11, 1) for each trajectory, and with 10 raters evaluating candidate pair (r', the Beta prior is updated according to the preferences expressed by the raters. For instance, in the first case, where 9 raters prefer I¹ and rater prefers 7", the Beta prior for r' would be updated to Beta(10 10, 2). In the second case, where 5 raters prefer 7' and 5 prefer r', the Beta prior for 7' would become Beta( 6, 6). Intuitively,
would be more confident with less uncertainty in the first case, as the majority of raters share the same preference.
The Beta distribution effectively captures this uncertainty. As shown in Equation (20), the variance of Beta(10, 10, 2) is smaller than that Beta 6, 6), indicating that Beta( 2) is 'sharper' and reflects less uncertainty, which aligns with our intuition.
Part 2. We prove that the prior the strength of a trajectory is proportional to Beta( 1.8 + 1)), i.., Po( (r
1, + 1), as follows.
Recall that the probability of trajectory with strength A**(T) winning against the average candidate is given by ((T) (A)*(T)) ATe) (0. Let A**(T) - A be denoted as A (T).
According to Equation we have that the Beta distribution over ((() o(A**))) (r) is:
(eta (Ā== (7)); B)x AH (7)) (7)))=((1 (1 a T (
(21)
The derivative of the sigmoid function is:
(22)
By incorporating Equation 21 and Equation 22 into Equation
(4). we have that:
(TT) ( (T)) -1- (1)
x. { (1))
(23)
D More Experimental Details
D.1 Experimental Settings
In this paper, we utilized a total of NVIDIA GeForce RTX 3090 GPUs, each with 24 GB of memory. The random seeds used for the experiments were 42, 43, and 44. Each experiment is repeated
17

---

using these random seeds, and the mean + standard deviation (std) of the results is reported. The learning rate is reset at the beginning of each round enhance stability. We trained the agents offline and selected the final epoch for evaluation across 56 parallel environments, each with 10 episodes. Additionally, we employed a transformer-basee architecture for the Beta model in the preference transformer (Kim et al., 2023).
D.2 Environmental Details
Manipulation Tasks. Robomimic (Mandlekar et al. 2021) is a large-scale robotic manipulation benchmark designed explore imitation learning and offiie reinforcement learning (RL). It consists of five tasks, each with a proficient human (PH) teleoperated demonstration dataset, and four tasks also feature mixed sercciint/non-rofffccee human (MH) demonstration datasets, resulting in a total of nine variants. In this paper, we focus on three tasks: Lift, Can, and Square. Specifically:
Lift: The robot arm must lift small cube. This is the simplest task.
Can: The robot must move a Coke can from a large bin to a smaller target bin. This task slightly more challenging than Lift, as picking up the can is more difficult than picking up the cube, and the can must placed accurately in the target bin.
Square: The robot is required to pick up square nut and place onto rod. This task is significantly more difficult than Lift and Can, as it demands high precision to pick up the nut and insert into the rod.
Transport: The robot needs to open the box and transport the hammer in the box to another robot.
The Franka Kitchen also a widely used environment for evaluating the performance of methods in learning complex, long-horizon tasks. Introduced in Relay Policy Learning (Gupta al. 2019), the environment features seven objects for interaction and includes a human demonstration dataset consisting of 566 demonstrations, each completing four tasks in random order. The objective is to execute as many of the demonstrated tasks as possible, regardless of their order, highlighting both short-horizon and long-horizon multimodal capabilities.
Locomotion Tasks. We evaluate our locomotion tasks using the D4RL benchmark (Fu et al. 2020) which is widely used in reinforcement learning (RL) for continuous control tasks. In this paper, focus on the Hopper and HalfCheetah environments. In these environments, the goal to maximize the cumulative reward within single episode by navigating sequence of actions that optimize the agent's 's movement and efficiency. More specifically:
Hopper: In this task, the agent controls 2D hopping robot, with the objective of balancing and moving the robot forward using as few steps as possible.
HalfCheetah: this task, the agent controls a 2D robotic cheetah, aiming to run as fast as possible while maximizing speed and maintaining stability.
We use the medium-exper and medium-replay datasets for both environments. The the medium- expert dataset combines expert demonstrations with suboptimal trajectories, while the medium-replay dataset contains the replay buffer from a partially trained SAC policy (Haarnoja et al. (017).
D.3 Computational Overhead
The additional computational overhead can primarily be attributed to the following components:
Diffusion policy. While diffusion policies incur higher computational costs than simpler architec- tures like MLPs, this overhead is partially offset by the action sequence prediction strategy (Chi et al. 2023). More importantly, diffusion models are widely adopted in RL for their strong generative capabilities and superior performance. practice, training time for diffusion is roughly twice that of the transformer in experiments.
18

---

Beta model. In this work, we use efficient techniques like the reparameterization trick to improve scalability, In practice, the computational cost of training the Beta model is similar training reward model in traditional PbRL. Since our method avoids training reward model, the added cost less effective compared to conventional PbRL.. Additionally, the extra computational cost only slightly increases training time-by few minutes-whne the subsequent RL phase is much more demanding, often taking several hours.
D.4
Manipulation Preference Dataset
For the robot manipulation tasks, we train two policies using behavior cloning: the BET policy and the diffusion policy. Training proceeds until a 40% success rate reached. To build the simulation environment, we deploy 56 parallel environments, each initialized with a different seed to ensure varied initial positions for the agent. We then collect 560 trajectories per policy. From these, we randomly select 500 trajectory pairs and label them based on the sum of their rewards. During training, each trajectory sliced using the observed steps as the stride, and these segments are compared. In the iterative update process, for each update round, we randomly select 20% of the trajectory pairs and apply a 50% reversal rate by swapping winner and loser. improve stability and convergence, the learning rate is reset at the start of each round.
D.5 Hyperparameters
Our experiments are primarily based on the codebase from (Chi al. the same hyperparameters for training the diffusion policy as specified in (Chi et al. 2023) for each experiment. The specific hyperparameters for Diff-UAPA are listed in Table
2023). Therefore, we retain
Table 7: List of the specific hyperparameters for the proposed Diff-UAPA. To ensure fair comparisons,
we maintain consistency in other parameters of the same neural networks across different models.
Parameters
Robomimic
Kitchen
D4RL
General
Training Epochs
600
600
600
Episode Length
400
280
1000
Beta Model
Network
256
256
256
Learning Rate
2e-5
2e-5
3e-5
Number of Attention Heads
4
4
4
Number of Layers
Batch Size
32
32
64
Initial Belief
a =
3
=
a
=
a =
3
D.6 Visualization Results
Figure presents visualization results from the manipulation tasks. is evident that the baseline method, Diff-CPL, which is trained using the MLE objective, struggles to handle certain critical scenarios, particularly those involving noisy preferences.
Figure presents visualization results from the worl-world pick-am and pod-pplac experiment. As shown, Diff-UAP! successfully completes the task by picking up and placing the banana, whereas Diff-CPL struggles to pick up the banana due to the impact learning from noisy preference labels.
19

---

Lift
Can
Square
Transport
Kitchen
Diff-CPL
graiitil
Failing
liiinghhight
Insufficient
Misaligninir
withethe Tralls
Ipcalsistiilcy of graboing
Failing grab handle
Diff-UAPA (ours)
Figure 3: Visualization results five manipulation tasks.
F16
t=32
t
64
Diff-UAPA Dift-CPL.
(ours)
Figure 4: Visualization results the pick-and and-place task.
E
Limitation
Offiline Trajectory Dataset. This paper primarily focuses on learning from an offline trajectory dataset with potentially inconsistent human preferences that are iteratively updated, where the agent cannot directly interact with the environment. This partial offline setup may limit the agent's ability to explore and discover improved strategies through interactive online learning. However, our method can also generalize to an online setting, where both trajectories and human preferences are dynamically updated over time.
Computational Overhead. The integration of training a Beta prior model through variational inference adds computational complexity compared to simpler MLE-based methods. However, by utilizing efficient techniques like the reparameterization trick to enhance scalability, the computational overhead of training the Beta model minimal in practice, adding only small additional time cost relative to the diffusion training process.
F Social Impact
The proposed Diff- UAPA framework presents meaningful implications for aligning Al agents with diverse and potentially conflicting human preferences. By explicitly modeling uncertainty through Bayesian prior, this approach promotes fairness and inclusivity in decision-making systems by preventing the marginalization of minority or inconsistent viewpoints. In high-stakes domains such as assistive robotics, healthcare, automated systems interacting with vulnerable populations, Diff- UAPA's robustness to noisy and heterogeneous feedback helps ensure safer and more equitable outcomes. However, care must be taken in interpreting preference data, biases in human feedback can propagate through the model. As with any alignment technique, ethical considerations related to whose preferences are prioritized and how disagreements resolved remain critical areas.
20

---

G Author Contributions
Sheng Xu and Guiliang Liu conceived the initial ideas, organized, and led the research project. Runqing Miao developed the code, conducted all the experiments, and prepared the initial manuscript draft. Sheng Xu contributed to the experimental design and refined and finalized the writing, Runyi Zhao assisted with the real-world experiments. Wai Kin Victor Chan provided valuable feedback, and Guiliang Liu offered overall supervision and constructive guidance throughout the study. Runqing Miao and Sheng Xu are co- first authors with equal contributions, listed in alphabetical order.
21

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They can support the main claims of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section B
22

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We rigorously prove our theoretical results, and the details can be found in Section A Section B and Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We report experimental settings and hyperparameters in Section and provide our code in the Supplemental Materials.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our code in the Supplemental Materials with a clear README file to reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public/eee.c www.ddddeeeooeuuummmcecom ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
24

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We report our experimental settings and hyperparameters in Section D and provide our code in the Supplemental Materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We perform the experiments under three random seeds and report the mean and std results.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the compute resources in Section D.I
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We strictly follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
26

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We strictly follow the corresponding licenses for existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
27

---

Answer: [NA]
Justification: We only use the simulated and public data, without involving crowd-sourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non ntn-standard components.
Please refer to LLM policy httppp////eerrpecoccco/ooerereep for what should or should not be described.
28


Reviews:
[{"review_id": "McAsCY9mB4", "reviewer": "Reviewer_avn8", "summary": "The paper proposes Diff-UAPA, a direct preference alignment algorithm for diffusion policies, designed to address inherent uncertainties in human preferences across diverse user groups. Diff-UAPA uses a MAP objective, guided by an informative Beta prior for capturing the uncertainty. Experiments on both simulated robotic and real-world tasks demonstrate the robustness of Diff-UAPA for the uncertainty in preference data.", "questions": "Q1. Can we come up with the proposed method's online counterpart?\n\nQ2. Could the author provide a discussion on the advantages and disadvantages of the proposed method over simple robust methods such as data filtering and label smoothing?\n\nQ3. Could the author consider other iterative method baselines for a fair comparison with Diff-UAPA?", "limitations": "The authors present a limitation section in Appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths.\n\n1.  The proposed method advances beyond traditional MLE objective in PbRL, which implicitly assumes a uniform prior and consequently lacks sensitivity to the inherent uncertainty in preference dataset, by using an informative beta prior and MAP objective.\n\n2. The paper provides comprehensive empirical validation, consistently demonstrating Diff-UAPA's superior performance and robustness across both simulated and real-world robotics tasks, and various human preference configurations including synthesized, realistic, and noisy data.\n\nWeaknesses.\n\n1. Training informative beta prior leads to additional computation costs.\n\n2. It seems that the baseline methods are not iteratively trained to adapt to evolving preference signals in the same manner as Diff-UAPA.", "was_revised": true, "final_justification": "The author's rebuttal has addressed all my questions and concerns. Considering the rebuttal, along with the soundness and completeness of the methodology and experiments, I keep my positive score.", "rebuttal": "Dear Reviewer, we greatly appreciate your constructive comments. We have seriously considered your suggestions, and we hope the following responses can address your concerns:\n\n---\n\n> **Q1.** Training informative beta prior leads to additional computation costs.\n\n**Response.** Thank you for raising this concern. As discussed in Appendix D.3, in this work, we use efficient techniques like the reparameterization trick to improve scalability. In practice, the computational cost of training the Beta model is similar to training a reward model in traditional PbRL. Since our method avoids training a reward model, the added cost is less effective compared to conventional PbRL. Additionally, the extra computational cost only slightly increases training time by a few minutes, while the subsequent RL phase is much more demanding, often taking several hours.\n\n---\n\n> **Q2.** It seems that the baseline methods are not iteratively trained to adapt to evolving preference signals in the same manner as Diff-UAPA.\n\n**Response.** Thank you for your insightful comment, and apologies for the confusion. As described in Line 260, our experiments consist of four rounds of iterative updates, each with a fixed number of episodes. This protocol is applied identically to all methods, including the baselines, so every algorithm is retrained on the newly acquired preference data at each round. We have clarified this procedure in the revised manuscript.\n\n---\n\n> **Q3.** Can we come up with the proposed method's online counterpart?\n\n**Response.** Thanks for your question. The proposed Diff-UAPA can be effectively generalized to an online setting due to the inherent flexibility of its core components, namely the Beta prior and MAP objective, which are not limited to offline datasets. The Beta prior’s ability to model uncertainty in human preferences makes it particularly suitable for handling dynamically evolving feedback from diverse user groups in real-time. To achieve this online counterpart, a feedback interface can be implemented to collect pairwise preference data (or from a pre-trained critic model) during or after trajectory execution in an interactive environment. This feedback can be stored in a dynamic preference replay buffer, which is incrementally updated as new trajectories are generated. The Beta prior parameters ($\\alpha$, $\\beta$) are then updated, enabling the diffusion policy to adapt continuously to new preferences while maintaining robustness to uncertainty.\n\n---\n\n> **Q4.** Could the author provide a discussion on the advantages and disadvantages of the proposed method over simple robust methods such as data filtering and label smoothing?\n\n**Response.** Thanks for your valuable question.  We would like to provide a more detailed discussion as follows:\n1. **Advantages:** UAPA offers significant advantages over data filtering and label smoothing. These Robust PbRL methods (e.g., label smoothing, data filtering) generally aim to exclude noisy or inconsistent data from the training process. While this may help reduce the impact of outliers, it also risks discarding valuable information if certain data points are mistakenly deemed as outliers. This filtering approach can result in lost opportunities for learning from diverse, potentially useful preferences. Besides, such techniques are usually static, lacking mechanisms to adjust to temporal or contextual changes in preference data. In contrast, UAPA employs a Beta prior to explicitly model the uncertainty inherent in diverse human preferences. Rather than eliminating uncertain data points, our framework assigns them lower confidence weights, allowing these samples to contribute conservatively to policy learning. This ensures that outliers or inconsistent preferences are not outright discarded but are instead integrated in a manner that minimizes their negative impact while preserving the richness of the preference dataset.  \n\n2. **Disadvantage:** Data filtering and label smoothing are computationally lightweight, as they involve simple rule-based preprocessing of the dataset. Training the Beta prior model in Diff-UAPA requires variational inference, which adds computational overhead compared to simpler methods. However, a small additional time cost (a few minutes) compared to the more demanding reinforcement learning phase is totally acceptable. (See more details in Appendix E)\n\nWe have included such a discussion in our revised paper to enhance clarity.\n\n---\n\n> **Q5.** Could the author consider other iterative method baselines for a fair comparison with Diff-UAPA?\n\n**Response.** Thank you for your question, and we apologize for any confusion. While none of our baseline methods include mechanisms to handle noisy preference labels, they all undergo four rounds of iterative updates during training, following the same procedure as Diff-UAPA (see Line 260). Therefore, these baseline methods are also iterative in nature.  \nTo better address your concern, we additionally included an iterative preference alignment method, IPO [1]. The results are as follows:\n\n| Method | Lift       | Can        | Square     | Transport  | p1(kitchen)| p2(kitchen)| p3(kitchen)| p4(kitchen)|\n|--------|------------|------------|------------|------------|------------|------------|------------|------------|\n| IPO   | 55.5 ± 1.4 | 54.0 ± 2.2 | 59.2 ± 5.3 | 56.7 ± 4.3 | 100.0 ± 0.0 | 97.4 ± 1.4 | 90.8 ± 2.8 | 61.5 ± 1.1 |\n| Diff-UAPA   | 56.1 ± 0.9 | 61.3 ± 2.2 | 68.1 ± 0.6 | 64.0 ± 4.0 | 100.0 ± 0.0 | 99.7 ± 0.2 | 95.4 ± 0.6 | 70.9 ± 4.6 |\n\nWe observe that Diff-UAPA consistently outperforms IPO, primarily due to its uncertainty-aware capabilities.\n\n[1] IPO: Iterative Preference Optimization for Text-to-Video Generation. arXiv, 2025.\n\n---"}, {"review_id": "7ArCub9pct", "reviewer": "Reviewer_jDLs", "summary": "This paper presents Diff-UAPA, an uncertainty-aware preference alignment algorithm for diffusion policies. The algorithm employs a maximum posterior objective with an informative Beta prior to align policies with regret-based preferences, enabling direct optimization without explicit reward functions while mitigating inconsistent preferences across groups. Extensive experiments on simulated and real-world robotics tasks demonstrate Diff-UAPA's effectiveness.", "questions": "1. In the introduction line 54, the authors mention \n> interpreting preference alignment as a voting process\n\nbut this concept is not elaborated upon or referenced again in the main text.\n\n2. It appears that UA-PbRL[1] and other uncertainty-aware preference alignment algorithms are not compared in the main experimental evaluation.\n\n3. While the paper frequently refers to iterative preference alignment, the model training appears to be offline, which seems contradictory.\n\n[1] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR 2025.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The method is reasonable, and the paper is clearly written.\n2. Diff-UAPA shows demonstrable effectiveness in simulated and real-world robotics tasks.\n\nWeaknesses:\n\n1.  This method builds upon several existing works [1,2,3], which may limit the novelty of the proposed approach. In my understanding, compared to [3], this paper primarily replaces the reward distribution with an advantage distribution and represents it using a Beta distribution (where the advantage derivation process is similar to [1]), while the optimization objective derivation for Diffusion Policy is largely based on [2]. The paper would benefit from clarifying the distinctions from these prior works to better highlight its contributions.\n\n2. The experimental evaluation mainly employs label flipping in preference datasets, which may not fully replicate the inherent uncertainties characteristic of human preferences.\n3. The necessity of the Beta distribution warrants further justification—whether directly predicting advantage distributions (as in [3]) or employing alternative distributions would be equally effective.  Moreover, the Beta distribution's unimodal nature may limit its ability to capture the diverse human preferences.\n\n[1 ]From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function. COLM 2024.\n\n[2] Diffusion model alignment using direct preference optimization. CVPR 2024.\n\n[3] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR 2025.", "was_revised": true, "final_justification": "While the author has resolved most concerns, the overall contribution remains at the same level. I maintain my original score.", "rebuttal": "Dear Reviewer, we sincerely value your time and effort in evaluating our work. We have prepared comprehensive responses and clarifications to address each point you raised. We hope these responses can resolve your concerns.\n\n---\n\n> **Q1.** This method builds upon several existing works ....\n\n**Response.** Thank you for raising this concern. We would like to provide a more detailed discussion highlighting the differences between this paper and the three prior works as follows.\n\n1. **Difference with [1].** Though both works start from the Maximum Entropy reinforcement learning, [1] uses the cumulative reward of the trajectory as the compare metrics, while ours adopt the advantage function.\n\n   - **Preference model.** In [1], the preference model employs the cumulative reward of a trajectory as the comparison metric, utilizing an **implicit reward model** defined as $r(s, a) = \\beta \\log \\frac{\\pi(s \\mid a)}{\\pi_{\\text{ref}}(s \\mid a)}$. Since our research is based on diffusion policies, we specifically define a chain advantage function for the diffusion policy to include all the latent variables in the middle. The marginalized form of the chain advantage function serves as the advantage function in our setting (Eq. 8). That means we need to consider the coupling of probability of the whole diffusion path rather than a state-action pair probability $\\pi(a \\mid s)$.\n\n2. **Difference with [2].** While we adopt some techniques from [2], such as Jensen’s inequality and the convexity of $-\\log(\\sigma)$ when deriving from Eq. 11 to Eq. 12 (we have accordingly cited [2] during the derivation), the problem setting, preference model, and derivation procedure are different. Additionally, we would like to highlight that the alignment of the diffusion policy via MLE is not our primary contribution. Instead, we primarily follow the approach developed for LLM in [2], and adapt it to the RL setting in our work.\n\n   - **Preference model and derivation procedure.** [2] employs a **reward-based preference** model, which relies on the reward of the LLM's final output (see Eq. 3 in [2]) while regularizing the **KL-divergence** with respect to a reference policy. In contrast, this work adopts a **regret (advantage)-based preference** model (see Eq. 5) within the **maximum entropy RL** framework. Although both preference models ultimately lead to similar objectives for aligning the diffusion policy, the derivation procedures are somewhat different. Specifically, the derivation of this work is based on trajectory-wise alignment, whereas [2] only addresses the step-wise alignment. To accomplish this using the advantage-based preference model, we introduce the **chain advantage function** (i.e., Eq. 8), which is defined based on the chain reward function presented in [2]. We then calculate the expected value of the chain advantage function with respect to the diffusion latent variable, which serves as a key index for evaluating the trajectory.\n\n3. **Difference with [3].** While both works utilize the Beta prior with a MAP objective to model uncertainty during the alignment process, this work differs significantly regarding the problem formulation, underlying motivation, and approach to incorporating the Beta prior. Specifically,\n\n   - **Implementation.** Specifically, [3] employs an iterative update rule, which is an approximation approach from numerical analysis (see Eq. 8 in [3]). This approach implicitly assumes an average reward of zero and allows for substantial fluctuation in the Beta prior parameters ($\\alpha$, $\\beta$). In contrast, our framework integrates the Beta prior directly into a trainable loss function for the target policy. We utilize a marginalized chain advantage function ($A(s,\\overline{a})$) as our comparison metric, which is computed by $\\mathbb{E}_{\\pi\\theta^*(a_t^{1:I}|a_t^0,s_t)}[A(s_t,\\overline{a_t})]$. This distinction means the assumption of a zero-mean reward function is not applicable to our method. Furthermore, we have to carefully control the numerical magnitudes of $\\alpha$ and $\\beta$ to appropriately regularize the influence of the prior loss during training. These fundamental differences in implementation highlight a notable gap between the two approaches.\n\n   - **Approach.** [3] adopts a **two-step** procedure, proposing a MAP objective for learning a **distributional reward model**. To achieve this, [3] introduces an **iterative update rule** that refines the reward model using the $\\alpha$ and $\\beta$ parameters output by the learned Beta prior model, until it converges to the maximum of its MAP objective. This reward model is then used for policy learning. In contrast, this work derives a unified MAP objective for directly aligning the **diffusion policy** in a **single-step** process, without the need for reward learning. We update our diffusion policy using a unified loss function (Eq. 17). By maximizing the likelihood of the diffusion policy's output under the learned Beta distribution, the process guides the policy to align the estimated $\\phi(\\tau)$ with their prior distribution $p_0(\\phi(\\tau))$. This approach is more straightforward and efficient.\n\n[1] From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function. COLM 2024.  \n[2] Diffusion model alignment using direct preference optimization. CVPR 2024.  \n[3] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR 2025.\n\n---\n\n> **Q2.** The experimental evaluation ....\n\n**Response.** We appreciate your observation regarding the use of label flipping in our experimental evaluation of preference datasets. To address the inherent uncertainties characteristic of human preferences, we incorporated multiple noise models to simulate inconsistent human behavior. Specifically, our evaluation, as detailed in [1], includes the Stochastic Noise Model, the Myopic Noise Model, and the Irrational Noise Model, with **the latter being equivalent to label flipping**. The experiment results are provided in Table 4. We believe these diverse models collectively provide a robust framework for capturing the complexities and uncertainties inherent in human preferences.\n\n[1] Robust preference-based reinforcement learning with noisy preferences. ICML 2024.\n\n---\n\n> **Q3.** The necessity of the Beta distribution ... diverse human preferences.\n\n**Response.** Thank you for your question. We would like to have a deeper discussion on Beta Distribution.  \n- **Effectiveness of Beta Distribution.** The Beta distribution proves highly effective for modeling human preferences in our framework due to its intuitive representation of **confidence and uncertainty**. It is the **conjugate prior** for the Bernoulli distribution on $\\phi(\\tau)$. Its two parameters, $\\alpha$ and $\\beta$, directly reflect accumulated \"positive\" and \"negative\" votes (e.g., preferred vs. unpreferred trajectory instances), allowing for a **flexible update** of beliefs as new preference data arrives (Lines 219–220). Other distributions are not naturally bounded on [0, 1] nor built for voting, which limits their suitability for proportion-based data. The Normal distribution’s unbounded support and symmetric shape, the Uniform’s fixed variance, and the Gamma’s semi-infinite range hinder their flexibility for diverse preference patterns.  \n\n- **Unimodality of Beta Distribution.** Since our study adheres to the standard setting of PbRL where preferences for a single sample are binary—either positive or negative, the aggregated win-probabilities remain effectively unimodal and are well-fit by a single Beta. For example, they either concentrate near 0 or 1 when consensus is strong, or near 0.5 when feedback is mixed. We acknowledge that truly multi-modal or list-wise preference patterns may necessitate more expressive priors such as mixtures of Beta distributions, which is a promising direction for future work.\n\n---\n\n> **Q4.** In the introduction Line 54 ... in the main text.\n\n**Response.** We apologize for the missing explanation. This process is very similar to choosing one of two candidates. Human annotators act as voters, casting preferences for trajectories (candidates) through pairwise comparisons in an election. The Diff-UAPA framework aggregates these \"votes\" using a MAP objective, incorporating a Beta prior to model uncertainty arising from diverse or conflicting preferences, akin to handling voter disagreements in a probabilistic voting system.\n\n---\n\n> **Q5.** It appears that UA-PbRL ....\n\n**Response.** Thanks for your question. In this paper, we compare UA-PbRL in our in-depth study (i.e., Table 4), evaluating its performance in MuJoCo environments under different types of noise. However, we acknowledge the importance of testing them in the main experimental evaluation, the results are:\n\n| Method |Lift| Can|Square|Transport|p1 (kitchen)| p2 (kitchen) | p3 (kitchen) | p4 (kitchen) |\n|-|-|-|-|-|-|-|-|-|\n| UA-PbRL | 55.3 ± 2.2 | 57.4 ± 2.3 | 62.3 ± 2.7| 60.6 ± 5.7 | 100.0 ± 0.0  | 98.7 ± 1.3   | 92.2 ± 3.7   | 66.3 ± 4.2   |\n| Diff-UAPA   | 56.1 ± 0.9 | 61.3 ± 2.2 | 68.1 ± 0.6 | 64.0 ± 4.0 | 100.0 ± 0.0 | 99.7 ± 0.2 | 95.4 ± 0.6 | 70.9 ± 4.6 |\n\nThis demonstrates that Diff-UAPA outperforms UA-PbRL, primarily owing to the powerful modeling capabilities of the diffusion policy.\n\n---\n\n> **Q6.** While the paper ... which seems contradictory.\n\n**Response.** Thank you for raising this point. The iterative preference alignment process in our framework is distinct from the offline or online RL setting. As outlined in Lines 126–134, iterative preference alignment refers to the method by which the preference dataset is constructed, labeled from various people, and utilized during training. This process does not specify the data collection mechanism, such as whether the agent interacts with the environment or relies on a pre-collected dataset. Thus, it is fully compatible with the offline training approach described in the paper, resolving any apparent contradiction."}, {"review_id": "sXnpqP0yo0", "reviewer": "Reviewer_UdRw", "summary": "The paper proposes Diff-UAPA, an uncertainty-aware framework for aligning diffusion policies with human preferences using a Beta prior to model preference inconsistencies. The approach integrates a maximum posterior objective with iterative preference alignment, demonstrating robust performance across simulated and real-world robotics tasks. Extensive experiments validate its effectiveness against baselines, including ablation studies and noise sensitivity tests. However, a critical question arises regarding the originality of the core contributions, as key components appear to build heavily on prior works.", "questions": "See weakness.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written and clearly organized, with a logical flow from problem statement to experimental validation.\n\n- Important Motivation: The focus on handling uncertainty in preference alignment is timely and relevant, as human preferences from diverse groups inherently contain inconsistencies. The work addresses a critical gap in existing PbRL methods that overlook such uncertainties.\n\n- The empirical evaluation is robust, covering multiple robotic tasks, real human preference datasets, and various noise conditions. The comparison with state-of-the-art baselines (e.g., BET, Diff-CPL, FKPD) and ablation studies (e.g., Beta prior ablation) effectively demonstrate the method’s superiority and validate the contribution of each component.\n\n**Weakness**:\n\n- The paper’s novelty is unclear, as the two key components seem to rely heavily on prior works:\nThe diffusion policy alignment framework in Section 4.1 builds directly on Wallace et al. (2024), which introduces Direct Preference Optimization (DPO) for text-to-image diffusion models.\nThe Beta prior model in Section 4.2 is inspired by Xu et al. (2025), which uses a Beta distribution for uncertainty-aware preference alignment in offline settings.\nWhile the combination of these components is novel, the paper fails to sufficiently demonstrate its unique innovations tailored to the problem at hand, weaken its claim to breakthrough innovation.\n\n- The theoretical contribution is primarily a reapplication of Beta priors (a well-known conjugate prior for Bernoulli distributions) to the diffusion policy context, without substantial new theoretical insights. The proof in Proposition 4.1, while correct, formalizes a standard Bayesian modeling approach rather than introducing a novel framework.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns, and thus I maintain my positive evaluation score.", "rebuttal": "Dear Reviewer, we sincerely value your time and effort in evaluating our work. We have prepared comprehensive responses and clarifications to address each point you raised. We hope these responses can resolve your concerns.\n\n---\n\n> **Q1.** The paper’s novelty is unclear, as the two key components seem to rely heavily on prior works: The diffusion policy alignment framework in Section 4.1 builds directly on Wallace et al. (2024), which introduces Direct Preference Optimization (DPO) for text-to-image diffusion models. The Beta prior model in Section 4.2 is inspired by Xu et al. (2025), which uses a Beta distribution for uncertainty-aware preference alignment in offline settings. While the combination of these components is novel, the paper fails to sufficiently demonstrate its unique innovations tailored to the problem at hand, weakening its claim to breakthrough innovation.\n\n**Response.** Thank you for raising this concern. We would like to provide a more detailed discussion highlighting the differences between this paper and the two prior works as follows.\n\n1. **Difference with [1].** While we adopt some techniques from [1], such as Jensen’s inequality and the convexity of $-\\log(\\sigma)$ when deriving from Eq. 11 to Eq. 12 (we have accordingly cited [1] during the derivation), the problem setting, preference model, and derivation procedure are different. Additionally, we would like to highlight that the alignment of the diffusion policy via MLE is not our primary contribution. Instead, we primarily follow the approach developed for LLM in [1], and adapt it to the RL setting in our work. Specifically,\n\n   - **Problem setting.** [1] is formulated in the context of **LLM alignment**, where rewards are assigned exclusively at the final step, and preferences are based solely on the **final output** of the LLM. This approach optimizes the LLM’s ultimate output, without considering intermediate steps or the overall process leading to the final result (see Eq. 14 in [1]). In contrast, Eq. 12 in this paper extends the framework to a **trajectory-wise** setting within the **RL field**. The key distinction is that in RL, we incorporate intermediate rewards, adjusted by a discount factor $\\gamma$, which accounts for the cumulative discounted rewards throughout the trajectory. As a result, this paper formulates preference alignment based on the entire trajectory rather than focusing solely on the final state-action pair.\n\n   - **Preference model and derivation procedure.** [1] employs a **reward-based preference** model, which relies on the reward of the LLM's final output (see Eq. 3 in [1]) while regularizing the **KL-divergence** with respect to a reference policy. In contrast, this work adopts a **regret (advantage)-based preference** model (see Eq. 5) within the **maximum entropy RL** framework. Although both preference models ultimately lead to similar objectives for aligning the diffusion policy, the derivation procedures are somewhat different. Specifically, the derivation of this work is based on trajectory-wise alignment, whereas [1] only addresses the step-wise alignment. To accomplish this using the regret (advantage)-based preference model, we introduce the **chain advantage function** (i.e., Eq. 8), which is defined based on the chain reward function presented in [1]. We then calculate the expected value of the chain advantage function with respect to the diffusion latent variable, which serves as a key index for evaluating the trajectory.\n\n2. **Difference with [2].** While both works utilize the Beta prior with a MAP objective to model uncertainty during the alignment process, this work differs significantly regarding the problem formulation, underlying motivation, and approach to incorporating the Beta prior. Specifically,\n\n   - **Problem formulation and motivation.** In [2], the authors introduced an uncertainty-aware PbRL framework that specifically addresses **epistemic uncertainty** arising from an **offline preference dataset** with imbalanced comparison frequencies across different trajectories. In their setting, the fewer compared trajectories introduce greater epistemic uncertainty in reward prediction, due to the insufficient data available for these samples. Conversely, our work targets the **aleatoric uncertainty** associated with human preferences in an **iterative updating dataset** as defined in Definition 3.1. In this setting, different groups of human annotators may provide inconsistent or even conflicting preferences for the same pair of trajectories, which causes the aleatoric uncertainty that represents inherent stochasticity. In other words, by interpreting the parameters $\\alpha$ and $\\beta$ of the Beta distribution as representing the counts of 'vote' and 'unvote' human feedback, [1] uses a Beta prior to model the difference in the absolute values of $\\alpha$ and $\\beta$ across different trajectories (e.g., Beta(10,2) vs. Beta(100,20), where the former exhibits greater uncertainty due to fewer counts). In contrast, this work employs the Beta prior to model the relative strength of $\\alpha$ and $\\beta$ for different $\\tau$ (e.g., Beta(6,6) vs. Beta(10,2), where the former exhibits greater uncertainty due to inconsistency of votes). (Part 1 of Section C)\n\n   - **Approach.** [2] adopts a **two-step** procedure, proposing a MAP objective for learning a **distributional reward model**. To achieve this, [2] introduces an **iterative update rule** that refines the reward model using the $\\alpha$ and $\\beta$ parameters output by the learned Beta prior model, until it converges to the maximum of its MAP objective. This reward model is then used for policy learning. In contrast, this work derives a unified MAP objective for directly aligning the **diffusion policy** in a **single-step** process, without the need for reward learning. We update our diffusion policy using a unified loss function (Eq. 17). By maximizing the likelihood of the diffusion policy's output under the learned Beta distribution, the process guides the policy to align the estimated $\\phi(\\tau)$ with their prior distribution $p_0(\\phi(\\tau))$. This approach is more straightforward and efficient.\n\n[1] Diffusion model alignment using direct preference optimization. CVPR, 2024.  \n[2] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR, 2025.\n\n---\n\n> **Q2.** The theoretical contribution is primarily a reapplication of Beta priors (a well-known conjugate prior for Bernoulli distributions) to the diffusion policy context, without substantial new theoretical insights. The proof in Proposition 4.1, while correct, formalizes a standard Bayesian modeling approach rather than introducing a novel framework.\n\n**Response.** Thank you for your insightful feedback. While both approaches indeed leverage the Beta prior within Bayesian framework, there are distinctions in how policies and Beta prior are updated.\n\n- **Iterative Alignment for Diffusion.** Diff-UAPA introduces a novel iterative preference alignment framework tailored specifically for diffusion policies (Definition 3.1). Unlike traditional static offline preference alignment methods, our framework incrementally adapts the diffusion policy to preferences from diverse user groups, leveraging the sequential nature of diffusion models. This iterative process, supported by the Beta prior, enables dynamic updates that account for evolving preferences over time (Lines 284–287). In contrast, the approach in [1] operates in a static offline setting, lacking mechanisms for iterative adaptation to new preference data.\n\n- **Dynamic Adaptation of Beta Prior.** Our proposed variant, Diff-UAPA-I, further distinguishes itself by enabling incremental training of the Beta prior, allowing it to adapt dynamically to changing preferences and environmental conditions. This dynamic adaptation is theoretically grounded in our iterative alignment framework, which ensures robust handling of evolving uncertainties in preference data. In contrast, the Beta prior in [1] is learned solely from offline demonstrations and does not incorporate mechanisms to adapt to new or evolving preferences, limiting its flexibility in dynamic settings.\n\n[1] A distributional approach to uncertainty-aware preference alignment using offline demonstrations. ICLR, 2025.\n\n---"}, {"review_id": "BeeKX6XOQY", "reviewer": "Reviewer_DPLP", "summary": "This paper addresses the challenge of sample efficiency and robustness in preference-based reinforcement learning (PbRL), where the learning agent relies on human preference feedback instead of explicit numerical rewards. The authors propose an uncertainty-aware PbRL framework that explicitly models the uncertainty in human feedback and integrates it into both the preference prediction and policy optimization processes. Specifically, the method employs a Bayesian preference model to capture epistemic uncertainty and leverages this information to guide more informative queries and safer policy updates. The paper demonstrates the proposed approach on several benchmark continuous control tasks, showing that the method achieves improved sample efficiency and robustness compared to existing PbRL baselines. The key contributions include the integration of uncertainty estimation into the preference model, the design of an active query strategy based on uncertainty, and an uncertainty-aware policy optimization algorithm.", "questions": "1. In Equation 11, multiple layers of approximation are introduced. Could the accumulation of these approximations affect the final performance?\n2. Proposition 4.1 assumes that trajectories are i.i.d, but in practice, trajectories may be dependent. How can you ensure that this is maintained in MDP?\n3. In Algorithm 1, how is the reference policy $\\pi_{ref}$ initialized? How sensitive is the final performance to the choice of $\\pi_{ref}$?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- This paper accurately identifies a crucial but often overlooked issue in PbRL: the uncertainty of preference data. This is an important topic in current alignment research. The authors propose using a Beta distribution as a prior for trajectory advantage values, combined with regret-based preference modeling and MAP inference, enabling the model to capture preference uncertainty explicitly.\n\n- The paper introduces Diff-UAPA, which is designed for both streaming and batch preference inputs, demonstrating broad applicability.\n\n- The authors tightly integrate the preference alignment problem into the diffusion policy framework, proposing chain reward and advantage modeling based on diffusion trajectories, which addresses limitations faced by DPO/CPL methods in this setting.\n\n- The experimental validation covers diverse scenarios, including synthetic tasks, real-world preference datasets, and real robot control tasks. Moreover, the authors conduct sensitivity analyses under various noise levels and types, along with ablation studies, to demonstrate the method’s advantages in dynamic preference iteration and uncertain data conditions.\n\n------\n\n**Weaknesses:**\n\n- Although the Beta distribution can theoretically represent uncertainty in the strength of preferences, its effectiveness under complex preference distributions (such as asymmetric or multi-modal cases) remains questionable. Additionally, the paper initializes the Beta parameters $\\alpha$ and $\\beta$ to 1. How would varying the initialization affect the model’s performance under different scenarios (e.g., the various noise conditions shown in Table 5)?\n\n- Equation 8 and Equation 17 introduce additional computational overhead. How does this overhead compare to previous methods? It would be helpful if the authors could provide a more detailed comparison of the computational cost relative to prior approaches.\n\n- The comparative experiments only benchmark against diffusion-based methods. Why not compare with non-diffusion baselines (such as INPO, IPO, or Nash-MD-PG)?\n\n**Others:**\n\n- Line 106: “datatset” → “dataset”", "was_revised": true, "final_justification": "The authors' rebuttal has addressed most of my concerns. Based on the current state of the paper and the authors’ clarifications, I recommend the paper is marginally acceptable.", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate the time and effort you have dedicated to reviewing our work. We have carefully addressed each of your comments with detailed responses and clarifications. We hope these will effectively resolve your concerns.\n\n> **Q1.** Although the Beta distribution can theoretically represent uncertainty in the strength of preferences, its effectiveness under complex preference distributions (such as asymmetric or multi-modal cases) remains questionable.\n\n**Response.** We thank the reviewer for raising this important point. To clarify, we do not use the Beta distribution to model **preference strength** directly. Instead, as detailed in Lines 213–217, we use a Beta distribution to serve as a prior on the **win‑probability** $\\phi(\\tau)=sigmoid(A^\\pi(\\tau)-\\bar{A}^\\pi(\\tau))$, i.e., the probability that trajectory $\\tau$ wins against the average candidate. Since human pairwise feedback is a binary comparison, which is inherently Bernoulli (win vs. loss). So the Beta distribution serves as a natural, conjugate choice: its two parameters flexibly capture both skewed (peaked near 0 or 1) and broad (centered near 0.5) probability without too much complexity. Empirically, even under asymmetric or multi-modal cases, the aggregated win-probabilities remain effectively unimodal and are well-fit by a single Beta. For example, they either concentrated near 0 or 1 when consensus is strong. We acknowledge that truly multi‑modal preference patterns may necessitate more expressive priors such as mixtures of Beta distributions, which is a promising direction for future work.\n\n---\n\n> **Q2.** Additionally, the paper initializes the Beta parameters $\\alpha$ and $\\beta$ to 1. How would varying the initialization affect the model’s performance under different scenarios (e.g., the various noise conditions shown in Table 5)?\n\n**Response.** Thank you for raising this question. In this work, we utilize $\\alpha_0 = \\beta_0 = 1$ as **an uninformed prior**. This corresponds to a uniform distribution over $[0, 1]$, which serves as a natural choice for uninformed belief. This choice provides **a neutral starting point with the least information**.\n\nIn terms of varying $\\alpha_0$ and $\\beta_0$ to other values, this operation indeed has an impact on the model performance. For instance, setting $\\alpha_0 = 9999$ and $\\beta_0 = 1$ will introduce strong prior information into the model (implying 9999 persons vote 1, and only one person votes 0).\nMoreover, the absolute values of $\\alpha_0$ and $\\beta_0$ critically affect prior strength: although both $\\text{Beta}(1,1)$ and $\\text{Beta}(10000,10000)$ have a mean of 0.5 (assuming every trajectory is an average player), the latter encodes a far stronger initial belief in such a “half-win” probability. Such a concentrated prior can dominate the posterior and hinder adaptation to true preference signals. Therefore, we choose the uniform distribution to initialize $\\alpha_0 = \\beta_0 = 1$ due to the above reasons.\n\n---\n\n> **Q3.** Equation 8 and Equation 17 introduce additional computational overhead. How does this overhead compare to previous methods? It would be helpful if the authors could provide a more detailed comparison of the computational cost relative to prior approaches.\n\n**Response.**\nThank you for your thoughtful comment. We address the computational overhead introduced by the diffusion policy (Eq. 8) and the Beta model (Eq. 17) in Appendix D.3.\nTo summarize, training the diffusion policy takes approximately twice as long as our Transformer baseline. However, this overhead is partially mitigated by the action-sequence prediction strategy adopted from [1], which improves sample efficiency. As for the Beta prior, it is learned using efficient reparameterization techniques, incurring a cost comparable to fitting a standard reward model. In practice, this adds only a few additional minutes to the overall training time.\n\n[1] Diffusion policy: Visuomotor policy learning via action diffusion. *International Journal of Robotics Research*, 2023.\n\n---\n\n> **Q4.** The comparative experiments only benchmark against diffusion-based methods. Why not compare with non-diffusion baselines (such as INPO, IPO, or Nash-MD-PG)?\n\n**Response.** Thank you for the comment. We would like to clarify that our evaluation does include non-diffusion baselines. Specifically, as shown in Lines 265–270 and Table 1, there is a line of baselines based on the Behavior Transformer (BET), which is a representative non-diffusion policy optimization method. We assume that the \"non-diffusion\" approaches you mention (INPO, IPO, Nash-MD-PG) correspond to \"non-DPO\" methods. Indeed, our experiments also include non-DPO methods like UA-PbRL and RIME (see Lines 336–340 and Table 4).\n\nTo solve your concerns, we conduct additional experiments using the baselines you mentioned.\n\n| Method | Lift        | Can         | Square      | Transport   | p1 (kitchen) | p2 (kitchen) | p3 (kitchen) | p4 (kitchen) |\n|--------|-------------|-------------|-------------|-------------|--------------|--------------|--------------|--------------|\n| InPO   | 53.2 ± 1.4  | 54.2 ± 2.2  | 57.2 ± 5.3  | 50.4 ± 4.3  | 100.0 ± 0.0  | 98.2 ± 1.4   | 90.1 ± 2.8   | 52.2 ± 1.1   |\n| IPO    | 50.7 ± 0.9  | 53.0 ± 4.3  | 54.7 ± 5.8  | 53.0 ± 12.1 | 99.3 ± 0.4   | 99.3 ± 0.4   | 92.3 ± 0.0   | 55.3 ± 1.4   |\n| Diff-UAPA   | 56.1 ± 0.9 | 61.3 ± 2.2 | 68.1 ± 0.6 | 64.0 ± 4.0 | 100.0 ± 0.0 | 99.7 ± 0.2 | 95.4 ± 0.6 | 70.9 ± 4.6 |\n\nThe results highlight the uncertainty-aware capability of Diff-UAPA. In contrast, InPO and IPO perform worse due to the absence of mechanisms for handling uncertainty in noisy labels.\n\n---\n\n> **Q5.** In Equation 11, multiple layers of approximation are introduced. Could the accumulation of these approximations affect the final performance?\n\n**Response.** Thank you for your valuable question. We assume that the 'approximation' you mentioned indicates the 'inequality' and 'expectation' in Eq. 11. We have discussed some reasons in Lines 193–197, and we have offered a more detailed justification of the necessity and rationale behind our model in Lines 193–197.\n\nTo clarify, approximating expectation via sampling is a common practice. When it comes to Jensen's inequality:\n\n- Jensen’s inequality gives an upper bound on the negative log-sigmoid, where the error does not grow infinitely with variance. So training the loss function of Eq. 11 yields similar results to Eq. 10.\n\n- Evaluating Eq. 10 directly means computing probabilities at every diffusion step, which is an expensive process where small errors can be amplified by the outer sigmoid. By applying Jensen’s inequality to the log-expectation, we obtain a simpler surrogate loss that is both faster to compute and more stable.\n\nIt is also worth noting that the derivation from Eq. 10 to Eq. 11 follows the standard practice for MLE-based diffusion policy alignment. Similarly, [2] applies an analogous approximation when moving from Eq. 13 to Eq. 14.\n\n[2] Diffusion model alignment using direct preference optimization. *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024.\n\n---\n\n> **Q6.** Proposition 4.1 assumes that trajectories are i.i.d, but in practice, trajectories may be dependent. How can you ensure that this is maintained in MDP?\n\n**Response.** We appreciate the reviewer’s question regarding Proposition 4.1. In fact, Proposition 4.1 rests on the classic Beta–Bernoulli conjugacy, which formally assumes that the $k$ wins out of $n$ comparisons behave like conditionally independent Bernoulli trials given a latent win-probability $\\phi$. Crucially, one only needs the weaker condition of **exchangeability**, and De Finetti’s theorem guarantees that any exchangeable Bernoulli sequence can be viewed as i.i.d. draws from some $\\phi \\sim \\text{Beta}(\\alpha_0, \\beta_0)$, yielding the posterior $\\phi \\mid k, n \\sim \\text{Beta}(\\alpha_0 + k, \\beta_0 + n - k)$. In episodic RL, independent environment resets, a stationary policy during data collection, and randomized initial states ensure that the sequence of win/loss labels across trajectories is effectively exchangeable (and hence conditionally independent given $\\phi$), even though state–action pairs within each trajectory remain correlated by the MDP dynamics. Therefore, Proposition 4.1 does not require trajectories to be strictly i.i.d., and it only requires the exchangeability of the binary preference outcomes, a condition naturally met by standard episodic rollout protocols.\n\n---\n\n> **Q7.** In Algorithm 1, how is the reference policy $、pi_{\\text{ref}}$ initialized? How sensitive is the final performance to the choice of $\\pi_{\\text{ref}}$?\n\n**Response.** Thanks for your valuable question. $\\pi_{\\text{ref}}$ is first randomly initialized and then trained via standard behavior cloning on the entire set of state-action pairs extracted from the preference dataset $D_{pref}$, until it reaches a predefined performance threshold (e.g., a target cumulative reward). To solve your concerns, in terms of the sensitivity of the performance $\\pi_{\\text{ref}}$, we conduct additional experiments by varying the performance of $\\pi_{\\text{ref}}$ in the Robomimic Lift environment.\n\n| Reference Policy | BET-CPL | BET-DPO | Diff-CPL | FKPD | Diff-UAPA-C | Diff-UAPA-I |\n|------------------|---------|---------|----------|------|-------------|-------------|\n| 25%              | 40.7 ± 4.2 | 37.3 ± 5.0 | 40.6 ± 5.7 | 42.0 ± 2.0 | 44.7 ± 4.1 | 41.7 ± 2.3 |\n| 75%              | 79.0 ± 7.8 | 76.3 ± 1.3 | 80.0 ± 2.0 | 82.3 ± 1.3 | 84.0 ± 3.5 | 82.7 ± 3.1 |\n\nFrom the experimental results, it can be seen that the quality of the initial policy has an impact on the final result. But the Diff-UAPA still outperforms other baseline methods.\n\n---\n\n> **Q8.** Some typos.\n\n**Response.** Thank you for your valuable suggestions. We have corrected them accordingly and thoroughly checked the paper to avoid similar issues."}]

Author rebuttal:

Meta-review: This paper studies diffusion model using the human preference data. The goal is to tackle the uncertainty in the human preference data. The key idea is to learn a beta distribution to capture the uncertainty and then incorporate it to learn the policy. All reviewers have acknowledged the contributions from this paper. The rebuttal also addressed all concerns raised in the first round. Overall, the paper has its unique perspective and made contributions to advance diffusion model. I will recommend an acceptance.
Decision: Accept (poster)