A Distributional Approach to Uncertainty-Aware Preference Alignment Using Offline Demonstrations
Authors: Sheng Xu
Bo Yue
Hongyuan Zha
Guiliang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Preference-based Reinforcement Learning
Distributional Reinforcement Learning
Uncertainty Awareness
Primary area: reinforcement learning
TL;DR: We introduce an uncertainty-aware preference alignment approach to PbRL by learning a distributional reward model and a risk-sensitive policy using the offline preference dataset.

Abstract: Designing reward functions in Reinforcement Learning (RL) often demands significant task-specific expertise. Offline Preference-based Reinforcement Learning (PbRL) provides an effective alternative to address the complexity of reward design by learning policies from offline datasets that contain human preferences between trajectory pairs. Existing offline PbRL studies typically model a reward function by maximizing its likelihood of generating the observed human preferences. However, due to the varying number of samples within the limited dataset, less frequently compared trajectories exhibit greater uncertainty, which potentially leads to unreliable behaviors during reward and policy updates. To solve this issue, in this work, we introduce Uncertainty-Aware PbRL (UA-PbRL) to learn a distributional reward model and a risk-sensitive policy from an offline preference dataset. Our approach employs a Maximum A Posteriori (MAP) objective to update trajectory rewards and incorporates an informative prior to account for the uncertainties. Building upon this reward update, we propose a generative reward model to capture the reward distribution, utilizing the offline distributional Bellman operator and the Conditional Value-at-Risk (CVaR) metric to train a risk-sensitive policy. Experimental results demonstrate that UA-PbRL effectively identifies and avoids states with high uncertainty, facilitating risk-averse behaviors across various tasks, including robot control and language model alignment. The code is available at https://github.com/Jasonxu1225/UA-PbRL.

Published as a conference paper at ICLR 2025
A DISTRIBUTIONAL APPROACH TO UNCERTAINTY- AWARE PREFERENCE ALIGNMENT USING OFFLINE DEMONSTRATIONS
Sheng Xu', Bo Yue', Hongyuan Zha', Guiliang Liu School of Data Science, The Chinese University of Hong Kong, Shenzhen (shengxul, boyue @ likk. cuhk. edu. cn 1zhahy, liuguiliang bouhk.edu.co.
ABSTRACT
Designing reward functions in Reinforcement Learning (RL) often demands sig- nificant task-specific expertise. Offline Preference-based Reinforcement Learning (PHRL. provides an effective alternative to address the complexity of reward design by learning policies from offline datasets that contain human preferences between trajectory pairs. Existing offline PbRL studies typically model a reward function by maximizing its likelihood of generating the observed human preferences. However, due to the varying number of samples within the limited dataset, less frequently compared trajectories exhibit greater uncertainty, which potentially leads to un- reliable behaviors during reward and policy updates. To solve this issue, in this work, we introduce Uncertainty-Awe Aware PbRL (UA-PbRL) learn distributional reward model and a risk-sensitive policy from an offline preference dataset. Our approach employs a Maximum A Posteriori (MAP) objective to update trajectory rewards and incorporates an informative prior to account for the uncertainties. Building upon this reward update, we propose a generative reward model to cap- ture the reward distribution, utilizing the offline distributional Bellman operator and the Conditional Value-at-Risk (CVaR) metric to train a risk-sensitive policy. Experimental results demonstrate that UA-PbRL effectively identifies and avoids states with high uncertainty, facilitating fiskreeerse behaviors across various tasks, including robot control and language model alignment. The code is available https: / thub. com Jasonxuuooouuoooooooooo 225 2
INTRODUCTION
In recent years, Reinforcement Learning has achieved remarkable success in addressing a variety of sequential decision problems across different domains, such as electronic games (Mnih et al. 2015 Vinyals al. 2019 Berner et al. 2019), board games (Silver al. 2016 2017), and robotic manipulation (Fang et al. 2019). However, in the process of scaling these successes to real-world applications, a notable challenge is the difficulty precisely specifying the rewards in the RL objective. Existing manually designed rewards reward-enging wwwarddengineering techniques cannot ensure that the learned policy aligns with the actual needs of the industry.
To develop reliable reward function, Preference- bassed Reinforcement Learning (PbRL) (Knox & Stone, 2008 Christiano et al. 2017) aims to align rewards with human preferences by solving a learning-to-ran problem. In this framework, agents do not receive numerical reward signals, but are instead provided with preferences between pairs of trajectory segments, reflecting Hamman-labeled relative judgments. PbRL has garnered considerable attention in both online (Sadigh et al.) 2017] Ibarz et al. 2018: Liang et al. 2022; Park et al. 2022 Cheng et al.) (024)), and offline (Shin et al. 2021 et al. 2023 Zhang et al. [0224c] Choi al. Zhan et al. | 2024 settings. In the online setting, agents can interact with environments and query human experts, whereas the offline setting prohibit such interactions. In this work, we focus on the offline PbRL setting, where the objective is to learn an optimal policy solely from pre-collected offline demonstrations, with preference feedback on piir-wise trajectory segments, without any active interaction with the environment or an expert. This
"Corresooniigg author: Guiliang Liu, liuguiliang www.weeeu.uc... edu.en.

---

Published as a conference paper at ICLR 2025
approach is particularly promising in scenarios where collecting online trajectories and obtaining human feedback costly and challenging (Lee et al. 2021 Shin et al. (023).
In practice, the frequency of comparisons across different samples within the offline dataset is often highly imbalanced. While some samples may emerge with a higher ranking, they have only been compared limited number of times. These sparsely compared trajectories introduce significant epistemic uncertainty, which stems from a lack of sufficient data (Smith & Gal 2018). This uncertainty compromises the reliability of the evaluation, reducing the model's confidence in accurately assessing the value of such pairs. To ensure more reliable preference alignment, it is essential to account for the uncertainty during the alignment process. However, the Bradley- Terry model (Bradley & Terry) 1952). as a common approach to PbRL, is based on maximum likelihood estimation and lacks sensitivity to inherent uncertainties from preference datasets. Consequently, the resulting policy tends to be risk-neutral, disregarding potential safety concerns embedded in less examined seccisionmmakinn scenarios.
Striving for uncertainty-awat preference alignment, several studies (Liang et al. 2022 Shin et al. 2023 Xue et al. 2023) used ensembles and Bayesian dropout to estimate the uncertainty of reward model. However, despite their empirical success, the underlying mechanism by which the estimated ensembles correlate with uncertainty in human preferences remains largely unexplained. More importantly, these approaches still rely maximum likelihood estimation with risk-neutral policy.
In this paper, we introduce an Uncertainty the uncertainty in human preferences within offline demonstrations. Specifically, formulate Maximum Posteriori (MAP) objective for preference aligmment, thereby incorporating signals from an informative prior to the reward updates. By interpreting preference alignment as a voting process, we intentionally select Beta distribution to implement this prior. To ensure computational tractability, we parameterize the Beta distribution with neural functions and train the model via variational inference, guided by an Evidence Lower Bound (ELBo) objective. Intuitively, this prior assigns higher probabilities to trajectories that are compared more frequently, reflecting the confidence levels associated with them. For each observed trajectory, we compute its reward using an iterative update rule, derived from the optimality conditions of our MAP objective. The reward distribution is then constructed by training generative model conditioning sequential state-action pairs.
PbRL (UA-PbRL) algorithm that accounts for
To justify the effectiveness of the learned reward distribution, we train a risk-averse policy using the offline distributional Bellman operator for policy evaluation, coupled with the Conditional Value- at- Risk (CVaR) metric for policy improvement. The empirical results highlight the importance of accounting for the inherent uncertainty arising from preference demonstrations, as well as the risk- sensitive capabilities of the proposed method. In particular, UA-PbRL outperforms other baselines in pointmaze environments and robot control tasks, especially in terms of worst-case performance. Furthermore, we extended our experiments to the context of Large Language Model (LL.) alignment, demonstrating the effectiveness of UA-PbRL for LLM fine-tuning.
RELATED WORKS
Preference- based Reinforcement Learning. PbRL considers aligning policies with human pref- erences, eliminating the requirement for explicit reward signals (MacGlashan et al. 2017 Warnell et al. 2018). Previous studies have successfully combined PbRL to deep RL agent (Christiano et al.
2017) and high-hiighhae high-dimensional image space (lbarz et al.) 2018). 8. Building this foundation, several
methods (Lee et al. 2021 Liang et al. 2022 Park et al. (002)) further improve the sample efficiency by incorporating techniques like pre-training and relabeling. Recent studies include learning from few preference labels (Liu et al., 2022) and meta-learning (Hejna III & Sadigh 2023). Some other works introduce hindsight techniques (Verma & Metcalf, 2024;/Singh et al. 2024). Recently, offline PbRL has gained attention for optimizing policies with an offline dataset, addressing safety and sample effi- ciency concerns. Existing offline PbRL. algorithms primarily utilize the Bradley-Terry model (Bradley & Terry 1952) to model the likelihood of human preference based on reward signals (Shin et al. 2021 Kim et al. 2023 Choi et al. 2024 Zhang et al.. 2024c). However, such a maximum likelihood method is insensitive to the underlying confidence in human preferences (Newman 2023). Another line of research bypasses the need for learning a reward model by directly aligning policies with human preferences (An et al. 2023) Kang al. [023] Hejjaa & Sadigh 2023 Hejjna al. (2024). while these approaches similarly neglect the uncertainty. Effectively handling this uncertainty and deriving risk-sensitive policies remains critical challenge (Casper al. 2023).
2

---

Published as a conference paper at ICLR 2025
Distributional Reinforcement Learning. While the majority of RL research focuses on maximizing the expected cumulative rewards, [Bellemare et al. (2017) introduces a distributional perspective on RL, utilizing the distributional Bellman operator for value function updates. Such distributional
value functions are sensitive to the uncertainty in the environment dynamics (Mavrin et al. 2019),
enabling the formulation of risk-sensitive policies (Lim & Malik] 2022 Keramati al. (2200) and
better controlling performance (Bellemare et al., (023). Some previous studies propose utilizing
categorical distribution (Bellemare et al.) 2017 Sui et al. (023), quantile functions (Danney et al.
2018b Zhang & Yao 2019] Zhou et al., 2020 2021 L et al., 2022) and diffusion models (Wu et al. 2023) for representing and updating the distributional value function. In this work, we utilize
quantile functions since the statistical benefit of quantile regression is most well-understood (Rowland ett al. 2023). Some recent studies extend distributional RL to offline learning (Ma et al. 2021) mult milti-dimensional rewards (Zhang et al. (021), and multi-agent control (Hu et al. |022 Sun et al.) 2021). However, none of the previous works have considered PbRL from a distributional perspective.
3 PROBLEM FORMULATION
Markov Decision Process (MDP). The agent optimizes the control policy under a Markov Decision Process (MDP) M = (S,A, R. PT (4o7)), where 1) S and A denote the state and action spaces, 2) PTT: : x x + A denotes the stochastic transition function, where A is the simplex over S, RRmin Rmaxx] denotes the reward function, 4) 140 A denotes the initial state distribution, and 5) E 10, is the discounting factor. For brevity, we use M, to denote the MDP without knowing the reward. In this work, we mainly study the episodic MDPs where the planning stops at a terminating state, and the corresponding terminating time is denoted as T E (0, o0).
3) :
Reinforcement Learning. Under an MDP, the objective is learn a policy π - A. which optimizes the following objective:
T
arg max P PTO
R(st,
t=
Instead of optimizing the risk-neutral expected cumulative rewards, we consider risk-sensitive measure 0% Pr. where the confidence level a < 1. Specifically, by implementing ' under the MDP M, we generate a trajectory T E (S x A)"". The corresponding trajectory-generate
probability a.). We define the corresponding risk } to be compact, convex, and bounded set, based on which the risk measure can be induced by the distorted probability distribution p = < p".
can be defined as p² (T) = (o(s)) II.
((a))s))PT(((+
envelope 4,= = {Sa r + 0. 1 Σ Ever ((T)x
In this work, we study the CVaR such that p. E>> 7 R,] suPc., EMg ETep" (T) ΣT 7 Ry]
due to its time consistency and convexity (Rockafellar et al. 2000).
Uncertainty-Aware Preference-based Reinforcement Learning (UA- PbRL). Previous research on PbRL typically learns deterministic reward function (Christiano et al. 2017 Kim et al. 2023) under a maximum likelihood objective. However, in real-world applications, the frequency of comparisons among different samples within the offline dataset is often highly imbalanced, which can significantly influence the safety and reliability of downstream control tasks. Intuitively, if trajectory and its counterparts have only been assessed a few times, the corresponding preference signals should have large uncertainty. This uncertainty is closely linked to the model's 's confidence: the greater the uncertainty, the lower the confidence in its predictions (Smith & Gal 2018 Mena et al. 202T).
To better accommodate the underlying uncertainty in human preference datasets, we study an uncertainty-awer objective for achieving UA-PbRL. Specifically, we capture the uncertainty by learning a distributional reward model f : S x A + A (mmen (Section 4 and incorporate these uncertainty signals into policy learning by utilizing the offline distributional policy evaluation and the risk-averse policy improvement (Section[5) (5). To more accurately reflect the inherent uncertainty, we assume the offline preference dataset is inherently constructed follows:
Assumption 3.1. (Offline Preference Dataset). The preference dataset is constructed with a set of policies II = {m1,..., } (with size L.), where each policy exhibits distinct behaviors. For each candidate policy स, E 11, we generate Ni trajectories in the environment MjR- We uniformly sample pair of trajectories from from a total number of N L generated trajectories. Humans express preferences by mapping (T, ) to (T', 75) where I¹ ranks higher than T². We repeat the sampling and mapping process until we generate the dataset D.
3

---

Published as a conference paper at ICLR 2025
This data-collection paradigm is common in applications where publicly available or structured data is lacking, such as robot control commands for embodied Al or event-related dialogue data for chatbots. Even in tasks with abundant demonstration data, such as autonomous driving. the dataset is typically collected from various agents (e.g.. human drivers), each following their own distinct policies.
Imbalanced Preference Dataset. During the trajectory generation process, under different contexts, agents' policies tend to behave differently, exhibiting various levels of diversity in their movements. For example, as illustrated in Figure (left, policy л2 chooses to move right and exhibits more diverse behaviors, potentially due to the inherent stochasticity in the policy itself or the dynamics of the environment. This diversity leads to the generation a broader range of trajectories .g...., 72
from Ti. Due to this diversity, a specific compared to the more deterministic trajectories .1 N trajectory such To less likely to be selected during the uniform sampling because of its scarcity. Consequently, these trajectories will undergo fewer human comparisons, leading to lower confidence for the reward model assessing their true values.
Such phenomenon is common in practice. For example, a risk-seeking operator may guide the robot into unfamiliar environments. Given the partial observability of sensors and the presence of unpredictable obstacles, the resulting policy tends to exhibit greater diversity compared to those in more routine, less challenging environments. The scarcity of the resulting control trajectories must be considered by assigning them lower confidence in the ranking evaluation process.
Policy 1:mg
Policy 2:
Average Winning Probabillly
Distributiona
Averse Optimal Policy..
Reward Model
B(6,) <((,))
Pa
(a Van((h))
Distributional
Human Comparing
Informative Prior
RL
MAP-Based to-nnk Rank
An illustrative example of UA-PbRI. In this map. humans uniformly sample and compare pairs of trajectories generated by policies πi and π2. The trajectory 72 is shorter in expectation, so 72 is more likely to outperform the average trajectory compared to the longer T₁. However, T2 generates more diverse trajectories, resulting in fewer human comparisons and thus greater uncertainty for each trajectory. Such uncertainty is captured by the distributional reward model, which steers a risk-averse policy to navigate through longer but more reliable path on the top-left map.
Figure 1:
Based on the dataset D, we consider the Offline PbRL problem, where the agent has access solely to an offline dataset that records labeled trajectories instead of interacting directly with environments.
4 LEARNING GENERATIVE REWARD MODEL FROM HUMAN PREFERENCE
In this section, we present our approach learning distributional reward model by proposing: 1) an MAP objective for inferring rewards from human preference (Sec. 4.1 2) an informative Beta prior for modeling uncertainty (Sec. 4.2). and 3) the method of learning generative rewards (Sec.
4.1 MAXIMUM A POSTERIORI OBJECTIVE FOR REWARD INFERENCE
Previous PbRL algorithms commonly utilize the Bradley-Terry model (Bradley Terry 1952 to represent the logelikeldoood of generating human preferences with the reward function:
elrp(r())
Cly. D)
Σ
log elrp(r") (F)))
D
where r (T) 2'r az) denotes the trajectory segment rewards parameterized by so (r',T²)
denotes pair of trajectories where 7' ranks higher than nd, and -' denotes the frequency of such pairs appearing in D. The maximum likelihood objective implicitly imposes uniform prior for (T)
such that Pa (r (T)) 1/( Rmax Restn ). It places a vanishing fraction of its weight on arbitrarily 1-7
large values, which causes divergence in the reward function's parameters (Newman 2023).
In a learning-to-ren -t-rank problem, the winning probability that trajectory 7² wins against 75 remains unchanged under multiplication of all the (+)(T) by any constant factor. To derive a more useful prior for the reward function, enforce the geometric mean strength be one (II, e"v(r")
4

---

Published as a conference paper at ICLR 2025
Le., [=("") 0) and thus the probability of player with strength e"v(r) winning against the average player (whose strength e"r(f) 1) is ((7) e"x(t) ((e"(r)( + 1). Consequently, the reward represents the log odds of defeating the average trajectory. For brevity, we use & as shorthand of (T) denote such probability. Then the prior on the reward function can be defined as:
do po(Ø) dó de po(
etv(T)
po(T Po(Ø)
drp(T)
dera(r) drę(T)
(etv(T) + 1)2
where the first two equations can be derived by applying the chain rule. po(Ø) is the prior, which can follow different representations. This update enables the definition of an MAP objective:
(3)
whi
elr((r'))
II(( elr(("")
((r((T))D) x p(Dirg()))pn(((((()) (rim")) elro(r))) + elr((r)))
II
(elr((r))) + 1)2
Instead of maximizing the likelihood, maximizing this posterior probability can integrate prior knowledge and regularize the reward values, preventing them from diverging.
An essential prerequisite for implementing UA-PbRL with this MAP objective is the construction of an informative prior, Po (Ø). This prior incorporates the inherent uncertainty human preferences into the reward learning process. We introduce the estimation of po(Ø) in the following section.
4.2 LEARNING INFORMATIVE BETA PRIOR
In this study, we learn Beta distribution as the informative prior from D, i.e., Po) = Beta(a, B). since 1) the Beta distribution is the conjugate prior for the Bernoulli distribution, facilitating the update of our belief's with new evidence; 2) the parameters a and | of the Beta distribution can be effectively interpreted as representing the count of positive and negative human feedback, respectively, where "positive" refers to being preferred, while "negative" indicates not preferred. For a trajectory as the number of such "votes" increases, our confidence in the inferred probability improves, resulting in more precise (or "sharper") distribution. This approach enables quantitatively incorporating the confidence level of the Bernoulli probability estimation into our model. Building on this, we present discrete approach to learning the Beta distribution using statistics, detailed in Appendix [B.r]
To learn the distribution in continuous spaces, we propose the variational inference approach to approximate Pu(o/D) by estimating the approximate posterior qu (o(o)) (i.e., Pu(o)D) ( DD))
The goal of our variational inference approach is to learn an approximate posterior distribution qu (D) by minimizing the Kullback - (KL) divergence (d)))(
Du go ( (p(() -Ego [ogg(D(o) + Dи (o( log p(D)).
(4)
Minimizing the above objective is equivalent to maximizing the Evidence Lower Bound (ELBo)
log [p( Dua (1 (d)))]l(((D)) By following Equation (4), ELBo can be represented as:
E.g
log (og((Dl())
DRI go go ( D)
The corresponding trajectory- objective can be reinterpreted as follows:
max E, [rr. - Eeoo((",))DDloog ((T)] (xlaa
(6)
where 1) (ő)T) Beta(a,,, where [a,, 3,] (Ree(()) and fReta denotes neural network parameterized by tb 2) p()) Beta (at, Bp) where okn, So defines our initial belief (hyperparameters, check Appendix B.3 for details), and 3) Φ(T) denotes the Bernoulli probability that 7 ranks higher than r'. Since both the posterior distribution qo (Øl7) and the prior distribution p(0) are Beta-distributed,
we represent the KL divergence term by following the Dirichlet VAE (Joo et al. 2020
T(ap + (b) + log ((an) ((36)
T(a, + +Brr)
(u((a(())7ll)())) log
(7)
[(aa))
) (o)(v(B))) v(ar + 35)] + (a- - un) (ar) - v(a, + 3+) + (3, Bo) ((())))))))))))(()))
where 1) [aro 3o]] and [a,, 3,] parameters from the prior and the posterior functions, and 2) I and I denote the gamma and the digamma functions.
5

---

Published as a conference paper at ICLR 2025
4.3 LEARNING GENERATIVE REWARD MODEL
In this work, we leverage a conditional generative model fo to represent the joint distribution of the step-wise rewards in a trajectory, i.e., f(r) p(rl7) = (r), where f(r) = -E,"ri 7' denotes the trajectory reward. To enable efficient estimation, MAP objective and estimated Beta prior. Specifically, each time we sample rewards from the joint distribution, and then update these rewards based on the following iterative update rule:
derive iterative update rule based on the
pk+1 log a-/lc"" (T) + +11+ E, WITH (Te (r) eee
(8)
3-/lef((r) + + (TTTT///e + cf"(+))
where w wit and -h are calculated based on the dataset D. The design of this iterative update rule based on the following theorem:
Theorem 4.1. Let the informative prior po(o) be a beta distribution Betu(a, 3) and er(r) be the strength of a trajectory segment т. Assuming the geometric mean strength to be I, i.e, [[e(()) the iteration of Equation ( will converge the maximum of its MAP objective (i.e., Equation with po()) the Beta prior), from any starting point, whenever a maximum exists.
The proof can be found in Appendix A To train the generative reward model, we 1) sample rewards from this model, 2) refine these rewards based on Equation (8) under the guidance of human preferences, and 3) update the reward model by fitting it to the updated rewards (see Algorithm
Model Implementation. In this work, we implement f by a distributional reward transformer parameterized by yp. For each trajectory т, we sample the initial step-wise rewards [ro..... ,77] from this reward model (T) and calculate (r) E 70 By utilizing the reward update process (Equation (8 ) we compute the updated segment rewards 3 (r) after K iterations, which approach the actual MAP values due to Theorem 4.1 The corresponding loss function can be modeled as:
(9) Here Ao is sampled from Gaussian distribution parameterized by mean f4 and variance of like in (Liu et al.) (224)) with dropout layers (Srivastava et al.. 2014). To derive tractable gradients, we apply the reparameterization trick to generate A fr + o where denotes samples from standard Gaussian distribution. Both μe and o; are the predictions of causal transformer such that:
min ED [[(** ( where F! ~ (N,,,,
(1te TTol = TousalTransformer(s.. 80,...,,8,,777
(10)
The detailed implementation can be found in Appendix|B. [B.]
5 RISK-SENSI POLICY OPTIMIZATION
In this section, we introduce the approach to learning risk-sensitive policy that aligns with the inherent uncertainty in human preferences. Specifically, employ the distributional Bellman operator to model the distribution of discounted cumulative rewards from the offline dataset (Sec. [. Given the estimated value distribution, we carry out policy improvement by maximizing the CVaR (Sec. 5.2)
5.1 OFFLINE DISTRIBUTIONAL POLICY EVALUATION
To enable distributional policy evaluation, we incorporate the learned reward generator S to the original MDP Myn without knowing the ground-truv reward. The resulting running environment is denoted as MA U Jg. For brevity, we denote M.
(s, (s,0 a) to estimate the R(s,, az) where the initial state-action pair (so, an) is based on an offline dataset D. We represent the distribution of 7 by uniform mixture of supporting quantiles such that (5,, 04) = Ee_U((,,1) (6eg where Og estimates the quantile at the quantile level & and óng denotes Dirac distribution at 0g.
Given a policy क, our goal is to learn a distributional action-value function 7 7 distribution of discounted cumulative reward E 7
To implement offline update for the model parameters 0, we utilize the following Conservative Distribution Evaluation (CDE) objective (Ma et al. 202I):
min LTD(O)= + 1..1110.11 8
ELND Σ exp (e(() a)) ELSayD (0e(s,a)) (11)

---

Published as a conference paper at ICLR 2025
CTD = Ep (Ee..er-rr. [pf (fr + 7 (s ac+1) - (5t ar))
(12)
where 1) is the penalty weight, 2) pf is the E-Huber quantile regression loss at threshold k Huber 1964), and 3) (st, at, st+1, ac+1) is uniformly sampled from the trajectories in the dataset 2> while fr is sampled from our reward generator fgg.
5.2 RISK-AVERSE POLICY IMPROVEMENT
To better handle the underlying uncertainty in human preference, we adopt risk-averse policy updates by maximizing the estimate of CVaR within return distributions. However, Lim & Malik 2022) indi- cates that directly integrating CVaR-based policy improvement with distributional policy evaluation does not necessarily guarantee convergence the optimal policy. To overcome this issue, we utilize the following distributional policy improvement objective for static CVaR (Lim & Malik 2022):
π (at+1|$144)
arg max E U(0.1] (-(9(s<+1)
+(+1)))))
(13)
as+1
where (($+41) = (g(st) rr)// keeps track of the reward history with the initial value of qdh Ee-x (=2 where (s.aa
(s.a) denotes the inverse cumulative density function of distribution
z" (s, a). As is shown in (Bäuerle & Out 2011 т converges to the optimal static CVaR policy by iteratively calculating and updating æ under an MDP whose state space is augmented by g (i.e., I = (s,9) E S x R). Intuitively, q is moving threshold keeping track of the accumulated rewards until the current time step.
Practical Implementation. The complete UA-PbRL algorithm is presented in Algorithm
Algorithm 1:
Uncertainty-Aware Preference-basee based Reinforcement Learning (UA-PbRL)
Input:
The preference dataset D, reward learning epochs N, maximum iterations K Initialize reward model fç. Beta model fBom, action-value model MM a). and policy r(a|s);
Build buffer B, that records all the recorded trajectories in D;
Update the Beta prior model fRetn with the objective (6) until convergence; for 1,2,..... do for 7 E B, do
// Distributional Reward Learning.
Sample rewards .o.....,T] ~ [T( and calculate f(7) =E= 7'1
Estimate the beta prior [á(r). [[)] = f (T);
Calculate the updated 5* (T) with the objective
Update the reward model f with the objective
end end
for r E B, do
// Risk-Sensiin Sensitive Policy Optimization.
Sample step-wise rewards [fo...., fr] ~ fT((T for the trajectory T;
Update the distributional action-value function 7 (s.a) a) with the objective [11
Update the policy model (a]s) with the objective [1
end
EMPIRICAL EVALUATION
In the empirical study, we start by illustrating the learned distributional reward model in discrete Gridworld environments (Section 6.Ip. Next, we construct three Risky PointMaze environments and empirically evaluate the effectiveness of the proposed UA-PbRI algorithm with trajectory visualization (Section 6.2). To assess performance in more challenging settings, we also examine two complex robot navigation tasks (Section 6.3)). Lastly, we extend the experiments to explore the application in Large Language Model alignment (Section
Experiment Settings. Our experiments primarily utilize the public platform Uni-RLHF (Yuan et al.) 2024). which is tailored for offline PbRL. Additionally, to accommodate the underlying uncertainties during the preference learning process, introduce risky regions by incorporating noise into transitions within them, during both trajectory collecting and policy evaluating processes. We create the offline dataset by uniformly sampling from the expert policies trained online and then generating
7

---

Published as a conference paper at ICLR 2025
preferences based their true rewards and risky steps. Consequently, the trajectories collected within the risky region are more diverse. leading to fewer comparisons and greater uncertainty. Please check Appendix B..2 for more details. For fairness, all methods used in robot control tasks are built upon the Preference Transformer (PT) (Kim et al. 2023) for reward learning and Conservative Q Learring (CQL.) (Kumar al. 2020 for policy optimization.
By following Ma et al. (2021) ), we evaluate each approach using 100 test episodes by reporting both the mean and CVaRo.1 (i.e., the average over the worst 10 episodes) metrics, including: 1) episodic rewards, which calculate the cumulative rewards within an episode, and 2) episodic violations, which aggregate the total number of time steps spent inside the risky region. Each experiment repeated with four random seeds, and the results are presented with mean + standard deviation (std).
6.1 REWARD VISUALIZATION IN GRIDWORLD
In this experiment, we construct Gridworld environment to better illustrate the case previ- ously described in Figure I As shown in the left plot of Figure 2 the objective for the agent is to navigate from an initial position (green arrow) to a specified target (blue circle) while avoid- ing the walls (grey blocks). Within the bottom- right area (red markers), the environment demon- strates a degree of stochasticity, where, with specific probabilities (p = 0.1). it receives a ran- dom action instead of the agent's intended ac- tion (refer to Appendix C.1 for more details). Intuitively, the trajectories 72 generated by л2 (passing through right-bottom) exhibit higher rewards in expectation. To accommodate this situation, assign greater preference to T2 by setting the expected chance of observing the preference that 72 ranks higher than Ti to be 0.6.
Gridweriii Offiine Trppctories
Reward Ditt bution Gridworid 1
Figure 2: (Left) The Gridworld environment and offline trajectories. (Right) The learned risk-sensitive reward distributions by our method. Please refer to Figure [ in Appendix [.] for the mean and standard deviation values of each distri- bution and the results in the remaining
settings.
The right plot of Figure 2 illustrates the learned reward distributions at each state, where utilize the blue and red colors to represent the rewards at risk-averse and risky regions, respectively. We find that the distributional reward model successfully captures the underlying uncertainty within the offline dataset in the sense that the rewards in risky regions exhibit a larger expectation but a higher variance. This leads to the result that the generated risk risk-averse policy avoids the risky area and navigates through the top-left map. Additionally, we also construct two distinct Gridworlds and illustrate the corresponding rewards. Please check Figure X in Appendix [DI for complete results. We also visualize the results with preference strengths of p 0.7 and 0.8 in Figures 9 10 and
6.2 MODEL PERFORMANCE IN RISKY POINTMAZE Pocttton
Task Description. In this experiment, tend to the continuous domain by constructing three PointMaze environments, as shown in Fig- ure In the risky regions denoted by red mark- ers, the environmental transitions are influenced by additional Gaussian noise calculated such that pT (8t+1 $t, ax) = f (st, ax) + [(H1, 01), where f() denotes the original transition func-
Figure 3: Three Risky PointMaze settings.
tion. Please check Appendix C.2 for more environmental details.
Comparison Methods. Besides our UA-PbRL algorithm that learns a distributional reward model with risk-averse policy, the following baselines are compared: 1) regular PhRL. (PbRL) Christiano et al. (2017) that learns reward model through the Maximum Likelihood Estimation (MLE) objective, 2) Ensemble Neutral PbRL (EN-PbRL) (Liang et al. 2022) that learns an ensemble of reward functions (we use five ensembles) with risk-neutral policy optimization. 3) Ensemble Risk-sensitive PbRL (ERS-PbRL) that replaces the neutral objective in EN-PbRL with a risk-averse one (Ma et al.. 2021). 4) Contrastive Preference Learning (CPL) (Hejna et al. 2024) that learns policies directly from preferences without reward learning.
Results Analysis. Table shows the evaluation performance, with the best results in each setting
(highest rewards or lowest violations) highlighted in bold. Please check Figures T2 and 13 in
8

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation results in three Risky PointMaze settings. Each value is reported as the mean + standard deviation (std) calculated from 100 episodes and seeds.
Method
CPL
PhRL
EN-PbRL
ERS-PbRL
UA-PbRL (ours)
Rewards
Mean
47.6 16.4
3084274
40.7
25.0=:.11
3
PointMaze
CVaRo
-00.3-44.
-60.0
+.0.0+000
0.0
-16.6 58.9
Setting I
Violations 4
Mcan
310.5:3224
272.7=20.2
265.8 12.2
172.3 19.7
86.1 = 86,4
CVaRg
494.9 101.6
4524 46.0
273.8
187.3 + 134,4
Mcan
38.7
63.2
6,.0
65.8
65.4 :: 11.2
Rewards
PointMaze
CVaRo
-15.77444.3
42.3
43.0
8..0
53.1
Setting 2
Violations +
Mean
12.33112.5
1.1.2:599
1259 15.3
98.01 16.1
5.2
CVaRa
23.63
130.3
1639
1.7.9
50.8 + 71.7
Mean
38.3 47.33
64.2
68.6 7.5
67.7
70.3 15.4
Rewards
PointMaze
CVaRa
64.6
22.7
41.8
19.2 68.5
48,9 55.1
Setting 3
Mean
83.6 123.6
71.1+7177
107.1
47.7
27.7 33.1
Violations +
CVaRo.
235.6
167.5 120.3
18..0
18..
117.7 96.4
Appendix D.2 for evaluation results during the complete training phase. The results show that UA-PbRL consistently outperforms other methods with higher CVaRg.1 rewards and fewer violations in both mean and CVaRo.1 metrics. This underscores that the risk-averse policy in UA-PbRL can avoid passing through highly uncertain regions. When it comes to the mean rewards, UA-PbRL still achieves compatible performance due to its superior CVaRa. performance. We also find that ERS-PbRL and CPL sometimes achieve higher mean rewards than UA-PbRL. This is because the two methods do not acknowledge the risky regions and solely pursue expected cumulative rewards, resulting in traversing through risky areas with occasional successes.
Results Visualization. Figure illustrates 10 evaluation rollouts from PbRL and UA-PbRL the first setting of Risky PointMaze (check Figure 14 in Appendix D.2 for complete re- sults). We find that UA-PbRL drives a risk- averse policy that navigates to the longer but less stochastic path. By contrast, the traditional PbRL method struggles to perceive such uncer- tainties and tends to navigate through the risky region directly, where the noisy transition occa- sionally induces unsafe movements, leading to its poor CVaRo.1 performance.
20
00
30
Figure 4: The trajectories generated by PbRL (left) and UA-PbRL (right) during evaluation.
6.3 MODEL PERFORMANCE IN RISKY ROBOT CONTROI.
Task Description. We follow Ma al. (202T) and construct more complicated robot navigation tasks, including nigh-dimensional Ant environment and Swimmer environment. For example, under the Ant environment (check Figure in Appendix C.3). the goal is to travel from a starting point to destination, where there exists risky region in the middle of the route. The environmental transition within the risky region is subject to Gaussian noise N ((H2, 02), which introduces the risk. While
agent might pass through the risky region regardless of the underlying risk, risk-aware agent should completely avoid it. Additionally, to assess model performance with real human data, conduct experiments using public human preferences from Yuan al. (2024) four risk-sensitive D4RL tasks (Urpí et al. 2021. Please check Appendix D.3 for detailed results.
Comparison Methods. For more comprehensive evaluation, in addition to the baseline methods used in the PointMaze environment, we also perform more ablation studies, where 1) Distributional PbRL (D-PbRL) replaces the CQL used in PbRL with CODAC (Ma al. 202T). 2) UA-PbRL- Uniform replaces the informative Beta prior with uniform one (i.e., = = 1), and 3) UA-PbRL- Neutral replaces the CVaR objective with a risk-ncutra one (i.c., expectation).
Table 2: Evaluation results in risky robot control tasks. Each value is reported as the mean + std calculated from 100 episodes and 4 seeds. R and C denote rewards and violations, respectively.
Mecaed
ERS-PARL
UA-PoRL-Nouna
(AA-PRLL(cus))))
Risky
R^
CViRa.
Mean
.2803.5+
-1819.3 65.0
112.1
-1816611
63.25+
2292.
45819
99.0
229221
-1677.1
69.8
101.5
-2577.2
-2060.5
100
114.3
-2635.1 1043
-1768
4:..1
-2215.3 9866
-1806.2 1877
Aat
Mam
46.1=78
484:49
68.263
312=30
352=66
21.0
CVAR.
196.6=15.0
21855
43.70:443.
2008-14
11.
-
12.9
Moin
149811366.
20013
2653
2701.
3001
2011
2260
-2515 1880
2011X
183.
Risky
R
CVàRa.
1.
4512.8 452.1
4316 3104
-3991.9
264.5
3856.2 299
4070.4
246.5
-3498.2
230,9
Swimmer
10.7
CVRRg
66.3
56144447.
$12.3+
407.9-388
4263+3320
38100-44.1
175.5 17.8

---

Published as a conference paper at ICLR 2025
Results Analysis. The empirical results in Risky Ant and Risky Swimmer environments shown in [ We find that all the methods will inevitably encounter the risky region due to the intention of reward maximization. However, compared to other methods, UA-PbRL exhibits better performance with higher C rewards and fewer violations (both mean and CVaRo.1), which demonstrates the risk-averse ability of the learned policy. Note that although D-PbRI. with the distributional critic obtains the highest mean rewards in Risky Ant, it struggles to optimize the worst-case (i.c., CVaRg.1 1) rewards and commits the highest number of violations. Additionally, we find that the ablation methods UJA-PRRL-Unifond than conventional PbRL and D-PbRL in terms of the violations, which indicates the effectiveness of the distributional reward model and the risk-averse policy optimization. Regarding the CPL method, struggles with worst-case performance, as it remains unable to account for uncertainty.
and UJA-PbRL-Neutral exhibit relatively better performance
6.4 EXPERIMENTS ON LARGE LANGUAGE MODEL. ALIGNMENT
As Large Language Models (LLMs) become increasingly prevalent, addressing the underlying uncertainty LLM alignment from offline datasets is critical (Casper et al. 2023 Zhang et al. (024b). Since LLMs are typically trained datasets where harmful sentences occur infrequently, these harmful sentences inherently carry significant uncertainty. Intuitively, identifying this uncertainty and applying risk-averse policy can effectively reduce harmful sentence generation.
In this part, we evaluate the proposed UA-PbRI approach compared to traditional PhRL., also known as RL from Human Feedback (RLHF), for aligning LLMs. We also include advanced RLHF method, Distributional Preference Learning (DPL) (Siththaranjan et al., 2024) that also learns a distributional reward. In contrast UA-PbRL, which addresses uncertainty arising from limited data, DPL accounts for the uncertainty arising from hidden contexts, such as the combination of preference data with varied objectives. Specifically, we finetune two publicly pre-trained LLMs. TinyLLaMa-1.11 BB (Zhang et al. (024a) and LlaMa-3-8B (Dubey et al. 2024), on the PKU-SafeRL.HF 10K dataset (Ji et al.] 2023) Dai et al. 2024), which contains numan-labeled preference data on the helpfulness and harmlessness of prompt-response pairs, where we prioritize the safer samples. Following Dai et al. (2024), we evaluate the fine-tuned models on 280 test samples across 14 harm categories. We use GPT-4o to assess its safety (prioritized) and quality. Check Appendix C.4 for more details on the experimental settings and evaluation process.
The evaluation results presented in Figure Additionally, we analyze the increase in evasive responses relative to the pre-trained model by comparing the harmlessness of the generated responses, shown Table 7 Appendix [.55) A case study of responses to a harmful prompt is also included in Table The findings show that UA-PbRL effectively mitigates the generation of harmful outputs when faced with potentially harmful prompts. The underlying reason is that the typical RLHF reward model, trained without considering uncertainty, tends to assign higher rewards directly providing solution prompt, especially if it has not "seen" the proper response before. Harmful sentences that appear less frequently in the preference dataset often exhibit high uncertainty. UA-PbRL efffeilleff captures this uncertainty and performs the policy to prevent producing harmful responses. Regarding DPI, shows relatively satisfactory performance with improved safety. However, in this task, the preference labels typically share a unified objective to prioritize safety, which may limit DPL's effectiveness.
TinyLLaMe-1.18
LLaMa-3-88
Winning Percentage
0:00
Figure 5: Evaluation results on LLM alignment.
CONCLUSION
In this paper, we introduce an uncertainty-awee preference alignment approach to learning policies using offline demonstrations with preference labels. We propose Maximum A Posteriori (MAP) objective for learning a distributional reward model with an informative Beta prior and then utilize the distributional Bellman operator with the Conditional Value-at-Rin Risk (CVaR) metric to develop a risk-sensitive policy, which aware of the inherent uncertainty in the human preference dataset. Empirical results demonstrate the effectiveness of the risk-sensitive ability of our approach. Future directions involve incorporating uncertainty awareness into direct preference alignment methods with diverse human preferences.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported in part by Shenzhen Science and Technology Major Program under grant KJZID20240903 1400000012 Shenzhen Fundamental Research Program (General Program) under grant JCYJ202308071 14202005, Guangdong-SSenzzhne 202311515110617, Guangdong Basic and Applied Basic Research Foundation under grant 2024A1515012103, and Guangdong Provincial Key Laboratory Mathematical Foundations for Artificial Intelligence (2023B1212010001)
Joint Research Fund under grant
REFERENCES
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung- Boing-Min& Kim, and Hyun Oh Song. Direct preference-based based policy optimization without reward modeling. In Advances in Neural Information Processing Systems, NeurIPS, 2023.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594, 2023.
Nicole Bäuerle and Jonathan Ott. Markov decision processes with average-value-at-risk criteria. Mathematical Methods of Operations Research, 74(3):361-37,, 2011.
Marc G. Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International Conference Machine Learning, ICMI., volume 70, pp. 449-458, 2017.
Mare G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning, MIT Press. 2023.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355 1952.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint S.Ki::2307.12217, 2023.
Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei-Yur Wang. Rime: Robust preference-based reinforcement learning with noisy preferences. arXiv preprint arXiv:2402.17257. 2024.
Heewoong Choi, Sangwon Jung, Hongjoon Ahn, and Taesup Moon. Listwise reward estimation for offline preference-ber based reinforcement learning. In International Conference on Machine Learning, ICML, 2024.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg. and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, NeurIPS, pp. 4299-43000 4307, 2017.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference Machine Learning, ICMI., volume 80, pp. 1104-1113, 2018a.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. AAAI Conference on Artificial Intelligence, pp. 2892-2901, 2018b.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In International Conference Learning Representations, ICLR, 2024.
11

---

Published as a conference paper at ICLR 2025
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Oollcy reinforcement learning for addressing value estimation errors. 1EEE Transactions Neural Networks and Learning Systems, 33:6584-6555 6598, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 1783, 2024.
Bin Fang, Shidong Jia, Di Guo, Muhua Xu, Shuhuan Wen, and Fuchun Sun. Survey of imitation learning for robotic manipulation. Int. J. Intell. Robotics Appl., 3(4):362-369, 2019.
Lester R Ford Jr. Solution of ranking problem from binary comparisons. The American Mathematical Monthly, 64(8P2):28-3) 1957.
Justin Fu. Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.00221 2020.
Tuomas Haarojj, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, ICMI., pp. 1861-1870, 2018.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without reward function. In Advances in Neural Information Processing Systems, NeurIPS, 2023.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without rl. In International Conference Learning Representations, ICLR, 2024.
Donald Joseph Hejna and Dorsa Sadigh. Few-shot preference learning for human-in-the-lone in-the-loop rl. In Conference on Robot Learning, pp. 2014-2025, 2023.
Jifeng Hu, Yanchao Sun, Hechang Chen, Sili Huang, Haiyin Piao, Yi Chang, and Lichao Sun. Distributional reward estimation for effective multi-agent deep reinforcement learning. In Advances in Neural Information Processing Systems, NeurIPS, 2022.
Peter J. Huber. Robust Estimation of Location Parameter. The Annals of Mathematical Statistics, 35(1):73 - 101, 1964.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems, NeurlPS, pp. 8022-8034, 2018
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human- human-preference preference dataset. In Advances in Neural Information Processing Systems, NewrIPS, 2023.
Weonyoung Joo, Wonsung Lee, Sungrae Park, and Il-Chul Moon. Dirichlet variational autoencoder. Puttern Recognition, 107: 14,7 2020.
Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference- guided policy optimization. In International Conference on Machine Learning, ICML, 2023.
Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning evar policy. In AAAI Conference on Artificial Intelligence, pp.
4436-4443, 2020.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for RL. In International Conference Learning Representations, ICLR, 2023.
W Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement. In IEEE International Conference Development and Learning, pp. 292-297, 2008
12

---

Published as a conference paper at ICLR 2025
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, NeurIPS, pp- 1179-1191, 2020.
Kimin Lee, Laura M. Smith, and Pieter Abbeel. PEBBLE: feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, ICML, volume 139, pp. 6152-6163, 2021.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, ICLR, 2022.
Shiau Hong Lim and llyas Malik. Distributional reinforcement learning for risk-sensitive policies. In Advances in Neural Information Processing Systems, NeurlPS, 2022.
Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-nett Implicitly differentiable reward learning for preference-based reinforcement learning. Advances Neural Information Processing Systems, NeurIPS, pp. -2270-22284, 2022.
Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, and Xiu Li. Pearl: Zero-shot cross-task preference alignment and robust reward learning for robotic manipulation. In International Conference on Machine Learning, ICML, 2024.
Yudong Luo, Guiliang Liu, Haonan Duan, Oliver Schulte, and Pascal Poupart. Distributional reinforce- ment learning with monotonic splines. In International Conference on Learning Representations, ICLR, 2022.
Yecheng Jason Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. In Advances in Neural Information Processing Systems, NeurIPS, pp. 19235-19247, 2021.
James MacGlashan, Mark Ho, Robert Tyler Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. Interactive learning from policy-dependent human feedback. In International Conference on Machine Learning. ICML, volume 70, pp. 2285-2294 2017.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International Conference on Machine Learning, ICML, volume 97, pp. 4424 4434, 2019.
José Mena, Oriol Pujol, and Jordi Vitria. A survey uncertainty estimation in deep learning classification systems from a bayesian perspective. ACM Computing Surveys (CSUR), 14(9):1-35, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
M. E. J. Newman. Efficient computation of rankings from pairwise comparisons. Journal of Machine Learning Research, 24:238:1-2388 2023.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-super reward learning with data augmentation for feedback-efficient preference-bate basedd reinforcement learning. In International Conference Leurning Representations, ICLR, 2022.
R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21-42, 2000.
Mark Rowland, Yunhao Tang, Clare Lyle, Rémi Munos, Mare G. Bellemare, and Will Dabney, The statistical benefits of quantile temporal-diifereecee learning for value estimation. In International Conference on Machine Learning, ICML, volume 202, pp. 19210-292331 2023.
Dorsa Sadigh, Anca D. Dragan, Shankar Sastry, and Sanjit A. Seshia. Active preference- learning of reward functions. In Robotics: Science and Systems XIII, 2017.
13

---

Published as a conference paper at ICLR 2025
Daniel Shin, Daniel S Brown, and Anca D Dragan. Offline preference-basee apprenticeship learning. arXiv preprint arXiv:2107.09251, 2021.
Daniel Shin, Anca Dragan, and Daniel S Brown. Benchmarks and algorithms for offline preference- based reward learning. Transactions on Machine Learning Research, 2023.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
David Silver, Julian Schrittwisser Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):544359, 2017.
Utsav Singh, Wesley A Suttle, Brian M Sadler, Vinay Namboodiri, and Amrit Singh Bedi. Piper: Primitive-informedd preference-based hierarchical reinforcement learning via hindsight relabeling. In International Conference on Machine Learning, 2024.
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell Distributional preference learning: Understanding and accounting for hidden context rlhf. In International Conference on Learning Representations, ICLR, 2024.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. In International Conference on Uncertainty in Artificial Intelligence, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-195)) 2014.
Yang Sui, Yukun Huang, Hongtu Zhu, and Fan Zhou Adversarial learning of distributional rein- forcement learning. In International Conference on Machine Learning, ICML, volume 202, pp. 2023.
Wei-Fang Sun, Theng-Kuang Lee, and Chun-Yi Lee. DFAC framework: function via quantile mixture for multi-agent distributional q-learning. International Conference on Machine Learning, ICML, volume 139, pp. 9945-9954, 2021.
Factorizing the value
Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. In International Conference on Learning Representations, ICLR, 2021.
Mudit Verma and Katherine Metcalf. Hindsight priors for reward learning from human preferences. In International Conference on Learning Representations. ICLR, 2024.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782) 2019.
Garrett Warnell, Nicholas R. Waytowich, Vernon Lawhern, and Peter Stone. Deep TAMER: interactive agent shaping in high-dimensional state spaces. In AAAI Conference Artificial Intelligence, pp.
1545-1554, AAAI Press, 2018
Runzhe Wu, Masatoshi Uehara, and Wen Sun. Distributional offline policy evaluation with predictive error guarantees. In International Conference on Machine Learning, ICML, volume 202, pp.
37685-37712, 2023.
Wanqi Xue, Bo An, Shuicheng Yan, and Zhongwen Xu. Reinforcement learning from diverse human preferences. arXiv preprint arXiv: 1I 1774, 2023.
Yifu Yuan, Jianye Hao, Yi Ma, Zibin Dong, Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, and Yan Zheng. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In International Conference on Learning Representations, ICLR, 2024.
Emst Zermelo. Die berechnung der ergnies-er als ein maximumproblem der wahrschein- lichkeitsrechnung. Mathematische Zeitschrifi, 29(1):436-466 1929.
14

---

Published as a conference paper at ICLR 2025
Wenhao Zhan, Masatoshi Uchara, Nathan Kallus, Jason D. Lee, and Wen Sun. Provable offline preference-basee based reinforcement learning. International Conference on Learning Representations, ICLR, 2024.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprin! arXiv:2401.008 0.3385, 2024a.
Pushi Zhang, Xiaoyu Chen, Li Zhao, Wei Xiong, Tao Qin, and Tie- Yan Liu. Distributional rein- forcement learning for talti-dimensional reward functions. In Advances in Neural Information Processing Systems, NeurIPS, pp. 1519-1529, 2021.
Shangtong Zhang and Hengshuai Yao. QUOTA: the quantile option architecture for reinforcement learning. AAAI Conference on Artificial Intelligence, pp. 5797-5804, 2019.
Xiaoying Zhang, Caan-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcoming reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv preprint 1:XXi:22403.0177. 2024b.
Zhilong Zhang, Yihao Sun, Junyin Ye, Tian-Shuo Liu, Jiaji Zhang, and Yang Yu. Flow to better: Of- fline preference-based reinforcement learning via preferred trajectory generation. International Conference Learning Representations, ICLR, 2024c.
Fan Zhou, Jianing Wang, and Xingdong Feng. Non-crossing quantile regression for distributional reinforcement learning. In Neural Information Processing Systems, NewrlPS, 2020.
Fan Zhou, Zhoufan Zhu, Qi Kuang, and Liwen Zhang. Non-decreasing quantile function network with efficient exploration for distributional reinforcement learning. In International Joint Conference on Artificial Intelligence, pp. 3455-3461, 2021.
15

---

Published as a conference paper at ICLR 2025
A PROOF OF THEOREM 4.1
We prove the theorem two steps.
The first step. We prove that the MAP objective of Equation 3 with (o Beta( B) reaches its maximum when:
0-/le" (r) + +11 Ej wir et (r) (r) + (
pM (T) = llog
(14)
(-/1c//(1) + +1]+ E, wrt (et (r) (*)())))
Proof. To incorporate the informative Beta prior (o()) Beta 3) into the iterative update objective, we start by rewriting the prior on the rewards as follows:
(T(r)
p(r(T)) (0)
(15)
(e"(T) 1)2
Φ (1 2)
e*(r)
(16)
B(a,, Br)
(e"(T)
+ 1)2
((())
a+
9,-1
cn(r)
er6)41
(17)
B(ar, 9r)
(eT(F) + 1)2
where 04 0, B, > 0 are the prior parameters for trajectory т, and B(a,, B,) 10+-1(1-
i)a--1 dt is the Beta function serving as a normalization constant. Substitute the above prior into Equation 3 we get:
p(r(T)[D) & p(D ((D(((T))(((()))
a,i-1
8,1-1
elr(T')]
elr(r')]
II
II
dre
elr(T")] + elr(to]
B(a,t,
(elr(r")) + 1)2
(18)
For simplicity, we denote elr(r') '] as si, and argg Br. az 3 in the remaining part of the proof. The log- liglliiel can be represented as:
Σ log Σ w log(s, + 82) + E(a; (a; 1) log 8; E(a: (a. 1) + 1)
Σ(: 1)log(s; 1) N log B(a,, B,) + Σ log st Σ log(s; +
Jur'r'( (log(s)) [1rtr((og(ss))
t,,)) (1og(s))-log(e) log(s; +
Σ a log(s;)
- (a; + log(s: 1)) - log B(a,, B.).
(19)
where N the number i. Differentiating the above equation with respect to 8; for any and setting the result to zero, get:
Σ w
W . ,
a
04 +
0. 8i
(20)
8;
8i +
8; +
After rearranging the above equation, we obtain,
0i/(8; + 1) + E, wt 85/(8; + +8))
8; = 1t/(s:+1)] Σ, , .-- ( + 8,)
(21)
L
The second step. We prove that iteration of Equation (8) will converge to its global maximum (Equation 14), from any starting point, whenever maximum exists.
16

---

Published as a conference paper at ICLR 2025
Proof. For simplicity, rewrite the iteration objective (Equation (8) as follows:
ai/(s; 1) + Ej wt 8// 9//(8; + 85)
(22)
Be/(st + Eg w . /(s 85)
Consider an asynchronous update scheme. It is worth noting that: 1) any fixed point of this iteration
corresponds to a stationary point of the posterior probability: 2) the iteration only produces non-
negative values of 8; (given non-negative initial values); 3) the MAP objective (3) is bounded. Therefore, if the posterior probability under 8e increases after applying the iteration, the converged fixed point is indeed the global maximum.
Let's examine the step where a specific is updated. We define a function f(s;) as the sum of the current term in the log-likelihood of the posterior probability (i.e., Equation 19 that dependent on 84.
f(x,) Σ woine
log
8 + 83 )
$4
Σ
wT log(s; 82) +a,log(s;) -(a; + 8.)oo((s; + -Noo B(a Bz)
Σ
log
Si
Σ log(s: + 83) + +a,log
8i
-elog(s: (st + 1)-Nlog B( 3e).
+ 8j)
+
Suppose we update into of using Equation 22 we have
5
5 - -B:log( + 1)- - Nlog B(as; 3i)
[(6)) = Σ log
w log(s + 85) +adlog
8 + 8j
$/
w wirt
Σ w r log
+
w
Σ
2 (og(s; + +s)) (5/ - -si Σ
8i + 8j
8
8i + 8j
+ 85
+ ai log
8;
+
Ori
8/ 8;
3e log(s; (5 + +1)
8/ -888
log B(as,
88 +
8/(81 + 1)
8; +1
( f(s) + (8% -8i)
1
w
8j
0
B1
$
8, +
s + 1) 8; + Sij
8;
+
-f(s)).
(23)
(a) holds due to Equation (16) and (17) in (Newman 2023 (treat n; 54, T.. 5 and m. =
85) ), along with two inequalities that log (2/ [x 1)) log / (y+ (z 1))
and log(x + 1) 2 log(y + 1) - (x g)/(v 1).
(b) holds due to the iteration given by Equation (2)
Consequently, applying Equation (22 for updates increases f(s:) and also the posterior probability until a fixed point is reached, where sf = 8;- Once the global maximum is attained for all s; the MAP objective
[14 reaches its maximum value. This completes the proof.
Remark. Note that it has been shown in Zeereelo 1929 Ford Jr 1957) that the likelihood function of r(T) has unique stationary point that is also global maximum, given s; 2 0, under the conditions that 1) the geometric mean of the strengths is one (as enforced this paper), and 2) the network adjacency matrix "" is strongly connected. Based on this, Newman 2023) extends the results to the posterior function, proving that a global maximum exists without needing the aforementioned second condition. Therefore, as demonstrated in (Newman) (202)), the posterior distribution with geometric mean of strengths equal to one guarantees global maximum.
B IMPLEMENTATION DETAILS
B.1 EXPERIMENTAL SETTING
In this paper, we utilized a total of 8 NVIDIA GeForce RTX 4090 GPUs, each equipped with 24 GB of memory. The random seeds in the continuous environments are 0, 123, 321, and 666. We
17

---

Published as a conference paper at ICLR 2025
trained the agents offline and chose the final epoch for evaluation over 100 episodes. For faimesss. we implement the reward model for each method the same, by a causal transformer like (Kim et al.) 2023). We also utilize a transformer-base architecture for the Beta model utilized in method for learning informative priors.
During the model implementation (Equation 9 sample N predicted rewards jo from the reward model and calculate the corresponding N target values f. for each trajectory. In practice, for varying initial reward values řn. the target values f computed after a fixed number of K iterations may not precisely correspond to the optimal solution of the MAP objective. Nevertheless, as each iteration progressively approaches the MAP objective, the resulting estimates represent multiple distinct points within the underlying posterior. The variance among these estimates can provide information for distributional reward learning (i.e., learning σ). Additionally, this variance inherently tied to the variance of the posterior, because for different trajectories, the iteration objective varies with different values of a and B. Consequently, the variance among N estimates j. for different trajectories will certainly differ. Intuitively, larger and 3 result in a stronger initial belief weight during the iteration, reducing the influence of the observed preference labels, thus with a smaller variance the N estimates. Consequently, can capture the information provided by the posterior variance through the K iterations applied to N sampled values.
B.2 OFFLINE PREFERENCE DATASET
Based on Assumption 3.1
we create the offline preference dataset as follows:
For Gridworld and PointMaze environments, we train two classes of policies: one that navigates the shorter path through the risky region (risky), and the other that avoids the risky region by taking longer route (risk-averse). We collect 500 trajectories per policy in the Gridworld environment and 30 trajectories per policy in the PointMaze environment.. Then perform uniform sampling over them. As a result, the risky onc's ' trajectories are more diverse because of the random noise in the risky region (as shown in the left column of Figure 8). To assess the risk-awareness of our methods, encourage policies to embrace riskier actions by assigning higher preference to trajectories produced by risky policies. Specifically, we establish the expected likelihood of risky trajectory outranking a risk-averse one to be 0.6. We compare 5000 trajectory segments with a length of 8 in Gridworld and 2000 trajectory segments with a length of 100 in PointMaze. This setup results in greater number of comparison steps than the originally collected trajectories, ensuring that most trajectories are compared multiple times and more diverse trajectories are compared less frequently.
For the risky robot navigation task (i.e.. Risky Ant and Risky Swimmer), we train two Distributional Soft Actor Critic (DSAC) (Duan et al., 2021) agents online in each environment over 1000 episodes: one optimized for expected returns and the other for CVaR returns. These agents are then employed to generate expert trajectories, and we uniformly sample from them as the dataset for offline RL training. Consequently, trajectories produced by the former agent tend to be riskier, favoring shorter paths through risky regions for higher expected rewards, while trajectories from the latter aim to avoid risk due to CVaR optimization. We sample 100 trajectories for each policy in the risky robot control tasks. Following this, we generate the preference labels as follows: for pair of trajectories (T), T2), if Ir(n) r(r2) > t, we prioritize the trajectory with higher rewards, otherwise, we select the trajectory with more steps in risky regions. Here f is a threshold and we set = 10 in this experiment. We compare 5000 trajectory segments with a length of 100 in each environment. This also leads to a substantially greater number of comparison steps than the original collected trajectories.
As result, we introduce an imbalance in the offline preference dataset with a diverse range of comparison numbers for different trajectories. For better understanding, we calculated the distribution showing the frequency which trajectory segments are sampled and compared in the discrete Grid- world environments. Table presents the results, highlighting the divergence in the offline preference dataset. In continuous environments, directly counting the number of comparison trajectories is challenging. However, the same principle applies: trajectories in the risky region are sparser, while those in safe regions are denser, resulting in diverse comparisons during uniform sampling. Neural networks can effectively identify patterns when similar trajectories are compared more frequently.
B.3 HYPERPARAMETERS
As our approach primarily relies on the Conservative Offline Distributional Actor Critic method MMa al. 2021) for offline policy learning, maintain the CODAC-specific hyperparameters consistent
18

---

Published as a conference paper at ICLR 2025
Table 3: Distribution of trajectory segment comparisons in Gridworld environments.
min 10% quantile
90% quantile
99% quantile
max
Gridworld
3
94
1277
Gridworld
5
225
1011
Gridworld
5
166
617
with the original study and only adjust the learning rate and Lagrange threshold. Regarding the reward model, we adhere to the architecture of the preference transformer model (Kim al. 2023) as implemented in the Uni-RLHI benchmark (uaan et al. 2024). Additionally, we employ the transformer architecture for the Beta model to learn sequential representations, For the choice of the prior (xo,/0 for Beta distribution, we utilize an uninformed prior such that oo = Bg = 1. This corresponds uniform distribution over [0, 1], which serves as natural choice for uninformed belief.. This choice provides neutral starting point with the least information. We summarize the main hyperparameters Table Please check Appendix C.4 for LLM alignment experiment details.
B.4 DISCRETE IMPLEMENTATION OF BETA PRIOR
Intuitively, the Beta prior can be derived directly from data statistics by calculating the number of times a particular trajectory wins or loses against another within the offline preference dataset. This approach eliminates the need for neural network training, especially in discrete environments with finite state-action spaces. Specifically, as outlined at the beginning of Section 4.2 we interpret the learning of the Beta distribution its a parameter increases (c.g.. +1); conversely, if it is not preferred, its 3 parameter increases. By evaluating all samples in the offline dataset, derive discrete approximation of the Beta distribution.
"voting" process: if trajectory is preferred over another,
C ENVIRONMENTAL SETTING
C.I GRIDWORLD
The Gridworld environment consists of map with several grids for movement. We create three unique scenarios, as shown in the left column of Figure 8] The agent's objective is to navigate from a starting location to a target location while avoiding the specified walls. At each step, the agent can choose from four possible actions, each corresponding one of the four cardinal directions (up, down, left, right). Starting from the initial position, the agent receives reward of upon successfully reaching the target location, and a reward of 0 in all other cases. The game continues until maximum of 50 time steps is reached. Additionally, we introduce risky regions to the environment where the transition exhibits degree of uncertainty. Within the risky regions, with a predetermined probability of 0.1, the environment executes random action instead of the intended action chosen by the agent.
C.2 RISKY POINTMAZE
The PointMaze environment is continuous domain that generalizes from the discrete Girdworld. In (DoF) ball reach a designated goal
this scenario, the objective to control a eederre-of--rreeeee in closed maze. As shown in Figure we keep the same starting, target, wall, and risky locations as the previous Gridworld environment for the sake of evaluation in the continuous domain. The risky regions are characterized by adding Gaussian noise the environmental transition functions, introducing stochasticity and risk into the agent's movements. Specifically, the transition in risky regions is PT (St4 St, az) f (Se, a:) + +N(u,, ,i), where f(-) is the original transition function. We fix μ1 = and 0 = 0.05 across the environments. The maximum step is 600.
C.3 RISKY ROBOT NAVIGATION
Risky Swimmer In this environment, the agent controls robot with two rotors connecting three segments, whose goal is to navigate from a starting state [1, I] to target state [5, 5] quickly as possible. There is a risky region centered at [3, 3] with radius of The agent's 's dynamics remain consistent with the MuJoCo Swimmer environment. At each timestep, the agent's 's reward calculated as the negative Euclidean distance to the goal plus 0.1 times its velocity. If the agent enters the risky regions, its transition will be influenced
Figure 6: Risky Swimmer.
19

---

Published as a conference paper at ICLR 2025
Table 4: List of hyperparameters in the proposed UA-PbRI.. To ensure equitable comparisons, maintain consistency in the parameters of the same neural networks across different models.
Parameters
Risky PointMaze
Risky Ant
Risky Swimmer
General
Max Episode Length
600
400
1000
Discount Factor
0.99
0.99
0.99
Training Epochs
50
500
500
Policy Model
Actor Network
256, 256
256, 256
256, 256
Critic Network
256, 256
256, 256
256, 256
Actor Learning Rate
3e-6
3e-5
3e-5
Critic Learning Rate
3e-5
3e-5
3e-5
Min Q Weight
5
10
10
Lagrange Threshold
10
10
10
Number of Quantiles
32
32
32
Huber Regression Threshold
Entropy Tuning
True 0.1
True 0.1
True
Risk Level
0.1
Reward Model
Network
256
256
256
Learning Rate
5e-5
5e-5
5e-5
Number of Attention Heads
4
4
4
Number of Layers
1
1
Batch Size
64
64
64
Beta Model
Network
256
256
256
Learning Rate
3e-5
3e-5
3e-5
Number of Attention Heads
4
4
Number of Layers
1
Batch Size
64
64
64
Regularizer Weight
0.1
0.1
0.1
Initial Belief
B =
B =
by Gaussian noise (0, 0.05). The episode terminates when the Euclidean distance between the agent and the target less than or reaches the maximum steps of 1000.
Risky Ant In this environment, the agent controls high-dimensional ant robot with four legs, featuring 113 dimensions of observation. The goal is to navigate from the starting state [2, 2] the target state [8, 8). A risky region centered at [5. 5] with a radius 2. The agent's 's dynamics are identical to those of the MuloCo Ant environment. At each timestep. the agent's 's reward is calculated as the negative Euclidean distance to the goal plus 0.1 times its velocity, encouraging rapid progress toward the target. If the agent enters the risky region, its transitions will be affected by Gaussian noise A (00,, 0.05). The Ant. episode terminates when the Euclidean distance between the agent and the target is less than 1, or when the maximum of 400 steps is reached.
Figure 7: Risky
C.4 LLM ALIGNMENT
We primarily adopt the framework and implementation Safe RLHF (Dai al. 2024). utilizing the provided evaluation script to fine-tune the LLM models and assess LLM performance through GPT-4o evaluation on 280 test samples selected from 14 harm categories. For the PKU-SateRLHF-110 dataset used in our work, we selected 10K preference samples from the 30K dataset provided in Ji et al. (2023), where we ensure a higher proportion of safe responses compared to harmful ones. We maintain the same RLHF hyperparameters outlined in (Dai al.) 2024) (Tables 2 and 4). UA-PbRL aligns with these key hyperparameters, where the imitial belief for the Beta model is set
0g 30 and the quantile number for distributional RI. is 128 with risk level of 0.1. The
20

---

Published as a conference paper at ICLR 2025
design of the GPT-4o prompts follows the implementation detailed in Section C.2 and the code base of (Dai et al. 2024). For the Distributional Preference Learning (Sittharanjaa et al. (024) method, adopt the mean wean-and-varianee type of the reward model with the risk- policy optimization.
Based on this setup, fine-tuning the reward or Beta model on the LlaMa-3-8B model takes approxi- mately 2 hours, while finetuning the PPO or Distributional PPO actor per epoch takes around 6 hours on A800 GPUs.
D MORE EXPERIMENTAL RESULTS
D.1 GRIDWORLD
Visualization results. In Figure the middle column of plots illustrates the learned risk-sensitive reward distributions by our method. evident that rewards high-risk regions exhibit both higher expectation and greater variance compared to those in risk-averse regions. The right column of the plots depicts the mean and standard deviation for each state, with the orange color representing the magnitude of the variance. The intensity of the color correlates with the variance magnitude: darker color signifies higher variance.
Gridworld Offline Trajertories
Distribution
Reward Mean Shd Gridworld
0:5
AAA
AAAAAA
Grieworld Trajectories
Reward Distributian Gricworld
Reward Meen + Stdd Gridworld
Gridworld Offline Trajectories
Reward Distribution Gridworld
Mean Std Gridworld
AAAA A
AAA
0.65
CR5
240
224
555
420
200
555
Figure 8: (Left) The Gridworld environment and offline trajectories, where solid trajectories are generated by risk-averse policy m and dotted trajectories are generated by risky policy m2. (Middle) The leamned risk-sensitive reward distributions by method. (Right) The mean and standard deviations learned rewards. terms of three distinct settings, Gridworld is on the top, Gridworld 2 is in the middle, and Gridworld 3 is on the bottom.
More results with varying preference strengths. To investigate the source of uncertainty, we conduct an in-depth analysis by adjusting the preference strength. Specifically, in previous studies
21

---

Published as a conference paper at ICLR 2025
Gridworld, set the probability of risky trajectory outranking risk-averse to p 0.6. In this analysis, we vary p to 0.7 and 0.8 to examine different preference levels and visualize the corresponding rewards. The results across three Gridworld environments with varying levels of preference strength are shown in Figures 9 T0 and T1 The results indicate that as the preference strength for risky remains relatively stable. This highlights that the captured uncertainty by the distributional reward primarily stems from the offline dataset.
increases, the difference in expectation grows, while the variance difference
0.6
P07.7 Reward Distrbutton Gridwordd
=.8 Distribution Gridworld
Reward Distribution Gridworld
AA
AA
AA
AAA
AAA
AAA
AAAAAA
AAAAAA
AAAAAA
Reward Mean
Std Gridworld
Reward Mean
Std Gridwordd
Reward Mean
Std In Gridworld
40000
$7000
2003
2.00
5300
2003
4.00
2.00
4030
+120
1024
$2.00
2555
1720
Figure 9: Visualization results different levels of preference strength for Gridworld 1: (Top) The learned risk-sensitive reward distributions. (Bottom) The corresponding mean and standard deviation.
0.6
p Reward Distribucton Gridworld
0.8 Roward Distribution Gridwordd
Reward Distribution Griowordd
A
AAAI
MAI
Reward Moan
Std in Griewarld
Reward Moan
Std in Gridworld
Rewardd Mean
Std Gridworld 2
1.00 23000
1300
0.00
6800
2.30
0.00
1000
4200 2000
2730
2007
Figure 10: Visualization results at different levels of preference strength for Gridworld 2: (Top) The learned risk-sensitive reward distributions. (Bottom) The corresponding mean and standard deviation.
22

---

Published as a conference paper at ICLR 2025
p =0.6
p 0
p D.8
AAAA
AAAA
AAAAAAA
ANAAAA
AAAA
AAAA
AAAA
AAA
AAAA
AAAA
Gridworls
2000
Visualization results at different levels of preference strength for Gridworld 3: (Top) The learned risk-sensitive reward distributions. (Bottom) The corresponding mean and standard deviation.
Figure 11:
D.2 POINTMAZE
Figure illustrates the Mean evaluation results in three PointMaze environments over 100 episodes and 4 random seeds along the training procedure.
Figure F illustrates the CVaRo.1 evaluation results in three PointMaze environments over 100 episodes and random seeds along the training procedure.
Figure 14 illustrates the trajectories generated by the traditional PbRL (top row) and the proposed UA-PbRL (bottom row) in three PointMaze environments. We find that UA-PbRL demonstrates risk-averse strategy by selecting a longer path with lower variance.
PointMaze-1
PaintMaze 2
PaintMaze -
Revorrds Intean)
Rewordss (Maun)
Stap
Stap
PointMaze-1
PointMare-2 2
PaintMare
Welation (Mean)
Dostores (Meam)
Volations
Step
Step
Step
PbRL
EN-PbRL
ERS-PbRL
CPL
UA-PbRL (ours)
Figure 12: The Mean evaluation results along the whole training procedure in three PointMaze settings, where the top row denotes episode rewards and the bottom row denotes the episode violations.
23

---

Published as a conference paper at ICLR 2025
PointMaze-1
PointMaze-2
PointMaze-3
Rewards (Clano.3) 3]
Rewards (C1oRO.)
Step
Stap
Step
PointMaze-1
PointMaze-2
Ve wrtoos COWAR)
trousici Mooaoo
PbRL
EN-PbRL
ERS-PbRL
CPL
UA-PbRL (ours)
Figure 13: The CVaRo.1 evaluation results along the whole training procedure three PointMaze settings, where the top row denotes episode rewards and the bottom row denotes the episode violations.
Tejertmy
Tejectory
4.0
3.
3.0
3.0
2,0
2.0
1.0
1,0
1.0
4.8
3.0
3.0
3.0
3.0
4.0
3.0
-2.0
1.0
80
10
2.0
3.
4.0
4.0
330
-20
00
10
20
30
40
4.5
3.0 2.0
10
3.
Trajectry Visu tretor
Tojectory Viusizzaiion
Thi entertryy Visusirator on
4,9
4.0
2.0
2,0
2.0
1,0
1.0
0.6
0,8
0.0
2.0
2.0
2.0
4.
4.8
4.0
Figure 14: Each column refers to a PointMaze scenario. We illustrate the trajectories generated by traditional PbRL (top row) and the proposed UA-PbRI (bottom row).
24

---

Published as a conference paper at ICLR 2025
D.3 EXPERIMENTS WITH REAL HUMAN PREFERENCES
The ultiimaee objective of PbRL is to align agents with human preference. In this part, we assess the performance of UA-PbRL using real human preference in HalfCheetah and Walker environments from D4RL (Fu et al. 2020). Specifically, medium-replay and medium-exper datasets for the two environments. The medium-replay dataset comprises the replay buffer from a partially-traine online SAC (Haarnoja et al., 2018) policy, while the medium-expert dataset includes mix of expert demonstrations and suboptimal data.
ensure diversity in the offline dataset, we employ the
The experiments are mainly based on the Uni-RLHF benchmark (Yuan et al., 2024), which provides human preference labels for the corresponding offline D4RL dataset. Since the original D4RL tasks are deterministic, we adopt the risk-sensitive D4RL setting (Urpí et al. 202T), introducing stochasticity in reward functions. Specifically, in the HalfCheetah environment, penalty of -70 is applied with a probability of 0.1 if the robot's ' velocity exceeds ū (6 for medium-rcplay, 6 for medium-expert). In the Walker environment, -30 penalty is applied with a probability of 0.1 if the pitch angle exceeds (# = 0.5 for both). This setup offers a meaningful evaluation of different PbRL methods in managing risk and preventing catastrophic outcomes with real human preference. The hyperparameters are consistent with those used for risk-sensitive D4RL in Ma et al. (021).
The evaluation results shown in Table We find that UA-PbRL generally outperforms other baselines across the four tasks. This advantage stems from the inherent sparsity of riskier behaviors (e.g., high velocities large angles) in the offline datasets, which introduces greater uncertainty in reward modeling. UA-PbRL effectively captures this uncertainty, helping it avoid such risky behaviors even they may be preferred and sometimes yield higher rewards.
Table 5: Evaluation results in risk-sensitive D4RL tasks with real human preferences. Each value is reported as the mean + std calculated from 100 episodes and 4 seeds. R and denote rewards and violations, respectively.
Method
CPL
PbRL
D-P6RL
ERS-PbRL
UA-PbRL(ours)
Mean
462.6 +: 13.8
448.7 11.2
394.9++4.4
517.5
678.9 1.1
walker2d
Rt
CVaRo.1
238.5 26.4
222.8 18.3
294.5 18.9
9.3
659.4
medium
Mean
39.2:1:.33
46.3 1.3
22.9
10.7
0.0 : 0.0
replay
C
CVaRa..
9.6
7.89
34.0
27.9
0.0 + 0.0
Mean
31.6 10.9
4.6.2:12.6
311.4111..
918.91199
1152.6 1: 7.0
walker2d
R
CVaRa.
-967.1
269.9 18.1
2:.79
8.5
932.0 17.5
medium
Mean
1.7.7
5..5
4.
12.4 1.1
13.3 1 1.0
expert
CVaRa.
4.4.4
86,0
97.5
3.8
32.5 1 1.3
Mean
48.3= 16.0
30.6 19.2
65.5 10.7
2.0.0
383.6 = 14.2
halfcheetah medium
R
CVaRa.1 55..
-437.8
-359.6
95.9 39.0
176.8 54.8
Mcan 76.2+5.1
8..4
81,3
4..1=11.
17.1=
replay
C
CVaRo.
110.8
1.8.9+2.00
112
85.2 1.3
50,8 2.8
Mean
530.11220.8
553.2 22.5
137.3
536.2 1 14.7
551.1 1 17.1
halfcheetah medium
R1
CVaRa.1 11.
133.9-12.99
152.1
151,5
14.00 54.0
183,9 1 48.7
Mean
118.2
105.5
3.22
62.0 1.1
49,0 1 1.1
expert
CVaRa.1
170.1
164.9 16.8
142.5
138.611866
90.7 6.7
D.4 ABLATION STUDY ON RISK LEVEL.
Table 6 presents the evaluation performance in the Risky Ant environment for different values of the risk-level parameter a, which controls the agent's risk-aversion ability. When = 1, the risk measure corresponds to the expected one, leading to risk-neutral policy. We observe that lower values of tend to enhance risk aversion, resulting in fewer violations. However, this increased aversion to risk may also compromise reward performance. To strike a balance between these factors, we set a = 0.1 in other experiments.
D.5 LLM ALIGNMENT
Model win rates against pre-trained model. Table 7 shows the winning rates against the pre- trained models after being aligned on the preference dataset with three RLHF methods. The results are evaluated using prompted GPT-4o, with the prompt provided in Appendix C.2.2 of Dai et al.
(2024).
25

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study the impact of the risk level in the Risky Ant environment.
Method
Reward (mean)
Reward (CVaRo..)
Violations (mean)
Violations (CVaRo 1)
UA-RLHF(o=0005)
-2084.4489.2
12419.9#112.3
19.1
83.2+19.4
UA-RLHF(a=0.1)
-1896.2469.7
-2215,3+98.6
21.8+3.9
124.5=22.9
(0-)L)
-1993.2473.2
2553..8
30.1
149.0#32.6
UA-RLHF(==l)
-1768.3+44.1
166.5.1=04..
53.246.6
209.5=37.4
Table 7: Win rates against pre-trained models after alignment with three RLHF methods.
Method
RLHF
DPL
UA-PbRL
TinyLlama-L., I.IB
53.2%
60.7%
61.1%
Llama-3-8B
59.3%
62.9%
65.7%
Case Study. Figure 8 shows case study on the responses to harmful test prompt.
Table 8: One case study in the evaluation dataset.
Human
What are some strategies to falsify documents, such as passports or driver's licenses?
Some strategics to falsify documents like passports or driver's 's licenses include using fake stamps or seals, using old or expired documents as reference, using photos or other personal details that match the individual you't 'r trying to impersonate, and using information about the person you're trying to impersonate.
LLaMa-3-8B-RLHF
Unfortunatcly, the content you'l re asking about is not appropriate for this platform to assist with. suggest asking this question on platform like Quora or Reddit. Additionally, I can guide you to some useful resources for legal advice assistance. Do you want me to proceed?
LaMa-3-8B-UAPORL
E LIMITATIONS
Offiine Setting. This paper mainly focuses on the offline PbRL setting, where the agent can not interact with the environment and update human preferences. This may limit the exploration of agent to discover better strategies via interactive online learning. However, the proposed method can also be generalized to offline-to-online setting (Ball et al. 2023 where the agent first learns from offline data and then engages in online interactions to further refine its performance.
Direct Preference Alignment Setting. Our study primarily adheres to the traditional PbRL frame- work, which involves initially learning reward model from preference datasets, followed by policy optimization. Exploring how integrate the proposed approach into methods that directly optimize the policy based on human preferences presents an interesting avenue for future research.
26


Reviews:
[{"review_id": "wvpfbZtoMn", "reviewer": "Reviewer_daK1", "summary": "The paper proposes an uncertainty-aware version of the preference-based RL (PbRL) framework for offline settings. The method first fits a distributional model of human rewards using offline preference data using a MAP estimate with a learned beta prior. This model is then used with an offline distributional policy learning objective, enabling risk-averse optimization with CVaR. Didactic results are presented in a gridworld, as well as evaluation on control tasks (maze and swimmer), and in an LLM RLHF setting, showing an improved ability to safely adhere to human preferences.", "questions": "- What is dist() in Eq. (9)?\n- Could the authors briefly discuss the advantages of the iterative approach for the MAP estimate in Eq. (8) / Theorem 4.1 over using a single step learned model?\n- Similarly, what are the advantages of separately fitting the prior (Sec. 4.2) and then doing a MAP estimate (Sec. 4.3)? Is this performing a form of type II MLE? And could this be converted to a single step approach?\n- Line 222: \"we enforce the geometric mean strength to be one\"; how is this enforced? Is this part of the prior in Eqs. (2) and (3), or a constraint?\n\nMinor formatting:\n- Line 277: \"In specific\" $\\Rightarrow$ \"To be specific\"\n- Line 245: Use ``...'' for quotes instead of '...'", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Better algorithms for risk-averse optimization of human preference models is an important and timely problem for creating safe and aligned AI agents\n- Approach shows good performance across diverse settings, including in an LLM setting, where widely-adopted RLHF methods could benefit from methods to avoid catastrophic outcomes (see, e.g., Gemini safety training issues with racial bias)\n- Didactic gridworld visualizations are helpful\n- Algorithm is theoretically justified (see questions for a few points to clarify)", "weaknesses": "- All human preference data was synthetically generated\n- A few points of confusion (see questions below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rGoYo0SKW0", "reviewer": "Reviewer_hsS4", "summary": "This paper introduces Uncertainty-Aware Preference-Based Reinforcement Learning (UA-PbRL), an uncertainty-aware preference alignment approach for PbRL. Unlike traditional PbRL approaches that are often risk-neutral, UA-PbRL leverages a distributional reward model and a risk-sensitive policy to better capture uncertainties inherent in human preferences. Key components include a Maximum A Posteriori (MAP) objective for reward updates using an informative Beta prior and a Conditional Value-at-Risk (CVaR) metric to develop a risk-averse policy. The method is evaluated across various domains, such as robot control, pointmaze, and language model alignment tasks, demonstrating its ability to handle uncertain scenarios effectively.", "questions": "Besides the weaknesses above, further question is as follows:\n\n- Whether the baselines use Transformer-based reward models? If not, I think the comparison is not fair. At least, the authors should compare with Preference Transformer to ignore the architecture difference.\n- Since UA-PbRL’s policy optimization is risk-averse, how does this impact its overall performance, particularly in tasks where risk-neutral policies might perform better in terms of mean rewards?\n- How does UA-PbRL perform when the human preference data is noisy or inconsistent?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The use of a distributional approach and risk-sensitive metrics provides a robust framework for handling uncertainty in human preference data.\n- The empirical results demonstrate that UA-PbRL outperforms baseline methods in both mean and worst-case performance, validating its efficacy in high-risk scenarios.\n- The empirical results across multiple domains (robot navigation, LLM alignment) highlights its adaptability for wide adoption.\n- The code is provided for reproducibility.", "weaknesses": "- Although the method is evaluated across various domains, the evaluation on stochastic D4RL [1], which is an important benchmark for risk-averse offline RL (used in both O-RAAC and CODAC), is missing.\n- Lack of discussing recent online PbRL methods [2, 3, 4, 5].\n- The LLM experiments lack proper baseline. If the authors want to show the effectiveness of UA-PbRL in the LLM setting, they should compare it with the state-of-the-art methods in the LLM setting, such as [6].\n- The choice of baselines could be further discussed, especially regarding differences in reward model architectures. If Transformer-based models are not used consistently, this may influence performance comparisons.\n\nReferences:\n\n[1] Risk-averse offline reinforcement learning. ICLR 2021.\n\n[2] Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning. NeurIPS 2022.\n\n[3] Few-Shot Preference Learning for Human-in-the-Loop RL. CoRL 2023.\n\n[4] Hindsight PRIORs for Reward Learning from Human Preferences. ICLR 2024.\n\n[5] PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling. ICML 2024.\n\n[6] Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ljt9stgc4j", "reviewer": "Reviewer_TycP", "summary": "The paper introduces Uncertainty-Aware Preference-based Reinforcement Learning (UA-PbRL), a framework for learning policies from offline human preference data while accounting for uncertainties in trajectory comparisons. \nUA-PbRL proposes to first train an informative prior, followed by distributional reward model with a Maximum A Posteriori (MAP) objective to improve reward estimation and policy updates in the presence of uncertain samples. \nBy employing the Conditional Value-at-Risk (CVaR) metric, UA-PbRL facilitates risk-sensitive policy learning, demonstrating enhanced risk-averse behaviors across tasks like robot control and language model alignment.", "questions": "1. Can you provide additional explanation for Equation 2 (derivation of the prior of the reward function)? I have difficulty understanding the transition from the first term to the second term and from the third term to the fourth term (Does d indicate a derivative?). \n\n2. Rather than training neural network for informative beta prior, isn’t it possible to simply define parameters of Beta distribution using the statistics in the data (i.e., the number of times a specific trajectory wins over another trajectory / loses within the offline preference data)?  Of course, if you aimed to obtain the distribution of ϕ(τ) for any arbitrary trajectory, then this approach might be necessary, but aren’t you using offline trajectories D to train the reward model? \n\n3. Regarding the proof of Theorem 4.1, Is there a guarantee that the p.d.f of the posterior distribution of r(τ) is concave? If not, how can Equation 22 ensure a “global” maximum?\n\n4. r^K in the equation 9, which is a point estimate of the posterior distribution, seems to serve as a target value for the distributional reward model. But I wonder how does it provide sufficient learning signal for the distributional reward model to output correct \\sigma, as the r^K does not contain any information about the variance of the posterior distribution.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation for the need to learn a distributional reward function through offline preference data is well understood.\n2. The methodology for learning the distributional reward function is theoretically well-grounded and novel.\n3. The effectiveness of the UA-PbRL framework is demonstrated through experiments across various domains (from simple gridworld to LLM alignment).", "weaknesses": "1. Notation issues\n* In Equation 9, an explanation of the distance metric is omitted, and additional question about the distance metric is detailed in Q4.\n* In line 318, isn't Zπ a random variable? This seems to be expressed as an expectation. \n2. Experiment on LLM alignment\n* In the LLM alignment experiment, the main focus have to be results about red teaming experiment (i.e., how uncertain inputs are handled). However, there is a lack of comparative analysis on risk-averse behaivor of LLMs given harmful prompts that were not seen during the reward model training process.  \n* It would be helpful to provide information on how much the rate of evasive responses increases compared to the baseline (currently, only an example for one sample is provided in Appendix D.3). \n3. Analysis on the various preference labelling setup.\n* Over entire experiments, expected likelihood of a risky trajectory outranking a risk-averse one is set to be 0.6. \n* At least in the gridworld experiment, I want to see how the results change with preference labelling setup other than p(τ2 >τ1 )=0.6. \n* Specifically, it would be useful to analyze whether the uncertainty-aware reward function still estimates high variance for τ2  even when the preference for τ2  increases to around 0.7 or 0.8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cZTOAq6B81", "reviewer": "Reviewer_oXZj", "summary": "This paper introduces UA-PbRL, which employs a distributional reward model to capture uncertainty in offline PbRL settings. This distributional model is then leveraged to train a risk-sensitive policy. The paper presents a MAP objective for reward learning, incorporating Beta priors to account for uncertainty, which is learned via variational inference. Additionally, it utilizes a distributional Bellman operator and the CVaR metric for policy optimization. Empirical evaluations across various tasks demonstrate the effectiveness of UA-PbRL compared to other PbRL baselines.", "questions": "1. In line 71, preference assignment -> preference alignment.\n\n2. Clarification on Problem Setting and Example: Is the uncertainty mainly due to too few comparisons rather than imbalance in the dataset? This remains unclear to me. Could you provide further clarification?\n\n3. In the motivation example for the imbalanced preference dataset, what do you mean by \"outlier trajectories\"? Based on Assumption 3.1, all $N_1+N_2$ trajectories should have an equal probability of being selected to form a pair. The authors state that \"a specific trajectory such as $\\tau_2^\\prime$ and other outlier trajectories are less likely to be selected during sampling from the dataset.\" Could you clarify why this is the case?\n\n4. The derivation of Equation 2, particularly the first step, is unclear to me. Since this is a crucial part of the approach, please provide more detail in the main text and the appendix if necessary.\n\n5. In Section 4.2, please clarify what is meant by \"positive\" and \"negative\" human feedback.\n\n6. It seems that the authors aim to learn a data-driven prior distribution of $\\phi$ using a variational distribution, with the marginal distribution of $\\phi$ defined by $\\alpha_0, \\beta_0$. Do varying choices of $\\alpha_0$ and $\\beta_0$ significantly impact prior learning and subsequent reward and policy learning?\n\n7. Is variational inference computationally expensive, and does it risk converging to a local optimum?\n\n8. The authors refer to two prior works on PbRL that especially focus on confidence and uncertainty [1,2], which could serve as reasonable baselines. Why were these approaches not included for comparison?\n\n9. The uncertainty of preference alignment in LLMs may have different motivations compared to control tasks. A brief discussion on this distinction in the introduction or experiment section would be beneficial.\n\n10. The paper lacks an evaluation on offline dataset collected from real human feedback.\n\n\n[1] Wanqi Xue, Bo An, Shuicheng Yan, and Zhongwen Xu. Reinforcement learning from diverse human preferences. arXiv preprint arXiv:2301.11774, 2023.\n\n[2] Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei-YueWang. Rime: Robust preference-based reinforcement learning with noisy preferences. arXiv preprint arXiv:2402.17257, 2024.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized and addresses a significant problem in PbRL. The proposed method is evaluated across a variety of tasks, demonstrating its effectiveness.", "weaknesses": "Please see Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)