Risk-aware Direct Preference Optimization under Nested Risk Measure
Authors: Lijun Zhang
Lin Li
Yajie Qi
Huizhong Song
Yaodong Yang
Jun Wang
Wei Wei
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
LLM Alignment
Risk-Aware
Primary area: deep_learning
TL;DR: We introduce Risk-aware Direct Preference Optimization (Ra-DPO), a novel approach that incorporates risk-awareness by employing a class of nested risk measures.

Abstract: When fine-tuning pre-trained Large Language Models (LLMs) to align with human values and intentions, maximizing the estimated reward can lead to superior performance, but it also introduces potential risks due to deviations from the reference model's intended behavior. Most existing methods typically introduce KL divergence to constrain deviations between the trained model and the reference model; however, this may not be sufficient in certain applications that require tight risk control. In this paper, we introduce Risk-aware Direct Preference Optimization (Ra-DPO), a novel approach that incorporates risk-awareness by employing a class of nested risk measures. This approach formulates a constrained risk-aware advantage function maximization problem and then converts the Bradley-Terry model into a token-level representation. The objective function maximizes the likelihood of the policy while suppressing the deviation between a trained model and the reference model using a sequential risk ratio, thereby enhancing the model's risk-awareness. Experimental results across three open-source datasets: IMDb Dataset, Anthropic HH Dataset, and AlpacaEval, demonstrate the proposed method's superior performance in balancing alignment performance and model drift.

Risk-aware Direct Preference Optimization under Nested Risk Measure
Lijun Zhang', Lin Li', Yajie Qi', Huizhong Song', Yaodong Yang", Jun Wang', Wei Wei!
1. Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, School of Computer and Information Technology. Shanxi University, Taiyuan, Shanxi, China.
2. Institute for AI, Peking University, Beijing, China.
3. University College London.
Abstract
When fine-tuning pre-trained Large Language Models (LLMs) to align with human values and intentions, maximizing the estimated reward can lead superior perfor- mance, but it also introduces potential risks due to deviations from the reference model's intended behavior. Most existing methods typically introduce KI. diver- gence to constrain deviations between the trained model and the reference model; however, this may not be sufficient in certain applications that require tight risk control. In this paper, we introduce Risk-aware Direct Preference Optimization (Ra- DPO), a novel approach that incorporates aisk-awaresess by employing class nested risk measures. This approach formulates constrained risk-aware advantage function maximization problem and converts Bradley- model into a token- level representation. The objective function maximizes the likelihood of the policy while suppressing the deviation between a trained model and the reference model using sequential risk ratio, thereby enhancing the model's risk awseenssss Exper- imental results across three open-source datasets: IMDb Dataset, Anthropic HH Dataset, and AlpacaEval, demonstrate the proposed method's 's superior performance in balancing alignment performance and model drift.
Introduction
Learning from human feedback, serving as a bridge to align LLMs with human preferences, is crucial for ensuring that the generations are more helpful, factual, and ethical, among other desiderata [ 2 4. Alignment methods such as RLHF ( and DPO [5] have consistently proven more effective than supervised finetuning (SFT) alone. Notably, DPO, featuring simple and straightforward training process, directly uses the likelihood of the policy to define an implicit reward fitted to the preference data, which has emerged as popular alternative since it bypasses explicit reward modeling challenges while delivering competitive performance. Subsequently, variety of DPO variants have been proposed, such as f-DPO (6, IPO 77. RDPO [8, and SimPO [9), enhance performance. However, a key limitation of these methods is that they only consider evaluation the sentence level, ignoring the fact that the generation of these responses occurs sequentially, following an auto-regressive approach.
Recently, a fresh perspective on LLMs alignment has been introduced, specifically sequential and token-level direct preference optimization known as TDPO [10]. This method allows for examining divergence in relation to reference model on a more granular, token-by-token basis. Specifically, inspired by Trust Region Policy Optimization (TRPO) [11] from reinforcement learning (RL) field 12 13,
TDPO redefines the objective of maximizing restricted rewards in a sequential manner and
"Correspondence <weiveidaxu. edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

bridges sentence-level reward to token-level generation through the Bellman equation. However, since the objective at each step is to maximize the expected return, a iiss-neutral criterion, which neglects the characteristics of the reward distribution beyond the mean, TDPO cannot guarantee low risk of deviation from the reference model during alignment training. This could be catastrophic for practical applications, as a significant deviation from the reference model typically implies the degradation of superior decision-making and reasoning capabilities.
Fortunately, in the field of RL, a series of risk-sensitive methods 14 15 [6] have been proposed that achieve superior performance by introducing various risk measures. Recently, some researchers have attempted to introduce this technology to align LLMs with human preferences. For instance, RA-RLHF [17] introduces static risk measure into the fine-tuning of RL, while KTO [18] introduces prospect theory [99] to fit human choice behavior when faced with uncertain events. However, these methods only consider the risk at the sentence level by analyzing the distribution characteristics of the preference data, thereby overlooking the inherently sequential and auto-regressive process of response generation.
In this paper, we focus on the risk in token-level generation when aligning LLMs with human values and intentions. Specifically, from risk-sensitive perspective, we investigate novel direct preference optimization method and provide corresponding theoretical and empirical results. Our main contributions are summarized as follows.
We design a new risk-aware, token-level objective function and prove that maximizing this objective leads to policy improvements. Furthermore, by deriving the mapping from the risk-aware state-action value function to the optimal policy and establishing the equiva- lence between the Bradley-Terry model and the Regret Preference Model, we obtain an optimization objective that is solely dependent on the risk-sensitive policy.
We propose a novel Risk-aware Direct Preference Optimization (Ra-DPO) method. The method maintains natural and simple loss function, specifically, the sum of the DPO loss and the negative sequential risk ratio (see Figure T This loss function maximizes policy likelihood while suppressing deviation from reference model through the sequential risk ratio, thereby enhancing ask-awarensss in striking a balance between alignment performance and model drift.
Experimentally. we evaluate the effectiveness of our proposed method across various text generation tasks and assess its sensitivity to the risk control parameter. The experimental results demonstrate that our method can effectively suppress the risk of model drift while enhancing its performance.
2 Preliminaries
2.1 Preference-based Policy Optimization
Considering preference-based language model fine-tuning task, let denote input prompt (question), and y denote the generated response (answer). The notation Vo > Y1 | x symbolizes the human preference data, where yu (win) represents a response that is more preferred by humans compared to (lose). Both x and / are sequences of tokens.
Bradley-Terry Model. In preference-batee based fine-tuning, to align with human preferences, prefer- ence predictor adhering to the Bradley-Tere (BT) [20] model has been widely employed for pairwise comparisons. The likelihood of preference pair is commonly expressed as:
exp (r* (xr, yw))
PET (yw yt x) exp (r* (I,,W)) + exp (r* (x, 3))
where (x, yw) and r" rx, 31) stand for the reward function at the sentence level from the preferred and dispreferred answers, respectively.
Direct Preference Optimization.
DPO [5] begins with the following RL objective:
max ExpDD [Ey>) [r (a. v) (3kr.. (TT( [x)]
x))]]]
(2)
where represents the human preference dataset, /3 is the coefficient of the reverse KI. divergence
penalty, Teet (- Iz) is the policy of a fixed reference model (typically selected to be the model that

---

has undergone post-supervised fine-tuning), and To (-Iz) represents the policy of the trained model, initialized with TTg Wref-
By reparameterizing the reward function in Equation between the reward model and the optimal policy:
(2) DPO establishes a direct functional mapping
Tz (v x)
r(z, 3) = 3 log
+ 3 Blog ((x),
(3)
Tret
x))
where (x) the partition function. Subsequently, Equation 2 can be reformulated as DPO loss:
LDPO (To; Tret) E (m.uu:))>D [log a (u (I, Vw, Ue)l
(4)
where u (z, ywu 31)) = 3 log Na(tula)) (a) B log (.rr())2))) (eat(
se(url))
2.2 Preference-based Markov Decision Process
Preference-based Markov Decision Process (Pb-MDP) can be formulated as modification the classical MDP: M = (8, A., P. r, T), where S and A represent the finite state and action spaces, respectively; : A S is the probabilistic transition function; represents the reward function over the entire prompt-response which is defined as (S x A)" + R; y is the discount factor, and T denotes the length of a trajectory or episode.
Specifically, for language generation, the state 8; = [a, S E usually consists of the prompt and the generated response up the previous gePP and action 0e = y A corresponds to the current []] is an empty sequence. Therefore, we denote [x] = [x. [] "" For given prompt x and the first
generated token. Additionally, note that y
tokens y of the response y, the probability distribution of the next token conditioned on [z: 5 is denoted by TT(. [xx, (*)).
2.3 Risk Measure
It is more desirable to keep risk under control for language generation tasks rather than relying solely on risk-neutral criterion, which ignores the distributional characteristics of rewards, especially in applications that may have potential broad societal impact. Therefore, we introduce the risk-sensitive criterion [21 22 to quantify potential hidden risks. More specifically, we provide an introduction to the risk-sensitive function and nested risk measure as follows.
Risk-sensitive Function. In this paper, the risk-sensitive function is required to satisfy the following properties for all Z' E Z: Concavity: V E [0 (AZ + () > An (Z) + A) (Z'); Translation Invariance:
IR (Z cc) n (Z) + e. This
class captures a broad range of useful objectives, including the popular Conditional Value-at-Risk
(CVaR) 23 24 25) and Entropic Risk Measure (ERM) (26 27].
Nested Risk- measuress. In the context standard Pb-MDP, nested risk measures be expressed in Bellman equation type follows:
28
can
Qn (1x, y <t] "1,"²) (kr,, 2/ " vt) + tt (Va ((11)) (1x, y xx Ex [2x ([x, y yrt 5²)l
(5)
Va [[x, xT]
([x,,
where D(-) is a risk measure function with a risk control parameter μ, Qx (f y²) and V, (Ix,, y (1) represent the state-action value and state value under the nested risk measures at timestep & E [1..... ,T, respectively.
Due to space constraints, we provide detailed survey risk measures in Appendix A. and the expanded version of the value function definition Appendix A.2
Methodology
This section proposes a novel language model alignment method named Risk-aware Direct Preference Optimization (Ra-DPO). Specifically, we first analyze the characteristics of nested risk measures and design new risk-aware token-level objective function by reformulating the constrained reward

---

maximization problem into token-level form. Subsequently, we prove that maximizing the objective function leads to policy improvements. Then, an optimization objective solely related to the risk- sensitive policy is obtained by deriving the mapping from the risk-aware state-action function to the optimal policy and establishing BT model equivalence with the Regret Preference Model. Finally, we conduct formal analysis of this optimization objective in terms of derivatives and derive the loss function for Ra-DPO.
3.1 Risk-aware Objective Function
In this subsection, we aim to design a new risk-aware objective function for preference-based language model fine-tuning. Unfortunately, although the recursive Bellman equation under nested risk measures was introduced in Subsection 2.3 it cannot be directly applied due to the following reasons: (1) For the Pb-MDP setting, the algorithm can only obtain the reward (an implicit reward fitted to the preference data) over the entire prompt-response and thus cannot compute the target value each step. (2) The nested risk-measures incorporate Bellman-type recursion and are not law-invariant 31. making them complex and difficult to compute.
To surmount these obstacles, straightforward approach is introduce the state augmentation method, that is, to reconstruct an augmented Pb-MDP [30], where the state at each timestep includes prompt and the first (- tokens e of the response. This approach has the property that the state at the previous timestep is subset of the state at the current timestep, i.c., x y x C [z * This approach can reformulate the recursive Bellman equation into classical Bellman equation while satisfying the standard requirements for transformer-basee
long-sequence modeling in LLMs. Therefore, this paper, we directly define the state as a combination the prompt and the generated response up to the current step to model the sequential and auto-regressive generation. Then, the nested risk awsre objective's 's Bellman equation in Equation
(5) can be rewritten as:
Qn (Jr, "I ,''')= d2 (V- (yt (1,,""),,")))
Vn (1z, y ( = Ex @r (1I, 1,88))
(6)
Vi ([I, = R ([z y<))))))
where Qr ([x,, "0 ,y²) and V. ([x; (Ix,y=') = represent the risk-aware state-action value and state value under the policy и, respectively. The operator o denotes the concatenation of the state and action.
is noteworthy that there is a significant difference in the calculation of Va ([x, x²*) and Va ([x, y (J). According to Lemma 3.6 in [30], we can obtain the following lemma, whose proof is provided in Appendix B.I
Lemma 3.1. For ggiiven Pb-MDP, the reward over the entire prompt-response posed as r El=rg x R ([x, ae.gr """)""). the relationship between the state value function Equation 55 and Equation (6) is as follows: Va ([x, =11 = Vz ([x, x**) + R1--1, where R1:-1 = Ei R ([z, [zz.y" yh) .2²)) denotes the cumulative reward of the 1 ~ t steps of the prompt-response, and V [x] and [=[ are equivalent.
be decom-
Subsequently, based on Equation 6 we define the risk-aware advantage function as follows.
Definition 3.2. For risk-sensitive Pb-MDP that satisfies the Bellman equation in Equation (6). the risk-aware advantage function can be defined as:
A ([., 5 22) = Qn (I. - "V(V, ((r,)~9)),
(7)
where = ~ no (- [x, (
The definition is reasonable: its derivation is provided in Appendix B.2
Furthermore, based on the we propose a new risk-aware objective
definition of risk-aware advantage function Definition 3.2
function:
max E. Et (11 Anne (Ir,8-1.2) - 3DKL (To (-1 ||rt
(-1
[z.
The objective function maximizes a risk-sensitive advantage function subject to a KI. divergence constraint, which accounts for risk during selecting the policy, thereby striking a better balance

---

between alignment performance and model drift. It is worth emphasizing that maximizing the risk-aware objective function in Equation (8) leads policy improvements, as stated the following lemma, whose proof is provided in Appendix B.3
Lemma 3.3. Given two policies a and n², for any state 8t = [ y Rgww A (12, y 5-1 (.. .2,]
V+ ([+)) 2 BEND [+([=))]
then we can conclude: EEED
3.2 Risk-aware Preference Optimization
In this subsection, we convert the BT model into risk-sensitive token-level representation, which is divided into steps: (1) derive the mapping from the risk-aware fate-action function to the optimal policy; (2) establish the equivalence between the BT model and the Regret Preference Model.
Specifically, starting from Equation (8). the mapping from the risk-aware state-action function @a to the optimal policy Th can be derived as stated in the following lemma.
Lemma 3.4. The constrained problem in Equation
@ has the closed-form solution:
Trof (2 [ar, y (1) exp (# (1r (=,,)=1)2))
Th (= (x,y
(9)
Z (1z, 35(]33)
E Ee (1 2n,e (ay (==) is the partition function.
where Z
The proof is provided in Appendix B.4, Then, by rearranging Equation (9. we obtain the expression of the risk-aware state-action function terms of the policy:
@xjec (I +)) 3log Tt (= [xz, y "I)
Tref (= 5T) Blgg Z ([x, 1
(10)
Subsequently, by utilizing the reward decomposition formula Σ Et=l - 1 R ([x, " from Lemma 3.T we establish BT model equivalence with the Regret Preference Model as shown in the following lemma, whose proof is provided in Appendix B.S
Lemma 3.5. Given a reward function r(x, y) of the entire prompt-response based on the relationship between the token-wise rewards and the reward function r(x, y) = Ei= - ([z, x<'))y") we can establish the equivalence between the Bradley. Terry model and the Regret Preference Model, i.e.
Ti Σ
TB Ar
PBT (y1 > 3/2 1==== =a
An ([,, I
v1)
Σ
([ 21 155
(11)
1=1
(=1
where (2) 1/ (1 (x)(()))()- the logistic sigmoid function for any random variable 2.
According to the definition of the risk-aware advantage function Definition 3.2, we can directly
establish the relationship between the optimal solution in Equation (10) and preference optimization objective in Equation [T].. this way, we reformulate the BT model to be directly tied to the risk-aware optimal policy лे and the reference policy Tref, which is summarized in the following theorem, whose proof provided in the Appendix B.6
Theorem 3.6. Given prompts I and pairwise responses (y1, U2), and the risk-aware objective function in Equation (8), the Bradley-Terry model expresses the human preference probability in terms of the risk-aware optimal policy Ti and reference policy Trefi
PisT (y1 > y2 | xx) = (u" (x, 31, (1) - (x, V1, (2)),
(12)
where u (x, y1, 31) represents the difference in implicit rewards defined by the risk-aware policy n and the reference policy Tref, weighted by 3, represented as:
To (yn x)
no (y2 x)
(x, V1 V2) = =lggg
Tref (y1 xx) log
(13)
Feel (3/2 x)²
and 8 (xx U2) represents the difference in sequential risk ratios between two pairs (z, v1) and
(x, v2), expressed as:
(x. V1, (2) = 3D (I, y23 Tref Te) -DDggR (I, 915 Tref Ta),
(14)
where DSeeRR (x, Tret TT) = ΣT t Duttrrr log me(e (I.)
5

---

3.3 Loss Function and Formal Analysis
Drawing on Theorem 3.6 we reformulate the BT model into a structure solely relevant to the risk- sensitive policy, which enables us to formulate a likelihood maximization objective for parametrized policy Te. The loss function given by:
DRo-DPO, (TT8 Tret) (.. [log a (u (2r, Mes yt) 5 (xr, Mue v1))
(15)
In Equation 15) the sequential risk ratio is explicitly introduced into the loss function, which incor- porates risk-awareness to balance alignment performance and model drift. To elucidate the benefits of the proposed method, we conduct the further analysis of the loss function and its corresponding gradient. For brevity, we use u to denote u (z, Bw. Mr), and to represent å (æ, you 31). By simple calculations, we can derive the gradient of loss function in Equation 15) with respect to parameter 0:
V DPO, (To: Tref) (a. [(-u Vou Voo]],
(16)
where (-u+ 6) serves as the weighting factor for the gradient.
From Equation (16), we can observe that the first part, (-u), corresponds to the weight factor in the first part of loss function of TDPO. Its value increases when the language model makes prediction
errors relative to human preferences, i.c., log seer(un (z) > log _e(p=le) [x) The second part, 8, consists
me(yi
of the difference between the sequential risk ratios of the dispreferred and preferred response subsets, which a distinctive component of our method. When selecting a convex function (risk-averse), such CVaR, as the risk measure, our method can automatically control the risk ratio balance.
Furthermore, building upon a common objective shared by our method and TDPO [IO], i.c., reducing risks stemming from model drift and ensuring training stability, we further provide a second version of our method, Ra-DPO2. The loss function of Ra-DPO, given by:
CRo-DPO, (IT; Tref) (r, .DD [log a (u (z, Yuu 31)
ad² (Ir, Yuu (1))]
(17)
where 82 (I, Ni.
= Re (I, 3/2 Tref TT)))
sg D3 (x, 1/1 Tref
TT)).
The operator sg represents the stop- gradient operator, which blocks the propagation of gradients. The pa- rameter 8 can control the deviation between RR (z, 12 Trof TT) and
Coro (Ro:ane)) -E logo
(Aloog
Toe (Mia [x) Blog (su 2))
Tpet (2/we
(sh [x))
(3DSoRR (Ir, y1; Tref | mo)). Ra-DPO2 modifies the loss function of Ra-DPO, by disabling the gradient propagation of
Croeo (Fot) loog
(Aling
no (Mke [x)
Fext (9n xx) sg (BoqgL (Jt, Ves Seat |eat (1)))
(stw ax)
log No (3u 2x)
-a (BDps (ct, an Reaellite)
DseqRR (z, ywi 7eel | тe) and treating it as a baseline term for alignment of DseaRR (z, yt; Tref TT). The aim of the modification is to ensure training sta- bility, rather than accelerate training speeding. To summarize, the comparison of the loss functions for DPO, TDPO,, and Ra-DPO, is shown in Figure []] In addition, we provide a procedure our method, and provide its pseudocode (Algorithm 1 in Appendix B.7
TRUTOPO (Mat -E liogo
(Me And (8km [x)
log (sx ax)
(9log
Teat (M)
-a (SDaqo (x., Tesellaa (3 (2t, Dusi We (())))))
Figure I: Comparison of loss functions for DPO, TDPO2 and Ra-DPO; methods. The sy denotes the stop-gradient operator.
4 Experiments
We empirically evaluate our method on several open-source datasets and pre-trained models, aiming to investigate the following questions: (I) How does the performance of our method compare with that of existing methods, particularly in terms of risk sensitivity when handling challenging text generation tasks? (2) How does the risk control parameter yu affect the performance of our method?
To answer these questions, conducted experiments IMDb Dataset (32. Anthropic HH Dataset [33], and AlpacaEval [34] for three different text generation tasks. Based on the origi- nal KTO implementation we trained Ra-DPO and baseline models using the same hyperparameters.
Available htttps://githbb http://cabb..coooooooaaattaaaaaa

---

115
TDPC:
13-
(root 0.20.0
(--rr) 0.20.2
60-
Step
Step
Step
(a)
(b)
(c)
The experiment on the IMDb dataset with GPT-2 Large serving as the base model. (a) and (b) present the progression of sequential KL divergence (the lower the better) for both preferred and dispreferred responses. (c) illustrates the reward accuracy curves (the higher the better).
Figure 2:
3
TOPOs #
Na OPONIVVIN =- 2-03p= 2.99
cano
Na DPO ACVAN)
p-0 p-0.pp 8.08
Re-DPO
D.5p- 2.97
Ro BPOUCCWRR) 0.5p= 0.95
6.00
20c
140 185
X56
100
37th
140
342
Soop
(a)
(b)
(c)
TDPO 2
Ro (POJERM) a=0.5 5m
Ra OPOVERM) a-05
Re.DPOATRMI a-055
4500
4412
2h.
48c
22:
426
68c
123c
3406
100
23c
430
63c
2004
10k
39k
500p
Stop
Step
(d)
(c)
(f)
Figure 3: The experiment on the Anthropic HH dataset with Pythia-1.4h serving as the base model. Left and Middle present the progression of sequential KL divergence (the lower the better) for both preferred and dispreferred responses. Right illustrates reward accuracy curves (the higher the better).
Specifically, for Ra-DPO, employed nested risk measures based CVaR 24 and ERM (7
We compare our method against the following algorithms: (1) DPO [5]. which considers evaluation the sentence level; (2) PPO
(35), an offline PPO variant provided by the original KTO implementa-
tion; (3) TDPO; and TDPO, 10, which convert the BT model into token-level representations: (4) KTO 18), which considers preferences in human decisions that are not aimed maximizing utility.
Experimental setup and results are reported in Subsections
and Appendix
4.1 Experiments on IMDb Dataset
Experimental Setup:
The IMDb dataset controlled semantic generation dataset within the context of movie reviews, serving as a valuable resource for training and evaluating sentiment analysis models. We employ GPT-2 Large 36 the base model and use the model checkpoint insub/gpt2-large -ine---DDb--ffeuuieee with risk control parameter μ E (0.99, 0.98, 0.97, 0.95) are shown in Figure
as the SFT model. The results of the versions of Ra-DPO, (CVaR)
Evaluation: Figure 2 shows that Ra-DPO, can outperform or achieve reward accuracy similar to the advanced TDPO algorithm while reducing model drift (i.e., lower sequential KI divergence), demonstrating the risk-awareness of Ra-DPO, in balancing alignment performance and model drift.
4.2 Experiments on Anthropic HH Dataset
Anthropic HH dataset contains 170k dialogues between human and automated assistant, where each transcript ends with a pair of responses generated by an LLM along with preference label denoting the numfe-preferred response. We Pythia- 1.4h and Pythia-2.8B [37 as the base models to test our method on Anthropic HH dataset, respectively. The
Experimental Setup:
IMDb-fine- tuned

---

69
OPOACVaR)
66
2=037
OPOxCVaR)
200
455
Conffrient
Conffident
63
(a)
(b)
(c)
0.
BROJERMO u
00-
Seavnd Aecurery
0:57
BERM p-?
6.3
SPO 5
03
85 677 05 05
05 36
03
63
24
35
66
05
(d)
(e)
Figure 4: The experiment the Anthropic HH dataset with Pythia-1.4B serving as the base model. Left and Middle presents the sequential KL divergence (the lower the better) for preferred and dispreferred responses, while Right presents the reward accuracy curves (the higher the better) under {0.3, 0.5, 0.7, 0.9).
reference models are trained by fine-tuning the base models on chosen completions. The results are depicted in Figure Figure 4 and Appendix C.4
Figure shows the performance of TDPO, and different versions of Ra-DPO, with respect the risk control parameter fa while keeping coefficient constant 0.5. Figure 4 presents the statistical results of different algorithms with coefficient a = {0.3, 0.5, 0.7, 0.9), and the cor- observe that Ra-DPO, almost always achieves superior performance (higher reward accuracy) and maintains min- imal model drift (lower sequential KL divergence), under both CVaR-based and ERM-based nested risk measures. Additionally, Figure illustrates that Ra-DPO, is highly effective in suppressing deviation from reference model, particularly when a takes on smaller values. We hypothesize that this phenomenon may be attributed to the fact that the sequential risk ratio accumulates step-wise risks through nested risk measures, enabling remain responsive to significant model biases even carries a relatively low weight.
Evaluation:
responding curve plots are provided in Appendix (C44 From Figures Bl and 4
when 8g (I, /1, 8/2) in Equation (17)
4.3 Experiments AlpacaEval
Experimental Setup:
To comprehensively evalu- ate Ra-DPO, in terms of generation quality, we conducted pairwise comparisons on AlpacaEval us- ing models trained on the Anthropic HH dataset. Following the official AlpacaEval implementation" we sampled responses with temperature coef- ficient of 0.7. The winrate comparisons based on oasst Biiusts_yhhi__a 1 are summarized in Table and Appendix C.4 Both winrate and length- controlled winrate (Le winrate) are evaluated based on oasst paast_pythia_llll _2b.
quality,
Table I: The compare between different Al- gorithms and gpr4 1106 preview.
Method
Winrate
Lc winrate
DPO
$1.11 1.9
44.7. 0.4
PPO
52.1 1.8
51.9 0.5
KTO
51.5 1.8
50.2+ 0.6
TDPO:
51.9= 1.8
53.0= 0.6
TDPO;
52.2 1.6
52.24 0.5
Ra-DPO,
53.5
1.8
$3.9 0.5
Ra-DPO:
52.1 1.8
55.7+
0.5
that under the two indi- cators of winrate and length-controlled winrate, most of the implemented algorithms can outperform the common default baseline gpt4_1 _____100__peee_ _preview (DPO is more prone to generating long responses). Among them, Ra-DPO, and Ra-DPO, demonstrate the highest level of performance, especially when it comes to the length-controlle winrate indicator.
Evaluation:
12b- epoch- 3.5

---

5 Related Work
5.1 LLMs Alignment
With the development of LLMs, numerous researchers have encountered challenges stemming from the misaligned next-token prediction task used in the pre-training stage [33 38 39 40) particularly in balancing adherence to human instructions (explicit objectives) with the pursuit of being helpful, honest, and harmless (implicit objectives). Therefore, typical post-training stage, referred to as preference optimization, is commonly performed to align pre-trained language models with human intentions, and has become an indispensable aspect in the fine-tuning of LLMs. Most approaches 41 6 9 only utilize KL divergence at the sentence level to limit significant deviations from the reference model. However, the generation of responses occurs sequentially, following an auto-regressvve approach. Recent works [TO] 42) introduce a fresh perspective, specifically the token-level direct preference optimization, which allows for examining sequential KI. divergence in relation to a reference LLM. However, due to their neglect of reward distribution characteristics other than the mean, these methods suffer from the trouble of being insensitive to risk.
5.2 Risk-aware Reinforcement Learning
RL has made groundbreaking achievements
12 43
through approaches such as Q-learning 45
and policy gradients 35
in sequential decision tasks, but it also faces challenges when applied
in the real world [22 46 47 A primary reason is that the risk-neutral criterion (maximizing the
expectation) ignores the characteristies of reward distribution other than the mean, which may be important for certain systems, especially in applications requiring tight risk control (28 [5]. In order to tackle this challenge, two types of risk measures have been introduced: nested and static risk measures. Static risk measures 48
49 50] are straightforward to interpret, but the
resulting optimal policy may not remain Markovian and may become history-dependent. Nested risk measures [5I] 29 30 utilize MDPs to ensure risk sensitivity of the value iteration at each step under the current state, resulting in a more conservative approach. In this paper, we prefer nested risk measures because they recursively adhere to the Bellman equation and allow the MDPs to be reconstructed through state augmentation, enabling them to remain Markovian.
5.3 Risks in LLMs Alignment
When aligning LLMs with human preferences, there are many factors that may pose risks, primarily
encompassing the following three types: (1) There may be conflicts among human preferences [52).
or human preferences is inherently affected by contextual choice effects (53), thus introducing uncertainty the objectives when aligning models with human preferences. (2) Humans do not make decisions by maximizing their expected value for uncertain events; instead, they perceive random variables in biased but well-defined manner [8 19. (3) Many popular methods, such as DPO (5). RDPO R. and simPO (9) introduce the new risks during the alignment training process because they only consider the mean of reward or utility, which is risk-n and does not capture the distribution characteristics of rewards efficiently. In this paper, we focus on the third type of risk.
6 Discussion
The core objective of preference optimization make models less harmful, more helpful, and more truthful. DPO [5] and SimPO [9] serve representative examples of reference- based and reference- free preference optimization methods, respectively. Although SimPO not only achieves superior performance but also significantly reduces memory consumption, several studies have also pointed out the following limitations: (1) the lack of reference model reduces training robustness and necessitates stricter conditions to prevent catastrophic forgetting; (2) SimPO introduces dual parameters, which introduce additional complexity on hyperparameter tuning. Therefore, a comprehensive comparison in terms of performance, stability and robustness, hyperparameter tuning complexity, and computational efficiency reveals that each approach has its own trade-offs. Here, we would like to emphasize:
For preference- tives and model fidelity still necessary, although the (reeene)))f) model may not be "safe" or "correct". For example, in LLMs safety alignment tasks, simple objective is to
language model fine-tuning task, trade-off between alignment objec-

---

enable the model to reject unsafe responses while preserving original reasoning capabili- ties [39 54] response that is safe but logically incoherent or semantically uninformative is of little practical value. Therefore, many studies 41 typically formulate such tasks as constrained reward maximization problems.
KL. divergence has typically been used to penalize excessive deviations from a reference (critic) model [55] [56]. In fact, numerous studies 6 (5)) have reported that KL constraint offers many beneficial effects, such as balancing exploration and exploitation, ensuring stability and robustness, preventing catastrophic forgetting, and preserving the model's fundamental capabilities.
7 Conclusion
A pressing challenge arises for language generation tasks in the area of risk control, the models, once trained, are often required to interact directly with humans. In this paper, we propose a novel direct preference optimization method that incorporates risk awareness by introducing nested risk measures into the Bellman equation, to align pre-trained LLMs with human preferences. Specifically, we design a new risk-aware token-level objective function by reformulating the constrained reward maximization problem into a token-level form and then prove that maximizing this objective function leads to improvements in policy performance. Then, an optimization objective solely related the risk-sensitive policy is obtained by deriving the mapping between the risk-aware state-action function and the optimal policy and establishing BT model equivalence with the Regret Preference Model. Finally, we conduct formal analysis of this optimization objective and derive the loss function of Ra-DPO, which has practical implications for language generation tasks.
8
The Discussion of Limitations and Impacts
8.1
Limitations
This paper focuses on the risks associated with token-level generation when aligning LLMs with human values and intentions. Our main contributions include theoretical analysis (see Section 3 and Appendix B practical algorithm (see Appendix (B.7) and simulation verification (see Section and Appendix These results characterize the performance of the proposed Ra-DPO in terms of reward accuracy and sequential KI. divergence. Below, we discuss the limitations of Ra-DPO from both theoretical and experimental viewpoints.
Our theoretical results are based on a class of risk-sensitive functionals that satisfy concavity, monotonicity and translation invariance for any random variables Z, Z' e Z. Concavity implies risk aversion; translation invariance is an important condition for the validity of Lemma 3.1 Our conclusions may not be valid when such assumptions do not hold. Fortunately, this class captures a broad range of useful objectives, including the popular CVaR 23 and ERM (27]
Theoretical Viewpoint:
Experimental Viewpoint: Ra-DPO may not be fully effective for tasks such as harmful content moderation and toxicity detection. This is because our primary goal is to reduce the risk of impaired makingg and reasoning capabilities due to model deviation from the reference model during LLM alignment. However, it worth noting that the problem we are addressing both widespread and of significant importance. Furthermore, we recommend a safe or low-risk approach that incor- porates awkennsssssss within the Safe RLHF 39 or SACPO [54] framework. These approaches explicitly or implicitly model both cost and reward functions while accounting for cost distributions. may require more computational resources due to the need to train additional models. However, our method can serve as a solid foundation for such potential approaches.
8.2 Impact Statement
This paper presents work aimed at making LLMs more helpful. Specifically, we focus on how to reduce the risk of impaired becision-making and reasoning capabilities due to model deviation from the reference model during L.MM alignment. Our work has many positive societal impacts, such as providing a theoretical foundation for risk-aware language generation task, none of which we feel must be specifically highlighted. There are no negative societal impacts on work.
10

---

Acknowledgements
This work were supported in part by the National Natural Science Foundation of China (No.62276666 No.62376013,
in part by the Basic Research Program of Shanxi Province 1294), and the Shanxi Provincial Overseas Study Fund Project (No.20240002).
References
[1] Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurlPS, 2017.
[2] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[3] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (3KI::2307.09288 2023.
[4] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In AAAI, 2024.
[5] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In NeurlPS, 2023.
[6] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. In ICLR, 2024.
[7] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS, 2024.
[8] Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405./9316 2024.
[9] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint 240.::2405..4744 2024.
[10) Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. ICML., 2024.
[11] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.
[12] Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017.
[13] Vincent François-Lavet, Peter Henderson, Riashat Islam, Marc G Bellemare, Joelle Pineau, al. An introduction to deep reinforcement learning. Foundations and Trends@ in Machine Learning, 11(3-4):19-355, 2018.
[14] Sebastian Jaimungal, Silvana M Pesenti, Ye Sheng Wang, and Hariom Tatsat. Robust risk-aware reinforcement learning. SIAM Journal on Financial Mathematics, 13(1):213-226, 2022.
[15) Lorenzo Bisi, Davide Santambrogio, Federico Sandrelli, Andrea Tirinzoni, Brian Ziebart, and Marcello Restelli. Risk-averse policy optimization via nest-nl policy optimization. Artificial Intelligence, 311:103765, 2022.
[16] Eduardo Candela, Olivier Doustaly, Leandro Parada, Felix Feng, Yiannis Demiris, and Panagio- tis Angeloudis. Rwsk-a controller for autonomous vehicles using model-based collision prediction and reinforcement learning. Artificial Intelligence, 320: 103923, 2023.
11

---

[17] Sapana Chaudhary, Ujwal Dinesha, Dileep Kalathil, and Srinivas Shakkottai. Risk-averse fine-tuning of large language models. In NewrlPS, 2024.
[18] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In JCML, 2024.
[19] Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5:297-323, 1992.
[20] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[21] Nicole Bäuerle and Ulrich Rieder. More risk-sensitive markov decision processes. Mathematics of Operations Research, 39(1):105-12) 120, 2014.
[22] Yuheng Wang and Margaret Chapman. Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control. Artificial Intelligence, 311:103743, 2022.
[23] Philippe Artzner. Thinking coherently. Risk, 10:68-71, 11,8-71,10 1997.
[24] Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of risk, 2:21 42, 2000.
[25] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. In NeurIPS, 2015.
[26] Hans Föllmer and Alexander Schied. Convex measures of risk and trading constraints. Finance and stochastics, 6:429 (:429-447, 2002.
[27] Jia Lin Hau, Marck Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted mdps. In AISTATS, 2023.
[28] Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. Risk-sensitive reinforcement learning: Near-optimal sample tradcoff in regret. In NeurIPS, 2020.
[29] Yu Chen, Yihan Du, Pihe Hu, Siwei Wang, Desheng Wu, and Longho Huang. Provably efficient iterated evar reinforcement learning with function approximation and human feedback. In ICLR, 2024.
[30] Yujie Zhao, Jose Efraim Aguilar Escamilla, Weyl Lu, and Huazheng Wang. Ra-pbrl: Provably efficient risk-aware preference-basee reinforcement learning. In NeurlPS, 2024.
[31] Jia Lin Hau, Marek Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted mdps. In AISTATS, pages 47-76, 2023.
[32] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACI, 2011.
[33] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05880 2022.
[34] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 24Xiv:2404004475, 2024.
[35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06447 2017.
[36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
12

---

[37] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' BrBren,, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Skowron Raff, Sutawika Aviya, Wal Lintang, and Oskar van der. Pythia: A suite for analyzing large language models across training and scaling. ICMI., 2023.
[38]
Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for aiftty-aligmment arXiv preprint arXiv:2308. 206.:22300009662,, 2023.
[39] Josef Dai. Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rihf: Safe reinforcement learning from human feedback. In ICLR, 2024.
[40] Min-Hsuan Yeh, Leitian Tao, Jeffrey Wang, Xuefeng Du, and Yixuan Li. How reliable is human feedback for aligning large language models? arXiv preprint 0rXiv:2410.01957, 2024.
[41] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-graine human feedback gives better rewards for language model training. In NeurlPS, 2023.
[42] Yichen Ouyang, Lu Wang, Fangkai Yang, Pu Zhao, Chenghua Huang, Jianfeng Liu, Bochen Pang, Yaming Yang, Yuefeng Zhan, Hao Sun, al. Token-level proximal policy optimization for query generation. arXiv preprint arXiv:2411.000 1.00222, 2024.
[43] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 241::24422.030000 2024.
[44] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12448, 2025.
[45] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Ku- maran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 518:529-523 533, 2015.
[46] Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications: comprehensive survey. Artificial Intelligence Review, 55(2):945-990, 2022.
[47] Shangding Gu, Long Yang. Yali Du, Guang Chen, Florian Walter, Jun Wang. and Alois Knoll. review of safe reinforcement learning: Methods, theory and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12211))::216111 1123,, 2024.
[48] Yingjie Fei, Zhuoran Yang, and Zhaoran Wang. sens-ven reinforcement learning with function approximation: A debiasing approach. In ICML, 2021.
[49] Osbert Bastani, Yecheng Jason Ma, Estelle Shen, and Wanqiao Xu. Regret bounds for risk- sensitive reinforcement learning. In NeurlPS, 2022.
[50] Kaiwen Wang, Nathan Kallus, and Wen Sun. Near-minimax-optinne risk-sensitive reinforce- ment learning with evar. In ICML, 2023.
[51] Yihan Du Siwei Wang, and Longbo Huang. Provably efficient risk-sensitive reinforcement learning: Iterated evar and worst path. In ICIR, 2022."
[52] Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward-free rlhf. In NeurlPS, 2024.
[53] Sebastiaan De Peuter, Shibei Zhu, Yujia Guo, Andrew Howes, and Samuel Kaski. Preference learning of latent decision utilities with a human-like model of preferential choice. In NeurIPS. 2024.
[54] Akifumi Wachi, Thien Q Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto. Stepwise alignment for constrained language model policy optimization. In NewrIPS, 2024.
13

---

[55] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprini arXiv: /9099999999,,,... 2019.
[56] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barobskk,,,,,kkk,,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 (14(13)3351-3352 2017.
[57] Junshu Pan, Wei Shen, Shulin Huang, Qiji Zhou, and Yue Zhang. Pre-dpo: Improving data utilization in direct preference optimization using a guiding reference model. arXiv preprini arXiv:2504. 15843, 2025.
[58]
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335- 359, 2005.
[59] Mo Chen and Claire Tomlin. Hamilton-jacohi reachability: Some recent theoretical advances and applications in unmanned airspace management. Annual Review of Control, Robotics, and Autonomous Systems, 1(1):333-358, 2018.
[60] Philippe Artzner, Freddy Delbaen, Jean-Mare Eber, and David Heath. Coherent measures of risk. Mathematical finance, 9(3):203-23) 2-8, 1999.
[61] Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial intelligence, 147(1-2):163-222 2003.
[62] Daniel Lowd and Jesse Davis. Learning markov network structure with decision trees. In ICDM, 2010.
14

---

A Supplementary Materials for Section 2
A.1 Risk Measure: A Brief Overview
For quantifying and managing risks, three main paradigms [21 15] have been developed: the risk-neutral paradigm, the worst-case (i.e., robust) paradigm, and the risk-averse paradigm. The risk-neutral paradigm aims to find a policy that maximize the expected cumulative reward, but ignores characteristics of the reward distribution other than the mean, which can be crucial for systems with safety concerns, For example, system may need to operate in a way that mitigates harmful consequences, even in rare and unpredictable situations. The worst-case paradigm [58 59 focuses on finding a policy that satisfies the constraints of specific cost function, generally assuming that the maximum possible cost can quantify bounded adversarial disturbances. However, since the worst-case approach assumes disturbances are bounded, it may not work well when those bounds are hard determine.
The risk-averse paradigm 23 21 15,
an intermediary paradigm between the risk-neutral and worst- case paradigms, has garnered extensive attention. It describes individuals algorithms that prefer outcomes with reduced uncertainty by seeking to optimize risk metrics of the possible cumulative reward, emphasizing its distributional characteristics. In general, there are mainly two types of risk measures: nested and static risk-aware measures, each possessing distinct advantages and limitations. Static risk measures [8
49 50] straightforward interpret, but the resulting optimal policy On the other hand, nested risk measures [51 29 30) utilize MDPs to ensure risk sensitivity of the value iteration each step under the current state, resulting in more conservative approach. We prefer nested risk measures because they recursively adhere to the Bellman equation and allow the MDPs be reconstructed through state augmentation, thereby enabling them to remain Markovian and ensuring that policy choices depend solely on the current state.
may not remain Markovian and may become History-dependen
In this paper, we employ a class of nested risk measures, which are variants of the popular CVaR and ERM. Below, we provide introductions to nested risk measures and the CVaR and ERM risk functions.
Specifically, let FF be measurable space. A risk measure over X is function p : X + R that maps uncertain outcomes X E X' to the real line. A risk measure of the total discounted return G' can be described as:
min (G), well
(18)
where the dependence on æ emphasizes that the underlying probability measure is induced by the chosen policy. The simplest example is p² E=, for which Equation (88 reduces to the standard risk-neutral RL problem.
Nested Risk-measures: Consider time horizon of length T E T. A nested risk measure of random return G Go + G1 + ......... + GT takes the form
p(G) = po (Go + P1 (Gi + .... + pT +
(19)
where each P: is a risk functional, i.c., a map from a space random variables to R U(oo}. We now introduce the the CVaR and ERM risk functions.
Conditional value-at-risk (CVaR): CVaR with iisk-aversion level (0. has been defined
1 EF [[G - m)+1
PevaR (G: (G; a) min EEB
1
and it has several advantages over VaR: quantifics the losses encountered in the tail, it can be expressed as minimization problem, and is coherent risk measure [60].
Entropic risk measure (ERM): ERM is popular method for measuring risk, defined as:
PFRM B) log E 1
where 3 > 0 indicates the degree of risk aversion. Optimizing ERM is equivalent to optimizing an exponential utility function (EU):
Peu 15 1900)
15

---

In this paper, we formulate the nested risk measure within the Preference- Markov Decision Process (Pb-MDP) framework, and express in terms of Bellman equation type. To we uniformly denote all such nested risk measures collectively by &², a risk measure with a risk control parameter μ.
simplify notation.
A.2
The Expanded Version of Value Function Definition
The definition of value function for nested risk measure in Equation can be expanded as
Qr (1 11) = R ([0, y " v²) + Th (R ([0r, y yet+]] x 2x,
(++))))
(20)
4tt (...D4 t (R [Ix, "- (- [z, (e.yy1))
V (1 = 12 (1,, -M (-1 y + $t (R y |
ग [x, (())))
(21)
+ (... T (R (1=, ग [
Similarly, the definition of the optimal value function, can be expanded as
@ ([,r, y "] .3²) max { ([ar, y = v34) + &t (R ( ([r, y [ a (- (x,))()]]))
(22)
+ +h (... T (R (RE,, (= - (-1 [o,
V (10, = max { ([z,, y y²*] x (-1 [x, (I) + o" (R ([z, y stt]] x (. (x)))))))))
+ 4th (..-0" ( (Il [12,0 क (- [2, <T ))))))
(23)
B Supplementary Materials for Section 3
B.1 The Proof of Lemma 3.1
Lemma 3.1 Restated. For given Pb-MDP, the cumulative reward over the entire prompt-response can be decomposed as r = Σi= y r-1 R (Ix, y y²)), the relationship between the state value
function Equation
55 and Equation (6) is as follows: Vz ([x, "I) = Vz ([z, y =²1) + R1æ-11 where
1-1 R1--1 Sh= R [zz.0] y x<) )²) denotes the reward of the 1 ~ t - steps of the prompt-
response, and Va [xr] and [>[x] are equivalent.
Proof. First, according to [61 62 30
reformulate the Pb-MDP as decision tree-like MDP:
(I) The state transition graph of the Pb-MDP is connected and acyclic;
(2) Each state in the Pb-MDP corresponds to unique node in the tree;
(3) There is a single root node from which every other node is reachable via unique path;
(4) The transition probabilities between states follow the Markov property, i.e., the probability of transitioning to any future state depends only on the current state and not on the sequence of events that preceded it.
Formally, let be the set of states and Piy be the transition probabilities between states s; and sy. For an Pb-MDP with tree-like structure, the probabilistic transition matrix P is defined such that:
Pij > 0 if there is an edge between S; and 8j in the tree, and Ptz = 0 otherwise.
(24)
Moreover, for each non-root node sy, there exists exactly one s; such that Pty > O, and s is the unique parent of s, in the tree structure.
To differentiate the two value functions, we denote the value from Equation (6) as Vi ([x,8 (1 and the value from Equation (5) as Vn ([x, y (1). Since the reward of the entire prompt-response can be decomposed as Σi= 2 R (Ix, y ",*")"") we have the following relationship:
VA ([z, (1r,y x() Vn ([x, =t1) R
16

---

where R1--1 prompt-response. We prove this relationship by mathematical induction as follows.
2h=1 S ((zz, y 5²) denotes the reward of the ~ t - 1 steps of
Initial Case. Using the tree-like Ph-MDP and the initial conditions of the Bellman equation, at the final step T, we have
Va (1 =Va V (L0, TT [1, + RI-T-1
(25)
= Vi ([x, (T] + R1-T-1
Induction Step. We now prove that if VA ([z, ((a,)(tt]])
V ([z, Riæ holds, then
Vz [[x,y<*]] = Va (1x, y " + R1--1 also holds. Since this policy x on tree-like Pb-MDP is fixed, it has only one path arrive 1-th state (st [x, y ())) denoted
E (ST,1) = En (ST:2)
VST,I. ST,2 E {sr S. (ST) [z. (
Therefore, R1--1 is unique.
Vz ([x,y~(] [] Thd (V= ([0, [)+11) + Ry:)):
= 4h (Vr ([0, y ((+1]) + R ([ [2, x-
i (.1 y ( + R1a-1),
(26)
= 4/" (Vr ([x,, s
+ R ([z,, y <t
x (.1 [z, (])) + R1a-1-
Vx ((z,5 + R111-11
where the third equality holds because the risk measure function + satisfies translation invariance. Then, by applying this conclusion, we observe that when t 1, V [xc] V=[=] holds. Thus, we have proven that for the Pb-MDP, the reward over the entire prompt-respons can be decomposed as r EL x01
R ([x "1 v²). and V [x] in Equation (5) and Va [x]] in Equation ( are equivalent.
B.2
The Derivation of Definition 3.2
Definition 3.2 Restated. For a risk-sensitive Pb-MDP that satisfies the Bellman equation in Equation 6 the risk-aware advantage function can be defined as
Ar ([ () = Q- ([v, , -p"(V= ((,,))')))
where mo (- [x. (())
The Derivation. In terms of designing the objective function at the token level, TDPO [0 provides us with valuable insight by introducing the advantage function from the TRPO algorithm in RL field as the target for each step. Building upon TDPO, we consider the risk associated with language generation at each step and devise novel risk-sensitive advantage function. First, based on assumption that Et. 7t 2-1 ([z, =*] y²)), we can get:
R (0
1=1
r-1 (11 100 N"
x (Vo(( ([x, - 0" (Va ( [Ir, y s(+11))
th
Vin ([x))) + EE r-1 R ((xx,="))
".. v²) + 2 (is (1z, ((11))
Th (iz (L2, ((.))1))) - - क" (vx ([1,y<r+]])
+ Σ 1-1 (Φ. (1.,5=,,0) Th (Vz ([m - - T. T (v=((z))
-T (V= (xt,))+
(27)
17

---

Next, note that y² = EOS denotes the end of the text sequence. Therefore,
00
V ( (
En
Σ
*
(h st 12. 12,1 0. (28)
Furthermore, we have
T (Va ((x))
Σ x--1 (Ør (LE,0"" r" (Va (xx:)((1)))
(29)
So, we definite the risk-aware advantage function as An (1o (1r, (5 "0 @ (V= (12,))51))) where To(: [ar, (
B.3
The Proof of Lemma 3.3
Lemma
3.3
Restated.
Given two policies क and x', if for any state St
[2, y 5* Ezwz'
A (1x, y 2) 2 holds, then we can conclude:
E E Vi. ([x))) 2 EEnD [V>([t))]
Proof. Let == (x) y. uug.y/ yy ....... )²) denote trajectory, where the expectation E.jer is taken over trajectories generated by policy n'. We then have
ENDD
[=+(
ENND [+<[[>)]
=B -///
R ([I
(110)))))))))))))))) .0)) +
77 0h (v, ([r, (1=2)))11))) - Va ((a)))
1-1 R ( [z. (11 "1) + 7 " (v= ((., T" (v= ([.,, ([..)
(30)
1-1 A (1 y x (.,,,,,))) 13/1).)))
Byg'a Ar ( (1r, 2 [.)]]
=I Eay
Since for any state 8t = 20 BeNnY An (1x, y x1] x1) 2)]
2 0, so we can obtain
ExnD
[(+((=))) EEND
[Va([tt) 2 O.
This completes the proof of Lemma 3.3
B.4 The Proof of Lemma 3.4
Lemma 3.4 Restated. The constrained problem in Equation 8 has the closed-form solution:
क (= [m, Tzef (2 [x, y <t exp 2 (Ir, (0,"""1,2
Z (1x, y (39))
where Z (1z, (1 Bermar Q (L..))))))) is the partition function.
18

---

Proof.
max A ([v,, 5- .2) 3KL. (To (.1 [ y =1] Iret (- [+, y (T)
((Omas
- Vinees ([x, 2 ()) + 3 log
Trel (z [2x, y y To [x, y
max B so (1),)(111)
Tref (z ( e # ( (121))))1))
max & go t (11 log
Viior ([x,ySt)
(z [2, y
Tref (z v e a ( (1.=)))))))
max BE (-Ila,,y log
7 (Ir, 1/(0] ]; Fe (= v<()
Vanue (1Z, + log [[z, y 5r [ :3)
[a, y (() ed Arse (1)=))));;)
Frof (z
-mxx -3DKT. Th (x)
7 (1or, y (
- -aae ((2)))<() + 3 log Z
([x, [x, 5 ;B),
(31)
where 7 (125, (1 ;) is the partition function:
Z y< :3) (((aar()) exp (n (1z, (12,249)22)
(32)
Then, we can derive the relationship between the optimal policy and the state-action function:
Fzef (= (5,, exp ( (3 (1,,, ((r,,<*),22)
क (= [I,
Z (12, ;3)
This completes the proof of Lemma 3.4
B.5 The Proof of Lemma 3.5
Lemma 3.5 Restated. Given a reward function r(z, y) over the entire prompt-response. based on the relationship between token-wiss wise rewards and the reward function r(z.y) () Ei= 70 ""r,y"",200 y²),
establish the equivalence between the Bradley-Terry model and the Regret Preference Model, i.e.,
Ti E Ax ([
PBT (11 > y2 x))
v1)
([2, .1.6))
1=1
where (2) 1/ (1 + exp( 2x) is the logistic sigmoid function for any random variable
Proof. Recalling to the BT model in Equation
cxp (x. 81))
PET (y1 y2 x) exp (r (I, v1)) + exp (r (I, 32))
(33)
and the equivalence between prompt-response reward and the risk-aware advantage function:
T
= T (V=
([x)
+
7 1-1 (Qx ([0, ",,"" - T" (Vs (1,,<11)))
t=1
T
$. Vz= ((x])
+
r-1 Än (1, "1,
[=1
Then, we have
PET (v1 > y2 x))
Σ
1- Ax ([z, vr (1.1,111)
Σ t=1
r- An (1 21 .15)
This completes the proof of Lemmal
3.5
19

---

B.6 The Proof of Theorem 3.6
Theorem 3.6 Restated. Given prompts x and pairwise responses (2), and the risk ass-ewwre objec-
tive function in Equation (8), the Bradley-Terry model expresses the human preference probability in terms of the risk-aware optimal policy T and reference policy Tre!:
PT (vn > V/ x)) = a (u" (x, y1, y22 - 8 (1,
31. U2))
where u (2,, ;n y2) represents the difference in implicit rewards defined by the risk-aware policy m. and the reference policy Tref, weighted by ,3, represented as
xo (11 x)
To (32 x)
(I, M1, y2) = 3 log
8 log
Tree (y1 a)
Tref (y2 x)'
and 8 (ar, M/, 122) represents the difference in sequential risk ratio between pairs (s, v1) and (z, V2), expressed as
(a. U2) SDqquu (I, U2i (2; Tref xо) DDseqau (I, y2: Tref mo)-
Proof. According to the Lemma 3.4 we have
T (z [z, (z,y [)) Feet (= [2r, x2 exp ( (3 (10,, (1,,0"))
(34)
(1x, 3))
where Z 2[2,,1 ;
E Exwzin (11 Aroa (1=,v is the partition function. Rearrange
Equation 34 we obtain
To (z 1(
@mccc (I = 3log
3 log Z ([x, ;8).
(35)
Je (2 (1)
From Lemma
3.5 we can get
Ti
T2
(7 1 Ar ([or, 22 (10,125) ..66)) ) (36)
PIT (y1 > U2 x))
( An (1,
Σ 1=1
By leveraging Equation
35 we can derive
Axper ([o,, ([x,y=*]..0 x=²)
(Oroer (12 ,,') Tx (Vroc (Lo,, w(11))
Σ
1-1 (Qric (1r, ". $² (Ōrne (Ir .2))
(37)
1=1
T (v"
[z, x(
Σ 1-1 B log
+ -Blog 2 ([I, xx(
Tyal (y'
[2r, y=() [x. y =1)
Φ Blog To (z
BBlog 2 (1x, 21 :9) ))
Feel (z
[2., y())
Note that
E E.wrier
[oog Z ([r, : = Blog Z ([r, (1r,""})))) :)
Therefore,
Axne (1x,v=t).1)
Tg (3/
[2, ((
(log T (2
[x, y =t)
log
(38)
Tyel (1²
[x, 5(
Tree (z
[x. y y
z (y1
[xr x(
T
Teet (2 [xc y xt1)
1-1 Iu tl llog
=B
log
+ 3
Tref (y²
[2i, =(
T (z
(x. y<()
1=1
20

---

When substituting into the expression, we obtain a more concise form:
To (y'
[2, (
Tref (z (or y DF91)
A (1,,0""00" = log
Tl
llog
Tref (y'
[r, (5))
To (2
I y (21)
==
1=1
TØ (y x)
=B
log
Deggtt (x,, Tref mt)
Tee (y x))
(39)
Feed (ees(e|(oa55)) znotior (log
where DeeRRR (a, yı Tree
TΦ) Ex=
$
To(2 (2,05-9))
Then, we let
To (Uh x)
To (2/2 x)
u (I, M1, 92) = 3 log
B log
(40)
(11 xr)
Tref (y2 x)
(z, 92) B.D (Ir, y23 Tref Te) ReeRRR (I, y15 Tref Te).
(41)
Substituting Equation 339 into Equation (36). we arrive
BT (y) y/2 x)) (u" (1, 8 (2,,1,
This completes the proof Theorem 3.6
B.7 Algorithm
In this subsection, provide the main pseudocode for Risk-aware Direct Preference Optimization (Ra-DPO), as outlined in Algorithm
Algorithm Risk-aware Direct Preference Optimization (Ra-DPO)
Input: Reference model Tref, Policy model тe. Coefficient o, B, Risk control parameter Learning
Input: Dataset D = { (ar, You, wn)*) of size N, Method M
Initialize:
My +- Tef
for each epoch do
Sample mini-batch Dm [x, Ywu y1) M m==1 from D Predict the probabilities To (yhe x)) and xo (v1 x) for (xe, Bwos Bt) in the mini-batch Dm using the policy model
Predict the probabilities using the reference model
(Nwr I)) and Mhat (m xr) for (a, Uwr w) in the mini-batch Dm
Calculate the function u (I,, Mue (1) 3 log me me(uml>))
- 3 log =e(ud)))
Compute the sequential risk ratio DseqRu (a,, Ywi Wref
No) for (ar, Fw in the mini-batch D.
Compute the sequential risk ratio DseqRR (x, yr3 Tref if Method M is Ra-DPO, then
TT) for (xr y1) in the mini-batch Dm
Calculate å (ær, Yors 31) ) = BDoRRR (ar, y15 Traf | 0 7) V V (w. [log a (u (Ir, Yor, y1) (2r, Dwrs 211)
TT) 8 DSSgRR (x, Yori Feef mo)
else [Method M is Ra-DPO,]
Calculate d2 (I,, Mw 31) = Re (I, y15 Fref Te) sg (3DseqRR (I, ywi Tret ma))
0 + + 77 V (=..
[log o (u (It, Bars B1) ad2 (I, Vw. 31 }1))
end if'
end for
C Supplementary Materials for Section 4
C.1 Experiments compute resources
All reported results of our algorithm and baseline algorithms are trained using A100 GPUs, each with 40GB of memory.
21

---

C.2 Assets
We have compiled the datasets, models, and benchmark codes used this paper and express our gratitude to all relevant sources.
Dataset:
IMDb Dataset
32
Anthropic HH Dataset
33
AlpacaEval
34
Model:
GPT-2 Large 36
Gtg.l/a....uiibiio//e/e///////rr -aa.:--o-----ag------------ tuned
Pythia-1.4B 37
Pyt 37
Oasst-stt-4-pylhia- Satpyythhaa----b-agg 12b-epoch-3.::
attps///uugglggaaoo..ooooossssss
passt-sft----p pythiia 12b--aa---b--ppp-p- epoch- 3.5
Code:
We trained Ra-DPO and the baseline models based on the original KTO implementation https //github. Aaiiaab..cmm/ConeeettaaaAAAAAA and our code can found in the supplemental material.
C.3
Experimental Details
In our experiments, followed the original KTO implementation for the main parameter settings. and both Ra-DPO and the baseline models used the same hyperparameters, as detailed in Table
Table 2: Hyperparameters in loss functions for different algorithms.
Method
DPO
0.1
KTO
0.1
TDPO;
0.1
TDPO:
0.1
(0.7,..5,
Ra-DPO
0.1
Ra-DPO;
0.1
(0.700.5,0
CVaR: [0.99, 0.98, 0.97, 0.95)
ERM: 79,7,5)
Table 3: Hyperparameters in network training.
Parameter
value
max length max prompt length gradient accumulation steps
512
256
learning rate optimizer
x 10
Adam W
C.4 Additional Experimental Results
Here, provide some additional experimental results, which are illustrated in Figures 5 14
22

---

1.0
2.000
Drean (talm))))
Reward Accuracy
2.622
0,530
24
/CoVaR) -- 0.3 p-3
2.523
DPO:ICVaR]
Ra DPO;/CVaR]
a= 0=0.3
2.500
(n-DPOs(CVaR) 0.3
2,433
30
23s
16h: 1Mk
160x
100K 120k
2.625
Reward Ncturacy
9.575
TDPOs == 0.5
1.3
(P.Da///CaaR] a- 0.5 4 0.98
21
DPOs/CVaR)
3,538
Ra-DPo,/CVaR) a-
0.035
30
0.0
23k
6Bc
G3t aik Step
100r
Lank
100c
105e
282
60c
B3c
120k
1000
108
216
60k
1000
2204
2000
Step
Stepp
Drean peittalnn)
15
Reward Accuracy
0.sc-
23
0.32
Ra- (P-DPP/D//R) 0-7. μm 0.99
22
(POX(CVRR)
--
21
Rai (PO:(CV
0 0.97
Re- (PO.)OVa))
0.7
200
83:
140k
200c
100k
Step
Reward Accuracy
TDPG,
02
Ra-DPO:/CVAR) 0- 0.99
Ra DPOs/CVaR) 0.9 0.98
0.52
(CCDPP a- 0.9 u 0.05
0.45-
Stop
Stop
St.p
Figure 5: The experiment the Anthropic HH dataset with Pythia-1.4D serving as the base model. Left and Middle present the progression of sequential KL divergence (the lower the better) for both preferred and dispreferred responses. Right illustrates reward accuracy curves (the higher the better).
TDPO: Ra-ara/ChRR 0PO.ICVaRI
6.6
00
c.0
15
DA
Coufficent
Figure 6: The experiment the Anthropic HH dataset with Pythia-1.4D serving the base model. Left and Middle presents the sequential KL divergence (the lower the better) for preferred and dispreferred responses, while Right presents the reward accuracy curves (the higher the better) under = (0.3, 0.5, 0.7, 0.9).
23

---

10
Drean yuittellee
(x.y0 (a.n andnn
Roward Accuracy
2,13
3,030
24
TDPO; 0.3
Rai (PO((EM) p
Ra (ROM(ERM) == 0.3 p=7
2,075
Ra-DPOX(ERM) pm
100.
1o0k 200c
L00K
Step
0.7
3.0
Doye (eyyunnnui
Reward Accuracy
0.54
0.54
2.3
0.04
Ra (POXEERM 0.5 5= Ra- DPOIEEMI
0..2
21
(E-DP) a- p=?
30
0.0
23t
obc
G3t
alk Step
100c 12kk
140x
163x
202
60c
B3c
1206
500c
300x
206
10k.
odec
100k
120k
2.00c
Step
Step
Drean:0 De. yalttelna)
Diee (e (e..unnddd n.dmm))
Reward Accurscy
2,900
233
TDPO; -07
3.525
DPOXERMI 9
2,000
Ra (POXERMI 0.7 u=
6.1
0,475
Ra- (EOXOMRE
0,450
200.
83:
140k
100k
Step
2.922
Obart Ooarttl-Jmiinn tedd
Devell mo Inn
Reeard Accuracy
3,630-
9,595
TDPO, 0.9
(POXLRMI
3.500
(FMM)) p1-
2,425
Ra. (POX())M) a-
3,45-
100h
Figure 7: The experiment the Anthropic HH dataset with Pythia-1.49 serving the base model. Left and Middle present the progression of sequential KL divergence (the lower the better) for both preferred and dispreferred responses. Right illustrates reward accuracy curves (the higher the better).
TDPP, Ra OPOVERRM)
6.64
03
6A
35
Coafficent.
33
05
06
0.
15
33
0.5 0.5 0.7
0.0
Figure 8: The experiment the Anthropic HH dataset with Pythia-1.44 serving the base model. Left and Middle presents the sequential KL divergence (the lower the better) for preferred and dispreferred responses, while Right presents the reward accuracy curves (the higher the better) under (0.3, 0.5, 0.7, 0.9).

---

2.00
230
1.75
1.30
Rowerd Rowerd Accuracy
6.42
131
TDPO; a=0.3
0.75
Ra-DRov/CVaR) e- p=033pm000
0.50
OPOO///C p
0.23
H ==
p=
6.5-
pm095
20c
40c
60:
Dók 10th 120: 182x 159c Step
20c
406
26k
1028
120c
2032
200c
22k
40h.
620
1006
2106
14GK.
2006
Step
Step
0.925
2,00
Lo.
Rewerd Acturacy
3.575
3.6
TDPO, 0.5
DPO:(CVaR) μ 0.99
3,500
Ra (POx(VVRR) = 0.5 y= 0.98
0,05
y
Ra- (C-DPO/C - 0.5 μ 0.95
2.00-
20
3,455
Stop
Stop
St.p
Dreen (e-Yeiffeellnn
Reward Accuracy
0.60-
3.6
6..5-
TDPO, 0.7
2.4
Ra. DPOxC/RR) - 0.7 pm
(P-DPP/CCV)) μm
0.50
Ro- (P./D//
a 0.7 μM 0.97
(PO.(C
0.95
23k
60c
62k
Btk 100. 1206 140c 2028 Step
20:
40k
60c
B3K
100K
1206
140k
2604
20c
406
60kc
Bik:
100k
220k
140K
2006
Step
Step
1.0
2,825
3.002
Otent Devee.e-.aettlll Ve-etellll
Dost Ik.yom (fteel
Reward Accuracy
2.322
6.6
(X-DPO/CC/R) e 0.9 μm- 0.99
(PO./C/RR) 0.9
0.055
(K-DPO/CVWR)
a- 0.9 um 0.97
Ra- DPO,(P/)) 0.9 0.05
0,422
5.433
100c
03k Step
540c 2004
40k
100k 2008
Step
The experiment the Anthropic HH dataset with Pythia-2.8B serving as the base model. Left and Middle present the progression of sequential KL divergence (the lower the better) for both preferred and dispreferred responses. Right illustrates reward accuracy curves (the higher the better).
Figure 9:
DPO.CVVR1 Nu.orouryyai
RA-0PO/CAR u-
335
03
CA
35
33
02
06
03
15
Figure 10: The experiment the Anthropic HH dataset with Pythia-2.8B serving as the base model. Left and Middle presents the sequential KL divergence (the lower the better) for preferred and dispreferred responses, while Right presents the reward accuracy curves (the higher the better) under a = (0.3, 0.5, 0.7, 0.9).
25

---

2.00
230
1.75
1.50
Heward Accuraty
135
2.532
3,535
0.75
TDPO, 0.3
0.50
OFO-IERM) p
0.23
Ra OPO.EERM) == p=
0.3
025-
2,433
2,43
20c
40
60:
tók 1028 ti0x 14s 189c Step
20c
406
60k
102k
120c
2032 200c
22k
40K
62
2006
200K
240K
2006
Step
Step
2.000
Lo
Reward Kcturacy
2,322
3.6
2.523
TDPO, 0.5
Ra (POX(ERM == 0.5 9
2,000
20
Ra- (POX))))) -- 5
0,450
Stop
Stop
St.p
3.625
Dreet Do-Yeiffeellln
Dvenn De.eyteeeel
Rewardd Accurssy
2.802
0.575
3.6
86-
2.0
TDPO> a= 0.7
Te.
2,322
(I-DPOJEEMM
u
Ra- (PO(EM)) - 0.7 pm? (POXE)))
2.430
23k
60c 62k
BEk 100. 1206 140c 2026 Step
20:
40k
60c
B3K
100k
1206
140k
260c
20c
406
60cc
L00k
220k
140K
2006
Step
Step
1.0
2,825
0.632
Dryan Hoittelin))
Deevet (e. meditel
Roward Accuracy
3.575
6.6
Se.
TDPO; 0.9
(PO.EM)) 0.0
2,00
Rai (POXERMM a 0.9 p
3,035
(PMM)
2.40
100c
03k
140k 2004
40k
1000 2006
Step
Step
Figure 11: The experiment on the Anthropic HH dataset with Pythia-2.8B serving as the base model. Left and Middle present the progression of sequential KL divergence (the lower the better) for both preferred and dispreferred responses. Right illustrates reward accuracy curves (the higher the better).
OPO.IERN) (PO.ERR))
3350
64
05
Di
14
26
The experiment on the Anthropic HH dataset with Pythia-2.8B serving as the base model. Left and Middle presents the sequential KL divergence (the lower the better) for preferred and dispreferred responses, while Right presents the reward accuracy curves (the higher the better) under = {0.3, 0.5, 0.7, 0.9).
Figure 12:
26

---

0.7-
Drean Heitall)))
305-
TDPO:
2.4
Ro:OPO μ 0.90
9.
Itwward
OPO:(CVR))
p
0.97
Va-DPOuLCVat)
20k
100%
bobs 196c
Stop
0.7
0.2-
1.6
0.92
Reward Acturars
0.02-
0.35
TDPO>
0.50
Rs-DPO (EEEM))))MM)
9
2.1
Ra: DFO.IRM
0.1.
Ra OPOVIERM) p=
0.48
Stap
Stop
Stop
Figure 13: The experiment on the Anthropic HH dataset with Pythia-1.4P serving as the base model. Left and Middle present the progression of sequential KL divergence (the lower the better) for both preferred and dispreferred responses. Right illustrates reward accuracy curves (the higher the better). For all algorithms, we report the average performance (solid line) across three random seeds, with the shaded region representing one standard deviation around the mean.
ReDFOs vo TDPO
86
17:27
10.87
DROs TDPO,
44.35
13.54
Ra DPC: vs KTO
43.35
15.78
40.87
PPO
RE-DPO, va DPO
34.04
34.53
31,43
60
80
110
lose
Figure 14: AlpacaEval comparison between DPO, PPO, TDPO, TDPO,, and Ra-DPO, methods. The win, tie, and lose rates evaluated based oasst-pythia-122
Figures 5 illustrate the experiment the Anthropic HH dataset with Pythia- 1ethia-1.4BB serving as the base model. We implemented TDPO,, and different versions of Ra-DPO, with respect to the parameters and μ.
Figures 12 show corresponding results using Pythia-2.8B as the base model. The same set of algorithms evaluated under varying and μ configurations.
Figure 13 13 illustrates the experiment on the Anthropic HH dataset with Pythia-1.4B serving as the base model. Let a 0.5, we implemented TDPO,, and different versions of Ra-DPO, with respect to the risk control parameter μ.. In the figure, for all algorithms, we report the average performance (solid line) across three random seeds, with the shaded region representing one standard deviation around the mean. We aim to highlight the statistically significant improvements achieved by the proposed method, although training large-scale models entails substantial computational costs in terms of time and resources. The figure illustrates that, under both the CVaR-based nested risk measure and the ERM-based risk measure with μ 5, the proposed algorithm achieves reward accuracy comparable to
27

---

that of the baseline method but with greater stability, while maintaining a consistently low sequential KL divergence.
Figure 14 illustrates the comparison between DPO, PPO, TDPO,, TDPO,, and Ra-DPO; methods through AlpacaEval. It presents a straightforward result: Compared to the baseline algorithms, Ra-DPO, achieves high winrate, demonstrating superior performance in assisting LLMs to generate high-quality responses.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both the abstract and introduction include the claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations of the proposed approach from both theoretical and experimental viewpoints, detailed in Subsection 8.1
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

This paper provide the full set of assumptions and complete (and correct) proof. Specifically, we provide complete proofs of the paper's lemmas, and theorems in Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the pseudocode of Algorithm I Appendix B.7 and provide detailed description the experiments in Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
30

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The paper provide open access to the data and code supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
moips.cc/pullic/guiddes /CoeeSumm ssi iessonPoolin mollicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper states the experimental setting/details in Section 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: The paper provide the statistical significance of the experiments in Figure in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provide the information on the computer resources in Appendix
C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://neurpp:: httpss/shttt////rr...../ppppp//a
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Ye]]
Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work performed in Subsection 8.2
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
32

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package and provide detailed information on how to access the datasets, models and codes in Appendix C.2
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide details of new assets in supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
34

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
35


Reviews:
[{"review_id": "Rg1qIrz81E", "reviewer": "Reviewer_mp5k", "summary": "The authors highlight a couple issues with industry standard techniques:\n1. The lack of granular token-level likelihood and optimization (issues faced by DPO and its variants). \n2. The lack of consideration of the overall distribution (i.e. the tails) during the reward maximization approach for DPO variants that do not suffer from the issue highlighted in #1 (such as for TDPO, which only optimizes the expected reward/the mean). Many variants that do address this fail to address #1.\n\nThey are proposing a new method that hits #1 and #2, and have termed this Ra-DPO. They are proposing a new method that considers the shape of the reward distribution when maximizing the expected return. They do this by replacing the expectation by a nested risk functional which pushes the downstream reward return distribution through a coherent risk analysis (to maintain convexity), by focusing on the worst outcomes (to focus on the tails parameterized by mu) and by propagating that token by token (basically if any downstream token contains a high risk tail, the current token reflects it). The authors also proceed to show the efficacy of it through empirical trials. \n\nThe contribution are two new objectives (Ra-DPO1/2) that directly address both of the issues with current widely adopted optimization objectives, and the theory that underpins those assertions + empirical results for those two new objectives compared to the currently adopted objectives.", "questions": "1. How does the efficacy of this metric change with the base model's entropy? How does it compare to when the base model is very peaky on the data this is being applied to versus when it is more OOD to the base model? I think this is my main concern with this approach, since it relies quite heavily on the base model's entropy and the ability to sample the tail. An explicit section on this (with respect to having concrete metrics from the base model itself to understand its capability and behavior on the evaluated task) would be helpful. \n2. How does the gradients and the E2E efficacy look on tasks with rewards that benefit from varying answer lengths? Some tasks require longer answers and others shorter, how does the efficacy of the methodology wrt the baselines shift on these tasks? I do not think the length controlled scores from AlpacaEval satisfactorily covers this, as the target of this question is the potential for noisy gradients as the length increases for tasks that demand it, while the AlpacaEval communicates better accuracy at lower generation lengths. \n3. Can you probe Ra-DPO on benchmarks that explicitly measure the rare but severe failure modes and report more tail-focused metrics from these? Ideally, it would be good to have some way to qualify the shift in model's answers specifically made possible by capturing the entire reward distribution -- including the tail.", "limitations": "Broadly yes, but the rigor of the baselines is a limitation of the paper. The paper is good and the idea is novel, but the paper can be made stronger by having more practical feasibility studies of the approach. In general, being able to provide reasoned guidance about the the dependency on the reference entropy, the gradient noise response with respect to the generation length, and more tail-oriented studies would greatly strengthen the paper. It is possible to do this with limited compute by primarily having good coverage of base models and tasks along these axis in my opinion.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality and Clarity: The quality of the paper is strong in my own opinion. I see a strong narrative whereby the authors start with drafting the limitations of the current methodologies in industry along two relevant axis (i.e. capturing the entire reward distribution and the need for token-wise maximization) and then directly address the limitations through a new formulation that is theoretically underpinned rigorously. The proofs are well written and is relatively easy to follow, and the limitations and workarounds are clearly indicated. I also appreciate the augmentations made to the formulations for mechanistic purposes, making it easier to adopt. \n\nThe practicality study of the methodology needs more work in my opinion to get a strong significance score. I fully acknowledge the three dataset (IMDB, Anthropic-HH, and AlpacaEval) + Two Model family (Pythia-XX,  GPT2) experiments, but they seem a bit limited due to how dated they are, and how saturated the results are.\n\nThe baselines are the most significant weakness of this paper, with IMDB (Figure 2) showing very saturated rewards across hyperparameters (almost impossible to discern impact). Figure 3 and Table 1 are more meaningful, but requires more model baselines. The varying in models is appreciated, but the authors do not sufficiently explore the connection of the base models to the task being studied despite relying so heavily on the reference policy entropy. I can conceive of a situation where the base policy is very peaky (due to heavy SFT, etc.) and the sampling of the tail being more difficult. This is especially true when the data being optimized is very in-distribution to the base reference policy SFT data. This relation does not seem to be comprehensively studied. It would be helpful to see how this approach practically translates to those situations, which are relatively common. Also, I don't see direct metrics that address their core claims, adding in more tail-sensitive practical metrics would be helpful here to directly address the claims that are made initially, instead of solely adding E2E numbers since they do claim that capturing the entire reward distribution is a big part of their contribution. Having concrete metrics that directly show it would be helpful. Overall strong paper, but this would improve it. \n\nThe idea of jointly optimizing for the reward distribution shape and a per-token reward maximization is fantastic. Even beyond safety, the tail is a critical part of the out of distribution quality, and ensuring minimal degradation with respect to the base policy is very meaningful (more so on verifiable and reasoning-benefitting tasks, which isn't covered here). Having a way of optimizing for both jointly is good contribution -- the only major limitation of the paper being the practicality and feasibility study comprehensiveness.", "was_revised": true, "final_justification": "The authors and I discussed their baseline methodology. The authors mentioned wanting to provide additional information in a revised version of the paper pursuant to some of the provided comments, and provided references to existing experiments they have run for the rest of the comments. The primary issue around the rigor of the baselines still exist, but the additional information strengthens the paper further along the lines that the paper was already strong in. However, since my original rating was already high and -- in my opinion -- accurately captured all the strengths of the paper and the additional information the authors provided only reinforced those strengths, I maintained my rating (as the edits has not bumped it up to the next tier).", "rebuttal": "We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.\n# Q1 & Q3. Regarding the model's entropy, practical metrics and model baselines.\n**Response:** To more directly demonstrate the characteristics of our algorithm, we have added experiments with multiple metrics under different model baselines.\n\n**Model baselines:** We conducted experiments with Pythia-1.4B and DeepSeek-R1-Distill-Qwen-1.5B [1] on the Anthropic HH dataset. \n\n**Practical metrics:** We added policy entropy and Sharpe ratio [2-3] to evaluation our method. The Sharpe ratio is a widely adopted metric for assessing risk-adjusted performance. In the context of language model alignment, we adapt it to assess the risk-adjusted reward. A higher Sharpe Ratio indicates that a model attains a given performance gain with lower behavioral deviation—thus reflecting more stable and reliable training dynamics.\n\nHowever, due to the NeurIPS 2025 policy prohibiting the inclusion of external links or PDF in the rebuttal, we summarize the key results in the table below.\n\n**Table 1. Performance and runtime of various algorithms.** (Anthropic HH, Pythia-1.4B, H100, $\\beta=0.1, lr=5.0e^{-6}$）\n\n| Model | Reward Accuracy | $D_{SeqKL}$ | Sharpe Ratio | Policy Entropy | Runtime | \n|-|-|-|-|-|-|\n| DPO | 0.51 | 9.61 | -2.86 | [2.93→2.82] | 179min | \n| TDPO2 | 0.51 | 0.61 | 0.08 | [2.93→2.89] | 185min | \n| Ra-DPO2(CVaR) | 0.55\t| 0.44 | 0.92 | [2.93→2.91] | 207min | \n| Ra-DPO2(CvaR-v2) | 0.54 | 0.45 | 0.83 | [2.93→2.91] | 196min | \n| Ra-DPO2(ERM) | 0.55 | 0.42 | 0.95 | [2.93→2.91] | 192min | \n\n**Experimental results in Table 1:** \n- The model trained with DPO exhibits low reward accuracy, high KL divergence, and a negative Sharpe ratio, indicating poor alignment performance and unstable policy optimization.\n- Models trained using different Ra-DPO variants maintain a stable policy entropy of approximately 2.93 across the entire training process. In contrast, models trained with baseline methods show a consistent downward trend in entropy, persisting until the end of training.\n- **Several variants of Ra-DPO consistently achieve high reward accuracy and Sharpe ratios, while maintaining low KL divergence.** Notably, these improvements are achieved without a significant increase in training runtime.\n\n**Table 2. Performance and runtime of various algorithms.** (Anthropic HH, DeepSeek-R1-Distill-Qwen-1.5B, H100, $\\beta=0.1, lr=5.0e^{-6}$)\n| Model | Reward Accuracy | $D_{SeqKL}$ | Sharpe Ratio | Policy Entropy | Runtime | \n|-|-|-|-|-|-|\n| DPO | 0.49 | 0.45 | -0.75 | [2.89→2.85] | 236min | \n| TDPO2 | 0.52 | 0.37 | -0.08 | [2.89→2.86] | 242min | \n| Ra-DPO2(CVaR) | 0.54\t| 0.34 | 0.93 | [2.89→2.88] | 261min | \n\n**Experimental results in Table 2:** \n- Experiments conducted using the DeepSeek-R1-Distill-Qwen-1.5B model yield patterns similar to those reported in Table 1, validating the consistency of our findings. \n- A closer look at Table 2 reveals a key distinction: while DPO and TDPO2 exhibit marginally negative Sharpe ratios, indicating limited risk-adjusted performance, Ra-DPO continues to achieve a substantially higher Sharpe ratio. **This underscores the stability and generalizability of Ra-DPO under varying model backbones.**\n\n**Experimental analysis:** \n- A comprehensive analysis of Tables 1 and 2 reveals that Ra-DPO2 delivers consistently strong performance across two distinct model backbones: Pythia-1.4B and DeepSeek-R1-Distill-Qwen-1.5B. **Specifically, models trained with Ra-DPO2 attain high reward accuracy, preserve policy entropy (indicating sustained exploration), achieve a high Sharpe ratio (reflecting superior risk-adjusted performance), and maintain low KL divergence (ensuring proximity to the reference policy). These results highlight the robustness and broad applicability of our approach.**\n\n**Note:**\n1. The values of reward accuracy, KL divergence, and Sharpe ratio reported in the Table 1 and 2 represent the average over the final stable phase of training, after convergence. The policy entropy of DPO and TDPO2 does not converge, but instead continues to decrease gradually throughout training.\n2. The experiments in Tables 1 and 2 were conducted on two H100 GPUs, each with 80GB of GPU memory.\n3. In our responses to the other reviewers, we provide additional analyses on complexity analysis, Sharpe ratio, hyperparameter sensitivity, and other key aspects for each baseline.\n4. Ra-DPO2 (CvaR-v2) is another version of Ra-DPO2 (CvaR), which effectively alleviates computational pressure by using top-k instead of quantile when calculating CVaR. Notably, our experiments show that the impact of this substitution on metrics such as reward accuracy and KL divergence is negligible.\n\n# Q2. How does the …… generation lengths.\n**Response:** We appreciate the reviewer’s insightful comment on the complex interaction among answer length, reward shaping, gradient variance, and overall E2E performance. Indeed, disentangling these factors is non-trivial and remains an open challenge in RL-based language generation. Due to time constraints, a comprehensive analysis is currently infeasible. However, as shown in Tables 1 and 2 and our supplementary material, **several key points can be observed:**\n- **There exists a certain trade-off when incorporating risk awareness into TDPO.** As shown in Tables 1 and 2, Ra-TDPO2 attain high reward accuracy, preserve policy entropy, achieve a high Sharpe ratio, and maintain low KL divergence, while requiring slightly more running time.\n- While our method does not explicitly control for output length, we observe in the Supplementary Material that **models trained with Ra-DPO2 tend to generate more concise and focused responses without sacrificing reward,** suggesting a potential regularization effect. \n\nWe will include a discussion on this point in the final version.\n\n## Regarding Supplementary Material:\n- In the directory: \"LLMs_eval_results / Q&A_evaluation_examples\" in Supplementary Material, the sampling results from AlpacaEval show DPO (with higher KL) tends to generate longer responses in many question-answering tasks, some of which contain several repetitive statements and lack logical coherence. In contrast, such issues are rarely observed with TDPO and Ra-DPO.\n\n- In the directory \" Q&A evaluation examples using LLMs\\3-Evaluation results of LLMs\" in the Supplementary Material, we present a comparative evaluation of our algorithm and baseline methods using DeepSeek and GPT-4o, based on a selected set of questions. The evaluation dimensions include:\n    - **Riskiness:** Whether the answer has potential risks or problems.\n    - **Effectiveness:** Whether the answer can effectively solve the problem.\n    - **Relevance:** Whether the answer closely revolves around the core of the question.\n    - **Redundancy:** Whether the answer has unnecessary repetitions or redundant information.\n\nThe evaluation results show that our algorithm (Ra-DPO) performs well in terms of riskiness and relevance, but there are differences in effectiveness and redundancy. \n\n\n#  Reference:\n1. Guo D, Yang D, Zhang H, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.\n2. Sharpe W F. The sharpe ratio. Streetwise–the Best of the Journal of Portfolio Management, 1998, 3(3): 169-85.\n3. Dowd K. Adjusting for risk: An improved Sharpe ratio. International review of economics & finance, 2000, 9(3): 209-222."}, {"review_id": "08WkqWjHjx", "reviewer": "Reviewer_wuGU", "summary": "This paper proposes Risk-aware Direct Preference Optimization (Ra-DPO), a LLM alignment method that includes risk-awareness by using a class of nested risk measures. The main idea of Ra-DPO is to incorporate the nested risk measure such as Conditional Value-at-Risk (CVaR) and Entropic Risk Measure (ERM) into Token-level Direct Preference Optimization (TDPO) to avoid the model drift from the reference model. This paper evaluates Ra-DPO on IMDb, Anthropic HH, and AlpacaEval. The experiment results show that Ra-DPO can improve the alignment performance, while avoiding the model drift.", "questions": "- Q1. How does Ra-DPO compare to the other method such as DPO and TDPO in terms of computational complexity?\n- Q2. Is there any trade-off in incorporating risk-awareness into DPO or TDPO?\n- Q3. In Figure 3, why does Ra-DPO with ERM work worse than Ra-DPO with CVaR?", "limitations": "The authors provide the limitations of their work in Section D.1 (i.e., Limitations) of Appendix. However, I am not sure that the limitation section can be in Appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The strengths of this paper can be summarized as follows:\n- S1. The authors derive the loss function of Ra-DPO by using the thorough theoretical analysis.\n- S2. Even though the evaluation benchmarks (IMDb and Anthropic HH) are relatively simple, the authors provide comprehensive experiment results on those datasets.\n\nThe weaknesses of this paper can be summarized as follows:\n- W1. Even though this paper is theoretically strong, the proposed method can be considered as a direct extension of the previous work TDPO (Token-level Direct Preference Optimization, ICML 2024).\n- W2. The loss function of Ra-DPO includes the term that calculates $D_{SeqRR}(x, y; \\pi_{ref} | \\pi_{\\theta})$ over the generated tokens $z$. This increases the computational complexity during training time.\n- W3. Even though Ra-DPO can effectively control model drift, the model performance does not seem outperforming. For example, in experiments on AlpacaEval (i.e., Table 1), the win-rate of DPO, TDPO_1, and Ra-DPO_1 are 52.1, 51.9, and 53.5, respectively.", "was_revised": true, "final_justification": "I maintain my initial rating, 4: Borderline accept. The main idea of Ra-DPO is to incorporate the nested risk measure into Token-level Direct Preference Optimization (TDPO). In my initial review, I pointed out three weak points and three questions: W1. an extension of TDPO, W2. increased computational complexity in training time, W3. model performance, Q1. comparison of computational complexity, Q2. trade-off in incorporating risk-awareness, Q3. comparison between Ra-DPO with ERM and Ra-DPO with CVaR. The authors provided thoughtful responses to my comments and questions. The response helped me understand this paper better.", "rebuttal": "We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.\n# W1: Even though …… (Token-level Direct Preference Optimization, ICML 2024). \n**Response:** As the reviewer points out, our method is an extension of TDPO. However, it is important to emphasize that we introduce nested risk measures to effectively enhance the risk sensitivity of the policy optimization process. **Our key theoretical contributions include:**\n- Incorporating nested risk measures into token-level policy optimization and providing a closed-form solution. Importantly, our method maintains a natural and simple loss function shown in Figure 1 (the sum of DPO loss and negative sequence risk ratio). \n- Establishing the connection between risk-aware state-action value functions and optimal policies. The key technical contributions lie in: \n- A risk-aware advantage function design under nested risk measures; \n- Proof of Bellman-type model equivalence with the Regret Preference Model under nested risk measures.\n\n**Main experiments results include:**\n- In the paper, we evaluate the effectiveness of the proposed method across various text generation tasks and analyze its sensitivity to the risk control parameter. The results show that our method effectively suppresses the risk of undesirable behavior while improving overall performance.\n- To further demonstrate the algorithm's properties, **we provide additional analyses on policy entropy, Sharpe ratio [1–2], hyperparameter sensitivity, and other key aspects for each baseline.** These can be found in Table 4 and in responses to the other reviewers. The results show that models trained with Ra-DPO2 achieve high reward accuracy, preserve policy entropy (indicating sustained exploration), attain a high Sharpe ratio (reflecting superior risk-adjusted performance), and maintain low KL divergence (ensuring proximity to the reference policy), highlighting the robustness and broad applicability of our approach.\n\n# W2 & Q1: Regarding the computational complexity during training time. \n**Response:** Thank you for the valuable suggestions. Before analyzing computational complexity, it is worth noting that we reformulate the recursive Bellman equation into a classical Bellman equation by constructing an augmented-state MDP despite introducing nested risk measures. As a result, **in terms of computational complexity, the main difference between our algorithm and TDPO lies in the use of risk measures such as CVaR or ERM, as opposed to the expectation.**  Below, we present their time and space complexity.\n\n**Table 1. Complexity of different risk measures.**\n\n| Method | Time Complexity | Space Complexity | \n|-|-|-| \n| Expectation | $O(B \\cdot T)$ | $O(B \\cdot T)$ | \n| ERM | $O(B \\cdot T \\cdot N)$ | $O(B \\cdot T \\cdot N)$ | \n| CVaR | $O(B \\cdot T \\cdot N \\log N)$ | $O(B \\cdot T \\cdot N)$ | \n\nFor a more intuitive presentation, we present performance and runtime of various algorithms on A100 and H100 GPUs under different models and datasets in the following tables. \n\n**Table 2. Runtime of various algorithms.** (IMDB, A100)\n\n| Model | DPO\t| TDPO2 | Ra-DPO2(ERM) | Ra-DPO2(CvaR)| \n|-|-|-|-|-|\n| Gpt2l | 77min\t| 80min | 82min | 83min| \n\n**Table 3. Runtime of various algorithms.** (Anthropic HH, A100)\n\n| Model | DPO | TDPO2 | Ra-DPO2(ERM) | Ra-DPO2(CvaR)\n|-|-|-|-|-|\n| Pythia1.4 | 262min | 271min | 356min | 369min | \n| Pythia2.8 | 13.7h | 13.9h | 14.3h | 15.2h | \n\n**Table 4.  Performance and runtime of various algorithms.** (Anthropic HH, Pythia-1.4B, H100, $\\beta=0.1, lr=5.0e^{-6}$)\n\n| Model | Reward Accuracy | $D_{SeqKL}$ | Sharpe Ratio | Policy Entropy | Runtime | \n|-|-|-|-|-|-|\n| DPO | 0.51 | 9.61 | -2.86 | [2.93→2.82] | 179min | \n| TDPO2 | 0.51 | 0.61 | 0.08 | [2.93→2.89] | 185min | \n| Ra-DPO2(CVaR) | 0.55\t| 0.44 | 0.92 | [2.93→2.91] | 207min | \n| Ra-DPO2(CvaR-v2) | 0.54 | 0.45 | 0.83 | [2.93→2.91] | 196min | \n| Ra-DPO2(ERM) | 0.55 | 0.42 | 0.95 | [2.93→2.91] | 192min | \n\n**Experimental analysis:**\n- Several variants of Ra-DPO consistently achieve high reward accuracy and Sharpe ratios, while maintaining low KL divergence. Notably, these improvements are achieved without a significant increase in training runtime.\n\n**Note:** \n1. In Table 1, B denotes the batch size, T denotes the sequence length, and N denotes the number of samples per distribution.\n2. The Sharpe ratio is a widely adopted metric for assessing risk-adjusted performance. In the context of language model alignment, we adopt it to assess the risk-adjusted reward. A higher Sharpe ratio indicates that a model attains a given performance gain with lower behavioral deviation—thus reflecting more stable and reliable training dynamics.\n3. The runtime reported in Tables 2 and 3 was measured on two A100 GPUs, each with 40 GB of memory, while the runtime in Table 4 was measured on two H100 GPUs, each with 80 GB of memory.\n4. Ra-DPO2 (CvaR-v2) is another version of Ra-DPO2 (CvaR), which effectively alleviates computational pressure by using top-k instead of quantile when calculating CVaR. Notably, our experiments show that the impact of this substitution on metrics such as reward accuracy and KL divergence is negligible.\n\n# W3: Even though Ra-DPO can…… and 53.5, respectively.\n**Response:** We acknowledge that the win rate of our method on AlpacaEval is not significantly higher than that of the baselines. However, it is important to note that the core idea of our approach lies in incorporating risk-awareness by employing a class of nested risk measures, which achieve a better balance between alignment performance and model drift under risky conditions. The token-level sequential risk ratio constraint makes the model more conservative in exploration when facing high-risk situations, leading to generated responses that are shorter and more moderate in length. As a result, our method achieves better performance in terms of length-controlled win rate as show in Table 1.\n\nMoreover, **there are several experimental results are worth highlighting:**\n- Figure 13 in Appendix C.4 presents results on the Anthropic HH dataset using Pythia-1.4B as the base model. For all algorithms, we report the average performance across three random seeds, with the shaded area indicating one standard deviation around the mean. The figure shows that, under both the CVaR-based and the ERM-based risk measure with µ = 5, the proposed algorithm achieves reward accuracy comparable to that of the baseline methods, but with greater stability, while maintaining consistently low sequential KL divergence.\n- In our responses to the other reviewers, we provided results on policy entropy, hyperparameter search, and other aspects for each algorithm. The results show that Ra-DPO (ERM) exhibits the most desirable risk-aware behavior among the evaluated methods, as evidenced by its high and stable Sharpe Ratio, as well as well-preserved policy entropy.\n- In the Supplementary Material under the directory \" Q&A evaluation examples using LLMs\\3-Evaluation results of LLMs\", we present a comparative evaluation of our algorithm and baseline methods using DeepSeek and GPT-4o, based on a selected set of questions. The evaluation dimensions include:\n    - **Riskiness:** Whether the answer has potential risks or problems.\n    - **Effectiveness:** Whether the answer can effectively solve the problem.\n    - **Relevance:** Whether the answer closely revolves around the core of the question.\n    - **Redundancy:** Whether the answer has unnecessary repetitions or redundant information.\n\nThe evaluation results show that our algorithm (Ra-DPO) performs well in terms of riskiness and relevance, but there are differences in effectiveness and redundancy. \n\n**The original conclusion is as follows:**\n- Some algorithms (such as Ra-DPO1 (CVaR, $\\alpha=0.5, \\mu=0.97$)  and Ra-DPO2 (CVaR, $\\alpha=0.5, \\mu=0.97$)  can provide high-quality practical information, while the effectiveness of other algorithms (such as Ra-DPO1 (CVaR, $\\alpha=0.5, \\mu=0.99$) ) and Ra-DPO2 (CVaR, $\\alpha=0.5, \\mu=0.99$) ) is relatively low and their redundancy is relatively high.\n- Future research should focus on optimizing algorithms to improve their performance in terms of effectiveness and redundancy, thereby enhancing their overall performance.\n\n# Q2. Is there any trade-off in incorporating risk-awareness into DPO or TDPO?\n**Response:**  There exists a certain trade-off when incorporating risk awareness into TDPO. As shown in the responses to W2 and Q1, Ra-TDPO2 achieves higher performance and maintaining lower KL divergence, while requiring slightly more running time.\n\n# Q3. In Figure 3, why does Ra-DPO with ERM work worse than Ra-DPO with CVaR?\n**Response:** This may be related to the inherent characteristics of the CVaR and ERM risk measures themselves. CVaR directly optimizes performance on the worst-case samples, whereas ERM applies exponential weighting uniformly across all samples, potentially diluting attention to extreme poor-performing cases. Additionally, ERM is more sensitive to hyperparameters, as its performance varies significantly with the choice of parameter. As shown in Figure 3, when the risk coefficient of Ra-DPO2 with ERM is set to $\\mu=5$, it achieves performance comparable to that of Ra-DPO2 with CVaR.\n\n# Limitations: The authors provide …… can be in Appendix.\n**Response:** In the new version, we will move the discussion on limitations from the appendix to the main text.\n\n# Reference:\n1. Sharpe W F. The sharpe ratio. Streetwise–the Best of the Journal of Portfolio Management, 1998, 3(3): 169-85.\n2. Dowd K. Adjusting for risk: An improved Sharpe ratio. International review of economics & finance, 2000, 9(3): 209-222."}, {"review_id": "7hVXY6D2Qx", "reviewer": "Reviewer_Wtor", "summary": "The paper presents Risk-Aware Direct Preference Optimisation (Ra-DPO), a new method for aligning LLMs with human preferences while taking risk into account. The method builds on token-level DPO (TDPO) and incorporates a nested risk measure from risk-sensitive reinforcement learning.\nThe authors want to include risk-awareness into LLM-training and take it into consideration, rather than simply maximising expected reward.\nIn the paper, risk is defined as the deviation (KL-divergence) from the original reference policy. The proposed loss function aims to maximise the likelihood of preferred responses while simultaneously penalising this token-level model drift.\nThe authors provide theoretical justification for their approach, deriving the loss function from a risk-aware advantage maximisation problem.\nThey present results of applying their method on IMDb dataset and Anthropic HH dataset, and evaluate the resulting models using AlpacaEval. They argue Ra-DPO achieves a better balance between alignment performance (reward accuracy, win-rate) and KL-divergence compared to the baselines.", "questions": "- How were the hyperparameters for the LLM experiments chosen?\n- Could you run the LLM experiments on across a few more hyperparameter values? (beta, learning rate)\n- Could you include performance of DPO (maybe with several different betas) in Figure 2?\n- i think the paper could be improved by maybe using an external \"risk evaluation\" model to score risk rather than using drift from the reference model", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "STRENGTHS:\n- Paper is clear, easy to read, well structured\n- The paper correctly identifies a limitation in current methods (DPO) as they are risk neutral (simply optimise for mean reward) and that TDPO, while moving to a more granular token-level approach, remains risk neutral. Proposing a risk-aware alternative method makes sense\n- The structure of the experiments is sound, training on IMDb, Anthropic and evaluating on AlpacaEval and Ra-DPO shows marginal improvements on the baseline (in AlpacaEval). In Figure 3 they also show improved reward accuracy and KL-divergence from the reference model compared to TDPO. I would have been interested to see the performance of DPO here too\n\nWEAKNESSES:\n- I fundamentally disagree with the authors' framing of \"risk\" as divergence from the reference policy. The whole point of preference optimisation is to make the model less harmful, more helpful and honest. Quite a few papers have shown the KL divergence is useful to keep the model from moving completely out of distribution rather than to maintain any good qualities [1]. A reference model is not inherently \"safe\" or \"correct\". Constraining the policy to stay close to it can limit the model's ability to learn and adapt to the preferences it is being trained on. So much that recent, high-performing offline preference optimisation loss functions (such as SimPO [2]) completely scrap the KL-divergence term and instead just add an SFT term to the loss. This also has memory advantages (as you don't have to keep a copy of the model weights in memory)\n- The paper's motivation that \"a significant deviation from the reference model typically implies the degradation of superior decision-making and reasoning capabilities\" is a strong claim presented without evidence.\n- This narrow definition of risk is acknowledged, but only in the appendix\n- Unconvincing Empirical Results: the improvements are marginal and no extensive hyperparameter search is performed. I understand the computational constraints, but given the marginal improvements I don't think the current evaluation is enough\n- beta of 0.1 is quite low for DPO I think? I've seen 0.5 being used usually\n\n[1] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study - Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, Yi Wu \n[2] SimPO: Simple Preference Optimization with a Reference-Free Reward - Yu Meng, Mengzhou Xia, Danqi Chen", "was_revised": true, "final_justification": "The reviewers did address some of my concerns regarding the use of the word \"risk\" and put it into the context of the existing literature (SimPO vs DPO). However, I'm still not entirely convinced by the relatively minor performance gains and I find the title misleading", "rebuttal": "We sincerely appreciate the valuable comments from the reviewer. We hope our responses below provide further clarity.\n#  W1: I fundamentally ……  in memory).\n**Response:** Thank you for the valuable suggestions. Below, we re-clarify the rationale behind defining \"risk\" as divergence from the reference policy and provide an explanation of the SimPO [1] compared to the DPO. \n## Regarding the define of \"risk\": \nIndeed, as the reviewer pointed out, the core point of preference optimization is to make the model less harmful, more helpful, and truthful. However, drawing upon existing literature and our own insights, we would like to emphasize:\n\n- **A trade-off between alignment objectives and model fidelity is still necessary,** although the original(reference) model may not be \"safe\" or \"correct\". For example, in LLMs safety alignment tasks, a simple objective is to enable the model to reject unsafe responses while preserving original reasoning capabilities [2-3]. A response that is safe but logically incoherent or semantically uninformative is of little practical value. Therefore, many studies [4-5] typically formulate such tasks as constrained reward maximization problems. \n- **KL divergence has typically been used to penalize excessive deviations from a reference (critic) model [6-7].** In fact, numerous studies have reported that KL constraint offers many beneficial effects, such as balancing exploration and exploitation, ensuring stability and robustness, preventing catastrophic forgetting, and preserving the model’s fundamental capabilities [8-9]. \n\n## Regarding SimPO: \nSimPO [1] is a reference-free preference optimization algorithm that not only achieves superior performance, but also significantly reduces memory consumption. However, there are also studies indicating that SimPO has the following drawbacks:\n- As pointed out in [10], **the lack of a reference model in SimPO reduces training robustness and necessitates stricter conditions to prevent catastrophic forgetting.** The experimental results in the paper (Table 1) show that SimPO does not consistently outperform DPO. Specifically, SimPO demonstrates a significant advantage over DPO when training the Qwen2.5-7B-Base model, while DPO slightly outperforms SimPO when training the Qwen2.5-7B-Instruct model.\n- As noted in [11], different from DPO, **SimPO reintroduces complexity through dual parameters $(\\beta, \\gamma)$,** which introduce additional complexity on hyperparameter tuning.\n- As pointed out in Subsection 3.8.2 of [12], **the reward function in SimPO emphasizes length-normalized next token prediction, which raises concerns about a potential deviation from the original alignment objective.**\n\n**From our perspective,** DPO and SimPO serve as representative examples of reference-based and reference-free preference optimization methods, respectively. A comprehensive comparison in terms of performance, stability and robustness, hyperparameter tuning complexity, and computational efficiency reveals that each approach has its own trade-offs. Consequently, they are suited to different applications and both continue to represent valuable research directions.\n\n# W2: Regarding the evidence of our paper's motivation.\n**Response:** We have already provided some supporting evidence in our response to W1. Below, we present additional direct evidence:\n- As pointed out in [7], **in stylistic text continuation tasks, optimizing reward without constraints leads to incoherent continuations,** whereas introducing a KL constraint encourages the policy to remain close to a language model trained on BookCorpus.\n- As noted in [13], TDPO outperforms DPO while maintaining a lower KL divergence. Furthermore, we implemented our Ra-DPO and reproduced both TDPO and DPO baselines. As shown in the sampling results from AlpacaEval (see Supplementary Material, directory: \"LLMs_eval_results / Q&A_evaluation_examples\"), **DPO (with higher KL) tends to generate longer responses in many question-answering tasks, some of which contain several repetitive statements and lack logical coherence.** In contrast, such issues are rarely observed with TDPO and Ra-DPO.\n- **Several of our unsuccessful experiments suggest that model instability or collapse is frequently associated with increased KL divergence.** For instance, when setting $\\alpha=0.1$, both TDPO2 and Ra-DPO show significantly elevated KL divergence, leading to poor performance on the AlpacaEval.\n\n# W3: Regarding the definition of risk.\n**Response:** In the new version, we will appropriately incorporate the explanation of risks into the main text.\n\n# W4-5 & Q1-3: Regarding the hyperparameters chosen and search.\n**Response: Hyperparameters chosen:** As mentioned in Section 4, we set the hyperparameters following the original KTO [14] implementation. Specifically, in the official KTO codebase, the default value of $\\beta=0.1$ can be found in the file “HALOs/config/loss/dpo.yaml”, and the default value of $lr=5.0e^{-6}$ is specified in the file “HALOs/config/config.yaml”. This configuration is also widely adopted in other literature, including in implementations of DPO (Hugging Face version) [5] and TDPO [13].\n\n**Hyperparameter search:** We have conducted extensive hyperparameter searches for various algorithms, including DPO, TDPO2, and Ra-DPO2, across different settings of $\\beta$ and learning rate (lr). \nHowever, due to the NeurIPS 2025 policy prohibiting the inclusion of  external links or PDF in the rebuttal, we summarize the key results in the tables below.\n\n**Table 1. Performance of various algorithms under different $\\beta$ settings.** (Anthropic HH, Pythia-1.4B, H100, $lr=5.0e^{-6}$)\n| Model | | $\\beta=0.1$ | | | $\\beta=0.3$ | | | $\\beta=0.5$ | |\n|-|-|-|-|-|-|-|-|-|-|\n| | Reward Accuracy | $D_{SeqKL}$ | Sharpe Ratio | Reward Accuracy | $D_{SeqKL}$ | Sharpe Ratio | Reward Accuracy | $D_{SeqKL}$ | Sharpe Ratio |\n| DPO| 0.51 | 9.61 | -2.86 | 0.52 | 1.05 | -0.95 | 0.51| 0.73 | -0.78 |\n| TDPO2 | 0.51 | 0.61 | 0.08 | 0.52 | 0.49 | 0.12 | 0.53 | 0.45 | 0.20 |\n| Ra-DPO2(CVaR) | 0.55 | 0.44 | 0.92 | 0.52 | 0.41 | 2.78 | 0.53 | 0.39 | 0.69 |\n| Ra-DPO2(ERM)  | 0.55 | 0.42 | 0.95 | 0.54 | 0.37 | 0.88 | 0.54 | 0.36 | 0.91 |\n\n**Table 2. Performance of various algorithms under different learning rates.** (Anthropic HH, Pythia-1.4B, A100, $\\beta= 0.1$)\n| Model | | $lr=1.0e^{-6}$ | | | $lr=5.0e^{-6}$ | | | $lr=1.0e^{-7}$ | | \n|-|-|-|-|-|-|-|-|-|-|\n| | Reward Accuracy | $D_{SeqKL}$ | Sharpe Ratio | Reward Accuracy | $D_{SeqKL}$ | Sharpe Ratio | Reward Accuracy | $D_{SeqKL}$ | Sharpe Ratio |\n| DPO | 0.51 | 0.32  | -0.23 | 0.51 | 15.5 | -3.10 | 0.49 | 0.20 | 0.01 |\n| TDPO2 | 0.51 | 0.26 | 0.34 | 0.53 | 0.61 | 0.05 | 0.49 | 0.20 | 0.39 |\n| Ra-DPO2(CVaR) | 0.53 | 0.23 | 1.03 | 0.57 | 0.50 | 0.84 | 0.50 | 0.20 | 0.91|\n| Ra-DPO2(ERM) | 0.54 | 0.23 | 0.99 | 0.58 | 0.43 | 0.87 | 0.50  | 0.20 | 1.13 |\n\n**Note:**\n1. Sharpe ratio [15, 16], a new evaluation metric, is introduced. A higher Sharpe ratio indicates that a model attains a given performance gain with lower behavioral deviation.\n2. The data in the tables represent the average over the final stable phase of training. It is worth noting that DPO's KL divergence and Sharpe ratio are often unstable.\n3. The experiments in Table 1 were conducted on two H100 GPUs, each with 80GB of GPU memory, while  the experiments in Table 2 were conducted on two A100 GPUs, each with 40GB of GPU memory.\n\n# Q4: Regarding the \"risk evaluation\" of model.\n**Response:** Thank you for the valuable suggestions. Measuring the divergence between reference model and policy model is a common practice [3, 5, 9, 13]. Introducing an external \"risk evaluation\" model could be a feasible direction for exploration; however, several foreseeable challenges may arise. For instance, during the initial training phase, the distribution of the trained model may differ significantly from that of reference model, which could destabilize or even collapse the training process. Selecting an appropriate reference model would thus be a critical challenge. In our future work, we will conduct further experiments in this direction.\n\nIt is note that in our response to Reviewer mp5k, we provide detailed results on policy entropy, Sharpe ratio, and KL divergence across different models (including Pythia-1.4B and DeepSeek-R1-Distill-Qwen-1.5B), which may offer additional insights into this issue.\n\n# Reference:\n1. Simpo: Simple preference optimization with a reference-free reward. NeurIPS, 2024.\n2. Safe RLHF: Safe Reinforcement Learning from Human Feedback. ICLR, 2024.\n3. Stepwise alignment for constrained language model policy optimization. NeurIPS, 2024.\n4. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.\n5. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.\n6. Trust region policy optimization. ICML, 2015.\n7. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.\n8. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017, 114(13): 3521-3526.\n9. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. ICLR, 2024.\n10. Pre-dpo: Improving data utilization in direct preference optimization using a guiding reference model. arXiv preprint arXiv:2504.15843, 2025.\n11. Repo: Relu-based preference optimization. arXiv preprint arXiv:2503.07426, 2025.\n12. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216, 2024.\n13. Token-level direct preference optimization. ICML, 2024.\n14. Model alignment as prospect theoretic optimization. ICML, 2024.\n15. The sharpe ratio. Streetwise–the Best of the Journal of Portfolio Management, 1998, 3(3): 169-85.\n16. Adjusting for risk: An improved Sharpe ratio. International review of economics & finance, 2000, 9(3): 209-222."}]

Author rebuttal:

Meta-review: The paper introduces Risk-aware direct preference optimization (RA-DPO), a novel LLM alignment method that incorporates risk-awareness with a class of nested risk measures. The authors addressed most of the concerns during the rebuttal, and all reviewers are positive about the paper.
Decision: Accept (poster)