Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
Authors: Nguyen Minh Phuc
Ngoc-Hieu Nguyen
Duy Minh Ho Nguyen
Anji Liu
An Mai
Binh T. Nguyen
Daniel Sonntag
Khoa D Doan
Venue: NeurIPS 2025 poster ()
Keywords: direct preference optimization
human preference alignment
Regularization
Primary area: deep_learning
TL;DR: an importance-sampling-based method to mitigate over-optimization in Direct Alignment Algorithms for language model alignment

Abstract: Recently, Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs) with human values. 
Surprisingly, while DAAs do not use a separate proxy reward model as in RLHF, their performance can still deteriorate over the course of training -- an over-optimization phenomenon found in RLHF where the learning policy exploits the overfitting to inaccuracies of the reward model to achieve high rewards.
One attributed source of over-optimization in DAAs is the under-constrained nature of their offline optimization, which can gradually shift probability mass toward non-preferred responses not presented in the preference dataset. This paper proposes a novel importance-sampling approach to mitigate the distribution shift problem of offline DAAs.
This approach, called (IS-DAAs), multiplies the DAA objective with an importance ratio that accounts for the reference policy distribution. IS-DAAs additionally avoid the high variance issue associated with importance sampling by clipping the importance ratio to a maximum value. Our extensive experiments demonstrate that IS-DAAs can effectively mitigate over-optimization, especially under low regularization strength, and achieve better performance than other methods designed to address this problem.

Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
Phuc Minh Nguyen', Ngoc-Hieu Nguyen', Duy H. M. Nguyen 3,7.0 Anji Liu An Mai",
Binh T. Nguyen", Daniel Somntag Khoa D. Doan 1.2 Cooleeee of Engineering and Computer Science, VinUniversity
2 VVinnni-llinnos Smart Health Center, VinUniversity
University of Stuttgart, *National University of Singapore International University VNUHCM, Uniieeriity of Science VNUHCM
7 "German Research Center for Artificial Intelligence (DFKI), *Oldenburg University Max Planck Research School for Intelligent Systems (IMPRS-IS)
Abstract
Direct Aligmment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs) with human values. However, these methods are more susceptible to over-optimization, in which the model drifts away from the reference policy, leading to degraded performance as training progresses. This paper proposes a novel pooorranne-sampllinn approach mitigate the over-optimization problem of offline DA This approach, called (IS- DAAs), multiplies the DAA objective with an importance ratio that accounts for the reference policy distribution. AS-DAAs additionally avoid the high variance issue as- sociated with importance sampling by clipping the importance ratio to maximum value. Our extensive experiments demonstrate that IS-DAAs effectively miti- gate over-optimization, especially under low regularization strength, and achieve better performance than other methods designed to address this problem. Our code is available at http::///gghhau...caaaaaerrreee
www.commmmmcic. research AIS- Sampling4DAss
Introduction
Preference learning has emerged as important part of the fine-tuning process align large language models (LLMs)) with human preferences. There are two predominant flavors of preference learning for LLMs. The first approach is online reinforcement learning from human feedback (RLHF)
[Ouyang et af.] 2022 Christiano et al.] 2017). RLHF typically involves a multi-stage procedure: fine-tuning a reward model to capture human preference and fine-tuning the LM policy to maximize the expected reward using online RL. algorithms such as Proximal Policy Optimization (Schulman al. 2017]. While empirically exhibiting good performance, this multi-stage procedure complex and computationally intensive, requiring repeated queries of the reward model and sampling from the current policy. A set of alternative methods called direct alignment algorithms (DAAs) (Rafailov et al., 2024 Tang et al. 2024cc avoids fitting separate reward models and instead simply trains the policy directly on the offline preference dataset. The most well-known examples are Direct Preference Optimization (DPO) [Rafailov et al. [2023), and Identity Preference Optimization (IPO) [Tang et al.) 2024b]. Since DAAS typically do not sample new responses from the LLM's policy during training. they are characterized as offline preference learning.
These methods often preferred when aligning LLMs thanks to their simple training pipeline. However, recent studies find that DAAS are more susceptible to over-optimization-2 phenomenon in which performance degrades training progresses- and underperform online methods (Rafailov
et al. 2024 Park et al. 2024a Liu et al. 2024b Tang et al. 2024a More specifically, Rafailov
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

et al. [2024] and Tang et al. [2024a] use the KL divergence between the preference-trained policy and reference SFT policy as a measure of budget and show that, as this budget increases. DAAs tend to experience greater decline in performance. Rafailov et al. (202) explains over-optimization in DAAs by the lack of training data coverage (i.e., using only offine samples); thus, optimizing their loss function can push up the probability of responses that are out of the offline data distribution.
This is not the case in online alignment algorithms, since they can sample online data and explicitly enforce reverse KL regularization to mitigate this behavior [Song et al. 2024].
In this work, we identify an important source of the performance gap between online alignment algorithms and DAAs: during training, the trained policy gradually diverges from its initial distribution used collect preference data. Meanwhile, the regularization effect in DAAs provides only local approximation of KL regularization around this offline distribution, which is insufficient to prevent this drift. One approach to mitigate this problem encourage the model to stay close to the reference policy (Song et al. 2024 Fisch et al.) 2024, Ding al. 2024) and train the policy in an online manner [Guo et al. 2024). This regularization explicitly prevents the LM policy from deviating from the initial distribution. However, these methods are
enforce an explicit KI. divergence penalty to
costly since they require repeated sampling from the current policy to estimate the KL divergence and are sensitive to hyperparameters. Instead, this paper proposes novel importance-sampling approach, called Importance Sampling DAAS IS-DAAss. Furthermore, the implementation of IS incurs minimal computational overhead, making it highly scalable.
Our main contributions are: First, we investigate the over-optimization problem in offline alignment methods and propose a novel importance sampling method to address this problem for DA Then, we fix the high-variance problem associated with importance sampling by clipping the importance ratios. Finally, our extensive experimental results indicate that IS-DAAs outperform DAAS and other regularized methods. More importantly, IS-DAAs are significantly better in mitigating over- optimization and early convergence issues, compared to previous regularization approaches.
2
Preliminaries
We provide the formulation and background of RLHF and DAAs in sections 2.1 and 2.2 respectively. The over-optimization phenomenon and regularization in DAAS are presented in section 2.3
2.1
Reinforcement Learning from Human Feedback
To align an LM with human preferences, the RLHF pipeline consists of three stages:
Supervised Fine- TuninTunng (SFT): Given pre-trained model and a dataset of prompts x and their responses y. the LM is trained for instruction following using maximum likelihood estimation (MLE) over next-tokens, resulting in the reference model (x)).)
Reward Modeling: In the second phase, the reference model क is prompted with to produce a pair of responses (y1, Y2) ~ Tret (-1x). These responses are then labeled by human experts; the resulting pair is denoted as y > y'Ix |x, expressing human preference for y over y'. This collected preference data is denoted as 'D = (x(0), y0((), y NN J Typically, preference labels are assumed to follow the Bradley-Terry model:
exp(r(x,)1))
ply: > Y2l0) = exp(r(x,y))) + exp(r(x,y2) = (r(x,),) x2),
where r(x,y) y) is a reward model, mapping a pair of an input prompt and its response to a scalar reward. We can then use D to train parametrized reward model (x(x,)) to maximize the differences between the rewards of y and y' using MLE, as follows:
CR(TS) = - Exxxy" -'')='''''>> [log a (ro(x, y") T ro y'))
RL Fine-tuning: The learned reward function is used to provide feedback in the RL phase, using an on-policy algorithm, such as PPO, with the following objective:
max E Px=DDymmoo (-1x)
(x, - SKL((T (rett)
where uo is the learning policy, and 3 is the hyper- syper-parameter controlling the KL regularization w.r.t the reference policy गer. This KI. constraint prevents the model from deviating too far from the region on which the reward model is well-trained, and mode-collapsing to only high-reward responses.

---

2.2 Direct Alignment Algorithms (DAAs)
Despite its superior performance in aligning the LMs with human preferences, the RLHF pipeline is complex and computationally expensive, DAAs address these problems by directly optimizing the policy πe over the preference data, avoiding the reward function estimation and RL training phases. Among these algorithms, DPO is the most popular approach; DPO derives the following closed-form solution for nT of Eqn. 3):
(
exp (y|x) (xx Trar(y|x)) exp
(4)
where ((x as the normalization function. According the above equation, we parameterize the reward function by the log- likelihoodd ratio between Tg and ref
To(y|>) x)
ro (x, y) = B log
+ 3 log (xx)
(5)
[y(x)
This enables us to optimize the LM policy aa directly on the preference data by plug the above equation into Eq. ①I and minimizing the corresponding negative-loglikelhooad
To(y"
To (x(
LDAA ket) (xxy" -') s/)= f log Ty=
Blogg
(6)
T [x)
where is convex loss function. By choosing the loss function f.
recover the standard
DPO objective [Rafailov et al., 2023] (f(z) log (x)), or the IPO objective (Azar et al. 2024]
(f(x)= (ar - 1).). Other objectives can be found in [Tagg et al. 2024b In this paper, we will focus on these two well-studied objectives.
2.3 Ontr-Optimization and the Performance Gap Between Online and Offline Alignments
In this study, we refer to online alignment methods as those that require sampling from the currently trained policy during the training process. This includes the two-stage RLHF and online variants of DPO or IPO [Guo et al.) 2024 Schulman et al. 2017]. In contrast, offline methods do not require on-policy sampling and are computationally more efficient.
Experiments from Tang et al. [2024a] demonstrate that both online and offline alignment algorithms suffer from optimizzii,,,nn wherein an alignment algorithm consumes a large optimization budger (such as how far the optimized policy drifts away from the reference policy Tet. measured by KL divergence) without improving (and even reducing) performance. However, offline methods exhibit a larger degree of degradation compared to online methods. Importantly, they show that offline data coverage and data quality cannot convincingly explain the performance difference.
In this study, we hypothesize that this performance gap is caused by inappropriate regularization when using only off-policy data in offline DAAS, making these algorithms more susceptible to over-optimization. In the standard RLHF (Eqn. (3). the trade-off between the surrogate reward scores and the negative KL divergence is explicitly formulated. In each optimization step, if the first term outweighs the second term, the learning algorithm optimizes the LM toward the direction that increases the surrogate reward scores the cost of increasing the KL divergence budget. Consequently, if the surrogate reward function is not good approximation of the true reward function, this behavior will result in an ineffective use of the KL-divergence budget, leading to over-optimization.
In DAAS, this tradeoff is implicit. To see this, we denote the log ratio difference in DAAs' loss
function as pa == log xo(y")) log zoly') and consider the Taylor expansion around pa which ")
" is the case when the finetuning starts with Fy = Feel
3')
B(x))"<>'>>>> (Bpo)) Ri f(0)- - ('())3 Exxx woDD w"> [po] +
Ex Moxyyyy'e yDDD [rē]. (7)
2
direct aligament loss
preference optimization
weighted squared
Tang et al. 2024c] show that " and y' are generated by wa and a labeling function p, then the expectation of the gradient of the weighted squared loss term recovers the update of the reverse KL. regularization, i.c.,
1 V Vo 0b = C. EE ExxD [VoKLL (ne( (-xx, Teet (-(x))
(8)
3

---

On-policy Optimization
Off-adiicy Optimization
IS Weighted Off-policy Optimization
Mede Mean
Figure The results minimizing the weighted square loss term with on-policy data (left), off- policy data (middle), and off-policy data with importance sampling weights (right). The reference distribution is modeled by a mixture two-dimensional Gaussians, and TT an unimodal Gaussian.
where constant. This equality suggests that DAAs with online preference data can enforce regularization by optimizing the weighted squared loss. However, in the offline setting, i.e., y" and y" are generated by nitt and as Te moves away from Titt Eqn. (8) does not hold. Instead, we have
E (x (D)
V 8
Pg
(y(x) V log No (y|xx) log Teer (y//)
-2 (xy))
2D [TTer( (-(() ((x.y))((,,ml
V log To (y(x))
The detailed derivation be found in Appendix B.2 In Equation (9) since the KL divergence is always positive, going in the opposite direction of the second term will push down the probability of both winning and losing responses from the training data. Moreover, the first term will only regularize samples from Tee-- Consequently, optimizing the weighted squared loss using offline data increases the weights of responses that are out of the reference policy distribution, and this effect is stronger when my deviates from Weet The visualization of this effect is plotted in Figure Since preference labels for OOD responses not available, increasing their probability is inappropriate.
Mitigating Over-optimization in DAAS with Importance Sampling
The regularization effect DAAs effective only when the policy Ty remains close to the reference policy Tief. Once To moves away from Ref the influence of the regularization diminishes (Section 2.3) and this can detrimental to performance. To mitigate this problem, a simple approach is to apply online sampling training to collect responses from the current policy Te [Guo et al. [0224.
Conline-DPO (Te, Teet)
To T Tief (y
Wa (y' (x) Tee ( (
- -Ex=D, y m() log
log
B log
(10)
However, online training is considerably more complex than offline methods, as it involves training an explicit reward model and sampling from the LM policy that is being trained, incurring significant computational costs. [Rafailov et al. 2023
3.1 Importance Sampling DAAS (IS-DAAs)
Motivated by the insight Section 2.3 which reveals inappropriate implicit KL regularization in offline DAAS, we design method to more effectively estimate this KI. divergence without requiring repeated online sampling from the my Specifically, we leverage importance sampling to estimate the expectation under the LM policy distribution given samples from the offline data. This leads to the following objective:
LIS.DPO (To, Teet)
To(y" B log Taet (y (x)
Te To('' (x) Trar xx) }
= Ex y (x)( w(xx " "r). loga (Blog
(11)

---

where the importance weights w(x, y", .y) Toly" x) To
Tarly" (x) "x)("" Here. the importance weight is
Furly)x; = IT= Nalze x.y=s.) The b update is multiplied by this importance weight to adjust the action probabilities so that the expectation according to the LM policy Ty can be calculated with samples from Tee.-
the ratio of sequence-level probability between Th and Tef. i.c., -
We next show that by multiplying with an importance ratio, the objective in Eqn. (1 is an unbiased estimation of the objective in Eqn. (Tō) without requiring recursively sample from the learned policy. Moreover, this also leads to the gradient of weighted squared loss, and the gradient of KL. divergence coincides under certain conditions. More formally, have:
Theorem Assuming Supp (Tny) Supp (mo), then the objective in Eqn. (TT) is an unbiased estimation of Eqn. (16) and the gradient concerning To of the weighted squared loss equals to the gradient of KL. divergence,
1 Ex--D.y"yl)-nddtt yxxx ,y'''))) Vo) = Ex [ (TT(-)x)), (-)x]]
(12)
Theorem | implies that by incorporating the importance ratio, we can ensure that minimizing the j-squared regularization also minimizes the KL divergence. The proof is in Appendix B.]
Mitigating Importance Sampling's high variance using clipping. Directly computing the im- portance weights during training can lead to extremely high variance Elvira and Martino (2021), ] potentially resulting in gradient explosions due to extreme values. To mitigate this problem, we propose to use Truncated importance weighting estimator [Elvira and Martino [021]] as follows:
"T(""" x) (x
w(x, " y²)
max
Feerly (x) "
(13)
where serves as a regularization trade-off between the bias and variance of the importance ratio.
3.2 An analysis of regularization effect in DAAs with Importance Sampling
This section provides a detailed example in which the weighted squared regularization term only serves as a local approximation of the KI. divergence when vo is near Tref and, by incorporating an importance ratio, we can enforce a more effective regularization in DAAs. We consider a multi-arm bandit problem with 4-action space A= {ao, a1, 02, 03}, }. the reference model Tee as Tee((a) = 0.5, (1) (a2)
1.0
0.1, and (a)) 0.3 and the reward as r(ag) = 0.5, r(a,) = r(,, = 0, Vj E (2.3). We can obtain the optimal policy analytically (a)((a) exp (1/3 x r(a)). We then learn a parametrized policy (a)a)) to imitate the optimal policy n (a) by minimizing the KL divergence. KL (TTl(T²))
0.8
x² (a) x
0.6-
cue VaKL, Vintal
cos Vall, Vae)
To demonstrate that incorporating importance sampling can en- force KL regularization without the need for online sampling, we compare the gradients in terms of their cosine similarities. Specifically, we measure the cosine similarity between the gradient of the KL divergence with respect to the parameters of To (denoted as VLKL) and the gradient of the weighted squared loss (denoted as Vep.. This is expressed as cos KL. Veu). Additionally, compute the cosine similarity between the KL gradient and the gradient of the weighted squared loss with importance sampling (denoted Vo14s). represented as (L, Vorus). This analysis helps us understand how closely importance sampling aligns with the KI. regularization as the learned policy Ty moves away from Ted. We experiment with B E (0.05, 1.0) to control the deviation of no from Tee--
0.2
1.5
088
10
Figure 2:
Cosine cos (VeKL, VoH) cos (V, 1, Volus) as a func-
similar- and
ity
tion of B.
As shown in Fig. 2 under high regularization. both the gradient Vents (To, Tref) and Vau (TO, Trer) ex- hibits high cosine similarity with V, (MT Teer). However, the cosine similarity cos ( (VoKL, Ven) starts to decrease quickly when 3 becomes too small and can even be negative. On the other hand, cos KL, V1s) shows consistently high similarity across all 3, this shows that by incorporating enforee an effective regularization even when the learned policy deviate
the importance ratio, significantly from Tee.
5

---

4 Experiments and Results
In this section, we empirically evaluate IS-DAAs ability to align language models with human preferences and mitigate the reward over-optimization problem. First, in the TL;DR Summarization task, we systematically study the trade-off between the policy performance and KI. regularization achieved by different alignment methods in controlled environment where we assume to have access golden reward model as the ground- truth preferences. Next, the Instruction Following benchmark, we evaluate IS-DAAs on three standard open-ended instruction following benchmarks. Under both settings, IS-DAAs outperform existing alignment approaches and better mitigate the over-optimization problem compared to existing approaches designed for this purpose.
Models: Throughout our experiments, we use Llama-3.2-3B [MetaAl 2024a b) as the pre-trained base model. For both summarization and the instruction following, we first supervised fine-tuning Llama-3.2-3B serve the initialization for subsequent preference training.
Baselines: In addition to IS-DAAs, we evaluate several existing baselines that address the over-
optimization problem in DAAs, including: DAA objective [Liu et al. [0224 Cen et al. 2025 Fisch et al. 2025), which augments the DAAS with an additional SFT loss, we refer to this approach
Regularized Preference Optimization (RPO), x-PO (Huang al. 2025), which combines x²
with KL regularization to enforce stronger regularization. Additionally, We also consider DAAS
with rintt-rrggularization approach [Park al., 2024a]. to address the length exploitation issue
common pattern of reward over-optimization Park et al. 22024 Chen et al. 2024b)
4.1 TL;DR Summarization
For the summarization task, we consider the filtered version of Reddit TL:DR summarization dataset [Stiennon et al. 2020). Following the "controlled" environment setups of [Gao et al. 2022 Tang et al.) [024a Rafailov et al. 2023]. we assume access to the golden (ground-truth) reward model. This golden reward model learned to encourage high-quality summaries of Reddit Posts. The golden reward model will provide preference feedback to create synthetic preference dataset Dgolden (x, y y! for preference training.
Setup:
The reference model is obtained through SFT on the dedicated SFT split. For preference training, we consider 2 epochs of training with varying regularization strength 3 to investigate the reward maximization and KL divergence trade-off under different approaches.
Evaluation: We evaluate the performance of the learned policy by the win rate against the chosen summaries from Dgolden- The win-rate determined by the golden reward model. Our main results shown in Fig. [] which presents the win rate and KL trade-off across different configurations after 2 epochs of training. Our main findings are:
Previous approaches is insufficient to mitigate Reward Over-optimization As shown in Fig. under low KL constraint strength (3 = 0.01), apart from X-po, prior aligmment methods even with their proposed regularization techniques still suffer from the over-optimization problem. These methods fail to constrain the learned policy to stay close to Tet leading to decreasing performance of the learned policy. Moreover, the results in Fig. demoonttrtte the early convergence phenomenon [Par et al. 2024a], where these methods reach their peak performance after training on only 10-33004- of the data, followed by performance degradation as the training progresses with increasing KL divergence.
IS-DPO is significantly more KL efficient than other methods. Considering the square root KL divergence as the resource to be spent, we can observe that IS-DPO "spent" KL more effectively than the other methods. Under low regularization, IS-DPO effectively regularizes the policy to significantly low KL budget and avoid reward over-optimization. Under stronger KL regularization (8 (8=00 0.1), IS-DPO spends higher KL budget compared to the other approaches achieve better performance (Fig. Interestingly, the effect of the penalty in IS-DPO is akin to the early stopping
phenomenon (Fig. 3). where the training stops at a specific KL divergence / KL (Tol ( (* 5.5) to avoid over-optimization without requiring any evaluation set. We discuss this result further in Section 4.3 of the Appendix.
IP- exhibits better performance while effectively mitigating early convergence problem. As shown in Fig. 4 across different regularization strengths, IS-DPO achieves the highest averaged win rates. Furthermore, while other approaches suffer from an carly convergence problem, IS-DPO

---

continues to improve or stay flat at later epochs with a significantly lower KL budget. This shows that IS-DPO is robust to the over-optinizatoon problem.
TL.DR scaling law curve lits
Authropic- HH scaling law curve tits
0.90
0.80
0.50
Win rale
0.70
0.70
Lengt's DPO
0.60
0.60
RPO
0.50
x-PO
0.50
2
< ( (me-)ee) (mad-11))
Figure 3: Result on over-optimization of different methods TL;DR and Anthropic datasets. Results show the win rate over the reference summary as judged by the golden reward model as a function of square root KI. divergence, with the proposed fitted curve from go0222sclliglaareeerrdddd..
Clipping ratio e Ablation. We conduct ablation studies better understand how the clipping ratio e influences policy performance and regularization in Fig. [ As can be observed, a moderate value of log c 1.0 yields the highest win rate ( 92%) and associated with the highest KI. divergence. These results are consistent with the alasvvariance trade-off of clipping ratio Specifically, a higher value of allows a larger model update and better reflects the on on-policy objective. However, setting too large can lead to only a small number of samples with excessively large weights that dominate the learning signals of the other valuable samples Park et al. 2024b
Win mito Evdiccion (3m(O0)
Win ru6 Evdlrrio (2-0.05)
Wil rale Evolution (3=2.3)
rat
rate
DS-DPO
Win
Win
Wis
Lengts
0.6
IUNO
DPD
0.4
0.4
0.6-
0.0
0.5
1.0
20
0.0
0.5
100
7.0
0.0
0.5
1.0
1.5
20
Fpech Fraction
Ejucc Prontion
Rouch Pootion
Evoittin
7.5
3.0
Vic.
2.5
2.0
Festiim
Eyach Praction
Rand Phadion
Figure 4: Results on optimization dynamics of different methods. The top row shows win rate over 2 epochs, while the bottom row shows the corresponding square root KL divergence. The shaded area displays standard error over 3 seeds. Under low KI. regularization strength, IS-DPO can achieve better performance across different baselines and exhibit no over-optimization phenomenon while maintaining a significantly lower KI. budget.
4.2 Instruction Following
Sctup: For the instruction following task, we consider the Anthropic Helpful and Harmless (HH) datasct [Bai et al.
2022), Ultraeeedback dataset for preference trainining. We consider epoch of training with 3
0.05 as standard configurations for offline alignments [Rafailov et al. 2024 Gao
ct al. 2024 Guo et al. 2024].
Evaluation: We evaluate methods fine-tuned on the UltraFeedback on 2 widely-adopted benchmarks: AlpacaEval 2.0 [Dubois et al.) 2025) and MT-Bench [Zheng al., [023). For Anthropic-HKs We evaluate the performance using a reward model trained on preference dataset and querying GPT-4 to serve as a proxy for human evaluation. We provide a detailed evaluation setup Section of the Appendix. Similar to the summarization task, we compare the learned policy against the chosen responses from the preference data and report the win rate of various methods using GPT-4 and the trained reward model.
7

---

5.25
0.92
5.20
Win-rate
0.90
5.10
0.88
0.25
0.5
1.0 0.25
0.5
1.0
Clipping milo loge
Clipping loge
Figure 5: The ablation results for the clipping ratio e in Eq [3). Results show model win rate and KL regularization effect over the summarization dataset across 3 random seeds. We observe that higher clipping ratios allow for larger policy updates, resulting in both increased KL divergence between the policy and reference model and improved win rate.
Results: Experimental results shown in Tab. D For Anthropic-HH, IS-DPO demonstrates notable improvements over the other baselines, in both evaluations using both the reward model and GPT-4. Specifically, IS-DPO achieves approximately 3% improvement GPT-4 and the reward model. On AlpacaEval 2.0 and MT--Bench, our results show that IS-DPÓ consistently outperforms state-of-the-arr methods such DPO and RPO, and demonstrate competitive results and achieve competitive performance with x-PO on AlpacaEval 2.0.
the standard DPO, judged by
Anthropic-Hi HH
AlpacaEval 2.0 (%) GPT-4 WR (%)
MT-Bench GPT-4 Score
Method
RM (%)
GPT-4 WR (%)
DPO
81.77 + 0.5
71.1 + 0.6
5.81
5.39
RPO
81.50 + 0.4
66.5 + 0.4
5.77
5.29
Length-DPO
83.85 0.8
58.0 1.0
3.71
5.24
X-PO
81.25 + 1.5
67.2 - 1.4
6,85
5.09
IS-DPO (Ours)
84.60
1 0.6
74.0 1 0.9
6.33
5.44
Table Average win rate and standard deviation from 3 different seeds against the chosen responses on the different dialogue datasets. The best results are bolded.
4.3 Importance Sampling as Support Constraint
This section analyzes why Importance Sampling can serve as an implicit early stopping mechanism, effectively avoid- ing over-optimization. Consider discrete distributions P(X)
Importance Batio
and Q(X) defined over random variable . We wish to es-
0.5
timate a function (xx under P: Ex-p [f(x)) and IS achieves
0.4
this estimation through Egn
Q(x) fx)) Under the support
P(z)
0.3
assumption supp (P) C supp (Q), IS will be unbiased
timation. However, when this assumption does not hold,
0.2
0.4
0.6
0.8
10
instead estimates Eupp(() (x)) which is biased.
Fraction of Epach
In the case of IS-DAAs, where me is P and Nee is Q, this biased estimate can be a blessing in disguise. To elaborate, minimizing the objective in Eqn. (TT) is equivalent to mini- mizing:
Figure Averaged importance ratio < ( xx) during training.
To ( (x)
log To Ty |x) T [x)) )
Ex-DDy D.y ylw.s log BBlog
(14)
[x)r)
where # is defined as:
1 (x)) > 0J (x
To (y(x)
(15)
Ey 1 [ (y(x) (y()
By adopting Importance sampling, IS-DAAs only provide updates to my when my samples are in Thet's support regions. When the support assumption violated, supp (mo) I supp (mer), IS-DAAs will not update ny. thus avoiding the extrapolation issue.

---

Empirically, we observe that, throughout the training process, the importance ratio Te. Thtt decreases as training progresses (Fig. 6). This means that several samples have low probability under ng. or <o(y|x) x) x 0). This leads to my s updates with fewer and fewer samples, inducing an early stopping effect. Without IS, further training on these samples with small importance ratios (no /ret) can lead to performance degradation.
5 Related Works
5.1 Reinforcement Learning from Human Feedback
In recent years, RLHF has been dominant framework for aligning LLMs with human preferences. The RLHF pipeline begins with supervised fine-tuning (SFT) of the LLM using next-token prediction objectives on dataset of high-quality, instruction-oolloving responses. This is followed by fine- tuning the SFT's LLM using reinforcement learning (RL) algorithms such as PPO (Schulman et al.) 2017] or REINFORCE (Williams 2004), to maximize an "explicit reward" (based on the preference data) with a KL regularization to the reference policy. Alternatively, DAAS, such as DPO [Rafailoy al. 2023] and IPO (Azar al. 2024), aim simplify the RLHF pipeline by directly optimizing the LLM on human preferences without an explicit reward model or RL.
5.2
Over-optimization in DAAS
Gao et al. 2022 refer to over-optimizaation as the situation where algorithm consumes large optimization budget without improving (and even reducing) its performance. In this study, the KL divergence KL [TTol is used as an optimization budget since it measures how far the optimized policy na drifts away from the reference policy Teer during training. Rafailov et al. (2024) study the trade-off between the KL divergence and the policy performance under three DAA objectives: DPO, IPO, and SLiC. They observe clear over-optimization after a certain point during training when an additional increase in the KI. budget leads to a decrease in the model performance. This pattern persists across model sizes, with smaller models often exhibiting clearer signs of over-optimization. Regularization methods, such as length regularization park20024disentanginn rafailov2024scalingg can not mitigate this problem. Furthermore, Tang et al. [2024a] observes that both online and offline variants of DAAs suffer from wwerooptimization, while the online DAAS achieve better budget and performance trade-offs than the offline kinds.
5.3 Performance gap between online and offline alignment
We connect the over-optimization problem in offline alignment algorithms to the distribution shift problem encountered in offline RL [Levine et al. [2200] Kumar et al. 2000).. Specifically, during training, the policy Ty is optimized using data generated by the reference model Tief. However, during deployment, the policy must act based on its own generated distribution, which may differ significantly from the training data. This discrepancy can lead to performance degradation, especially when the LMs encounter states that differ significantly from those in the offline data [Chen et al.) 2024a]. While (DAAs) are designed for off- aff-policy learning, they still suffer from this distribution shift problem Rafailov et al. 2024 Tang et al., 2024c). Our work demonstrates that, under such conditions, DAAS exhibit weak regularization and fail to regularize the learned policy when the LM deviates far away from its original (reference) distribution. Inspired by this observation, we propose IS-DAAs, which estimate an on-policy learning objective given only the offline preference data and effectively mitigate the distribution shift problem in DAAS.
6 Discussion
We study reward over-optimization in Direct Alignment Algorithms (DAAs). We show that one of the main sources of reward over-optimization in DAAs is the mismatch between offline distribution and the LM policy. To reduce this distribution gap problem, we introduce IS-DAAs, a simple yet effective method estimate samples under the LM policy distribution given samples from the offline distribution. The proposed method is also able to overcome the high variance issue of importance ratio estimation. Our results showed that IS-DAAs outperform other regularization methods and effectively resolve the over-optimization issues in DAAS.

---

Acknowledgements
This project is supported by VinUni-Illinois Smart Health Center (VISHC), VinUniversity. Duy M. H. Nguyen and Daniel Sonntag are supported by the No-IDLE project (BMBF, 01IW23002), the MASTER project (EU, 101093079), and the Endowed Chair of Applied Artificial Intelligence, Oldenburg University. Anji Liu is supported by the National University of Singapore under its Start-up Grant (Award No: SUG-25IRE22505).
References
M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS, pages 4447- 4455, 2024. URL httpp:://ppoceeeddnnngggg/// mlr /hress/w2B//hhhhhaahhhi--- gheshlaghii-zzrr242224 htmml
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, Conerly, S. E. Showk, N. Elhage, Z. Hatfield- Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. B. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862, 2022. URL htttht: org/10 10. 48550/ arxiv 2204.058633
S. Cen, J. Mei, K. Goshvadi, H. Dai, Yang. Yang, D. Schuurmans, Y. Chi, and Dai. Value-incentivized preference optimization: A unified approach to online and offline RLHF, In The Thirteenth International Conference on Learning Representations, 2025. URL https //openreveww ntt/:/rumtt//oommmm/m iidiimddddiiiiiiii tQuoow
A. Chen, S. Malladi, I.. H. Zhang, X. Chen, Q. Zhang, R. Ranganath, and K. Cho. Preference learning algorithms do not learn preference rankings. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024a. /ttpss:////p/eerree
net/ wet/eorrmiiddoeeeeeeeeeerree /formmmmmd-D0002gg000
L. Chen, C. Zhu, J. Chen, D. Soselia, T. Zhou, T. Goldstein, H. Huang, M. Shoeybi, and B. Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHE. In Forty-first International Conference on Machine Learning, 2024b. URL
X. Chen, D. Diao, H. Chen, H. Yao, H. Piao, Z. Sun, Z. Yang. R. Goebel, B. Jiang, and Y. Chang. The sufficiency of off- off-policyness and soft clipping: Ppo is still insufficient according to off- policy measure. Proceedings of the AAAI Conference on Artificial Intelligence. 37(6) Jun. 2023. doi: 10.11699aaiii.vv3 w3w.........6..... URL https: aaai. org/ index.ndexx./ wwppppppaaaaa
article/vien/256 25864
F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: //p.oe/ed/..cceeeeen/eo...ee http://rrrreiiin....eeeeerrppppp files/ http://pppprooetttt///////aaaaae
M. Ding. S. Chakraborty, Agrawal, Z. Che, A. Koppel, M. Wang, A. Bedi, and F. Huang. SAIL: Self-improving efficient online alignment of large language models. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024.
Fdd
Y. Dubois. B. Galambosi, P. Liang, and T. B. Hashimoto. Length alpacaeval: A simple way to debias automatic evaluators, 2025, URL BtRp:stss///axxi// org/ 4bs/2444400445
V. Elvira and L. Martino. Advances in importance sampling, Feb. 2021. URL http://dx.doi 00000/7781118445111 01100000000/9781114441112222.oo2 stttt0088
A. Fisch, J. Eisenstein, V. Zayats, A. Agarwal, A. Beirami, C. Nagpal, P. Shaw, and J. Berant. Robust preference optimization through reward model distillation, 2024.
A. Fisch, J. Eisenstein, V. Zayats, A. Agarwal, A. Beirami, C. Nagpal, P. Shaw, and J. Berant. Robust preference optimization through reward model distillation. Transactions on Machine Learning Research, 2025, ISSN 2835-8856. URL https: //tp:///e/////eee//e/mmmm
10

---

L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization, 2022. URL https://arxiv.c org/ abs/2210. 10700
Z. Gao, J. D. Chang, W. Zhan, O. Oertell, G. Swamy, K. Brantley, Joachims, J. A. Bagnell, J. D. Lee, and W. Sun. REBEL: Reinforcement learning via regressing relative rewards. In ICML 2024 Workshop on Models of Iuman Feedback for Al Alignment, 2024. URL https: //openreview. hwwp:/neeefeeeeeeee net/ffoum www.oormmimiiiiiiii 1d=4SkidUUPP
S. Guo, B. Zhang. T. Liu, T. Liu, M. Khalman, F. Llinares, A. Ramé, T. Mesnard, Y. Zhao, B. Piot, J. Ferret, and M. Blondel. Direct language model alignment from online ai feedback. CoRR, abs/2402.04792, 2024. URL https : //dp:///r////....////. 48550/ arviv. 2402.04799
A. Huang, W. Zhan, T. Xie, J. D. Lee, W. Sun, A. Krishnamurthy, and D. Foster. Correcting the mythos of KL-regularization: Direct alignment without overoptimization via chi-squared preference optimization. In The Thirteenth International Conference on Learning Representations, 2025. https://t:///e
H. Ji, C. Lu, Y. Niu, P. Ke, H. Wang, J. Zhu, J. Tang, and M. Huang. Towards efficient exact optimization of language model alignment. In Forty-firsi International Conference on Machine Learning, 2024. htt.htttt///pperevrrrie www.:/wwwweeeeeeeeeeeeeeeeeeeeee feret/forummit (it.mmmmdddkkkke ss33p3
A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems. volume 33, pages 1179-119) Curran Associates, Inc., 2020. URL https::/prooeeddnnss neurips..... ctt.cc//ppp cw/paperrppppe files/ pptps//pppppp///////// pdf
S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
Z. Liu, M. Lu, S. Zhang, B. Liu, H. Guo. Y. Yang, J. Blanchet, and Z. Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In The Thirty-
eighth Annual Conference Neural Information Processing Systems, 2024a. URL https //openreviev.o wwtp:eiwwwweeeeeeeeeeeeeeeeeeeee www.:eeetorrri.... satiummddddca0PPhehee
Z. Liu, M. Lu, S. Zhang, B. Liu, H. Guo, Y. Yang, J. Blanchet, and Z. Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024b. URL
https://opperrrv//
.nt/
W. Meng, Q. Zheng. G. Pan, and Y. Yin. Off-policy proximal policy optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8)9162 9170, Jun. 2023. doi: 10. 1609/aaai. v37i8. 7.0999 URL https: / //js.. aaai org/index. iid/xxiiexxxxx php/ haPP/AAAI/artte Awt.///aaieeeevee///////..
MetaAI. Introducing llama 3.1: Our most capable models to date. 2024a. URL htt.h:ttps//ee meta com/blog/metee www.g////eaaaaeem 1111111111 amaamma11111 1/
MetaAl. Llama 3.2: Revolutionizing
edge ai and vision with open. URL https://ai. meta. com/blog 1lama 3- 2- connect- -000--2224--00 2024-vision vision- edge-mobile devices/
customizable models.
2024b.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F Christiano. J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages sociates, Inc.,, 2022. URL httpattp///rrooceiiingg mew.eeeededddineeeeeeeeeeeeeppee newrips cc capppr lfcerrrfilllo wtt.ef//papee/////// ftiebbateeeee34475555555503333.. ConfeePeeeeeeeroroeeeeeeeee pdf
Curran As-
R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 4998-5017 Bangkok, Thailand and virtual meeting, Aug. 2024a, Association for Computational Linguisties. URL httpss// ttttps://aclantholle. www.llannhoooooo......o 2024. findings----iin aaciinngg-ll...... 297
11

---

S. Park, K. Frans, S. Levine, and A. Kumar. Is value learning really the main bottleneck in offline RL? In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024b. URL
1h7NH11
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-sevendl Conference on Neural Information Processing Systems, 2023. URL https::/openreeeeettee HPUSIXJaa9
R. Rafailov, Y. Chittepu, R. Park, H. Sikchi, J. Hejna, W. B. Knox, C. Finn, and S. Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. In ICML 2024 Workshop on Models of Human Feedbuck for AI Alignment, 2024. URL
id=WTrwDXdRTS
Schulman. Approximating kl divergence, 2020. URL html
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017. /tttttss:://// /tattps://ariie org/abs/ 07707003377
Y. Song, G. Swamy, A. Singh, D. Bagnell, and W. Sun. The importance of online data: Under- standing preference fine-tuning via coverage. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024. URL sIcYfmG6Zu
net/forum?id=
N. Stiennon, Ouyang, J. Wu, D. Ziegler, Lowe, C. Voss, A. Radford, D. Amodei, and Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processsing Systems, volume 33, pages 3008-3021. Curran Associates, Inc., 2020. URL httppe:/ neurtiigggn.e.....poeie..... neurips cc/ paper_ files/ http:///000/01118662222226666660
Y. Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y. Cao, E. Tarassov, R. Munos, B. Ávila Pires, M. Valko, Y. Cheng, and W. Dabney. Understanding the performance gap between online and offline alignment algorithms, 2024a.
Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello. R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. A. Pires, and B. Piot. Generalized preference optimization: A unified approach to offline alignment. In Forty-first International Conference Machine Learning, 2024b. URL https //operreiee net. meet/forumidd -iniaiiidddgnnnnA....
Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. A. Pires, and B. Piot. Generalized preference optimization: A unified approach offline alignment. In Forty-first International Conference on Machine Learning, 2024c. URL https
R. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-2::0 2566, 2004. URL BtRLsttp::// http:/ppp..emanttcccccccocooo/// org
CorpusID:2332513
S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mci, G. Wang, C. Yu, and Y. Wu. Is DPO superior to PPO for LLM alignment? a comprehensive study. In Forty-first International Conference Machine Learning, 2024. URL htttps:/
net mett/forummi1--XHHBTTT.
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, 7. Wu, Y. Zhuang, Z. H. Zhang, J. F. Gonzalez, and I. Stoica. Judging LLMM-as-ajuddee with MT-hench and chathot arena. In sevethh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. BoLLitttps://pperrvviee
Lin, Z. Li, D. L. F. Xing,
B. Zhu, M. Jordan, and Jiao. Iterative data smoothing: Mitigating reward overfitting and overop- timization in RLHF. In Forty-first International Conference on Machine Learning, 2024. URL https://openreviek net net./foruum//deWWeeMMMllo....
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provided experiments to support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We included a discussion section 6 and the Limitation discussion in section A Appendix.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
All assumptions are already stated in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: we provided experiment descriptions in the main paper and appendix. Further- more, we will release GitHub implementation if the paper is accepted.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: We will publish the code when the paper is accepted or through an anonymous link if some reviewers ask for it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipps.cc/pulli///uuu guwdes/ddee/// wwt.:Suwwisssispoooooooooooooooi
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided details for experiments in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct experiments on diverse datasets and run same configurations with different seeds to report error bars suitably and follow the protocol used by previous works for fair comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
15

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these information in section C Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///haarrps..ccccppp//////e
Answer: [Yes]
Justification: Our paper follows the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work is to advance alignment algorithms that avoid over-optimization and ensure the development of models that are safe for real-world deployment. We have mentioned in section A Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We experiment with public models and public datasets. Our work does not pose such risks as we are training these model for alignment.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited papers and resources used in experiment. The pretrained model comes from 3la-m-3.22-3B which are classified under the Community Both the TL:DR,
License agreement: https://huggingfaace http:///ugginiffccc.coooemaaaaaa Anthropic-HH, and UltraFeedback datasets used in this work use the MIT License. Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasettcoodemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We don't have experiments involving crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
18

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: We LLM for grammar checking only.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non -ta-dtaddrrd components.
Please refer to our LLM policy (https: http::/://oourpssscc////oooooorp for what should or should not be described.
19

---

A Limitations and Societal Impacts
Our discussion highlights ineffective regularization with direct alignment algorithms used widely to align to human preferences. In this work we analyze and resolve these issues. However, We still assume an underlying Bradley-Terry model of human preferences, as these models might not be accurate in explaining the ways humans give preferences and do not experiment with larger models due to limited computational resources. Furthermore, our IS-DPO implicitly assume hat the policy produced the offline dataset 'D is generated from Tef while might not holds in practice. Our work is to advance alignment algorithms that avoid over-optimization and ensure the development of models that are safe for real-world deployment.
B Proofs and Derivations
B.1 IS-DPO is an unbiased estimation of online DPO
Theorem Restated. Assuming Supp (Tyy)) Supp (Te). then objective (Eqn. (T)) is unbiased estimation of the objective in Eqn. (16) and the gradient concerning нe of the weighted squared loss equals to the gradient of KI. divergence. Proof. Online-DPO objective can be expressed as:
COnline-DPO(PR Tzet)
Blog To(y"pp x)
To T |x) Teet (e (x [x)
Ex y"" w()oo((11))) log
Teat (y B log
Expanding the expectation leads:
Toly", x11x) log a Glog (x(
TT (xx -Blog
=-ExD
Σ
Toerly
x (x)
y'
Σ "y y' (x) To(y"" yy xx)
Tr y logo Blog To To(y"
-Blog To (1) Trefly' (x))
ExxD
Trerly (1))
y"
To(y" y log Blog Te (x) Ty
To (x) Tyer (x)
Ex
Bly merl))))))))))
B Blog
" " . yf(xx x)
Where we denote Ty 20 y²) = ( (x) (> (x). This yields Eqn.
Similarly, given prompt x, consider the gradient of KL divergence:
T x) Trer (y(e)
(KLL(Tol(TTe))) = Σ (ylo)) + Σ log
(y(o())
We can drop the first term since Σ VeTooY|X= = V Σ [
VL
0. We now consider
the gradient of
loss with Importance Sampling:
Ty ('1xx xx)
V log Ty xx
To / |x
27
E Ex~DD Dxy" (() (1)
log
2
Tieer (y w yd xx)
Treel Trer(y"0xx
( (y'(x
log Te T xx)
To ((((((()
ExwDD,y )--
log
(vo log Te "y Vo
lolog log No (y' (x'lx))
Treel Trer (x)
Trer(y! (y xx)
log To(y" xx)
To To(y"11 (x)
Ry of (+bx)
Vo log To (y (x)
log
Vo log moo (8/1x))
Ty xx)
Teee (y'1x)
(x)x)
Ex D.ywDaynmti (
log Trerly/ xx)
Vo log (x(xx)
EXnD (V ( Tree)]
Which concludes the proof.
20

---

B.2 Detailed derivation of regularization effect in DAAs
V0 2
Vo N02Pi PQ
Te (y ///x)
-20(x.))
log
V log n0 (y/x)
Te (y(x)
Te (y(y(x)
2E (B(xx))((..aa)
log
E -
V log To (y|x)]
Txo ()(x)
log Te (y' (x)
V log no (y|x) +
Trer(y|x x
2DKL. [Trer (xx (xxx)) (V log (y(()
C Training Details
For the following dialogue history a chatbot, which response is more helpful?
Dialogue history:
<dialogue history?
Response A: <Response A>
Reaponse <Response B>
FIRST provide one-sentence comparison of the two responses and explain which you feel is more helpful. SECOND, a new line, state only "A" "B" to indicate which response is more helpful. Your response should use the format:
Comparison: <one-sentence comparison and explanation>
More helpful: <"A" or "B">
Table 2: Prompt for GPT-4 evaluation on the dialogue generation task. Texts in blue are placeholders to be substituted by the real data.
SFT Training For the summarization task, we use the SFT split of Reddit TL;DR summarization. For Anthropic-HH we the chosen responses from the preference dataset for SFT stage. We pool together both datasets into a single SFT dataset.
Preference Training For TL.DR summarization dataset, we train all methods for epochs. To evaluate the efficiency of addressing the over-optimization problem, we vary the regularization strength 3 E {0.01, 0.05, 0.1).
Across all SFT and Preference training, we use global batch size of (with 4 gradient accumulation steps), and AdamW optimizer with a learning rate of x 10-6 (cosine learning rate scheduler warm- up for 100 steps) and a max length of 640.
Golden Reward Training details We follow the synthetic setup where the golden reward model serves human evaluation and provides preference labels [Gao et al. 2022 Tang et al. 2024a
We first initialize the golden reward model with a SFT version of Llama-3. 3.B-BB on the pooled SFT data. We then train the golden reward model on the combined preference of the TL.DR and Anthropic-HH dataset. Specifically, we use a batch size 128, with a learning rate of 10 we train for one epoch with cosine learning rate schedule.
21

---

The golden reward model achieves high validation accuracies with 75.2% validation accuracy, showing high correlation with human preferences.
Details of KL Divergence Estimation this paper, we construct an unbiased estimate of KI.(To) (rer ((ee) by sampling. More specifically, first sample N prompts (x*) from the evaluation set, for each prompt æ², we sample response y ~ TT(-(²²) from the learned policy иT. Following [Schulman 2020 The KL divergence is estimated as follows:
To Tee (y'(z²))
To(yy'(z²) -1)
log
+
N
Teerlyi
Compute Resources Specification We train and evaluate our models using NVIDIA 4xH100 GPUs. All evaluations are computed with "gpt-4o-mini" as judge, with random positional flips to avoid position bias.
D Evaluation Details
Golden reward Evaluation we sample 2 completions per prompt from the learned policy with 512 prompts from the evaluation set. We sample with temperature т = 0.7 and top-p sampling with p 0.95. to evaluate its performance. To calculate the winrate, we consider all combinations of pairs between the completions generated by the learned policy and the reference completions from the preference dataset, and then compare the scores from the golden reward model on the pair of generations to calculate win-rate.
GPT-4 Evaluation For GPT-4 evaluation, we sample 256 prompts from the evaluation set. For each prompt, we sample I completion from the learned policy. To evaluate the dialogue generation task, use the prompt shown in Table 2 similar to [Ji al. 2024 with random position flipping avoid position bias
E IS-DPO as an approximation of Online-DPO
TL; scaling law curve fits
1.00
0.80
Win rate
0.60
IS-DPO
DPO
Online-DPO
0.40
2
4
6
8
KL
(T
x
(re)(-(2))
Figure 7: Scaling law curve fits between DPO, IS-DPO and Online-DPO
have conducted experiments understand how well the IS-DPO can better approximate the Online- Guo et al. 2 Our findings are below:
22

---

IS-DPO can closely approximate Online-DPO up to specific KL divergence threshold: We observe that IS-DPO also follows a similar performance trajectory Online-DPO in the low to moderate KI. divergence regime. In this region, both methods show a similar and steady increase in win rate. This suggests that IS-DPO is capable of closely approximating the behavior and performance of Online-DPO within a certain KI. "budget".
On-policy benefits from large KL "budget": While IS-DPO does not show signs of over- optimization, it also does not extrapolate to higher KL regimes, while Online-DPO can find better policies under higher KL regimes. This indicates that on-policy optimization remains beneficial when the optimal policies lie farther from the as Online-Dee Diin is able to leverage a larger KL budget.
We also provide comparison between Online-DPO and DPO. Interestingly, we observe that under a low KI. budget, DPO appears to outperform Online-DPG However, it sharply declines as KL increases, indicating over-optimization, while IS-DPO and DPO effectively utilize larger KL. budget to achieve higher performance.
Comparison with DPO:
F The role of Importance Sampling in IS-DPO and PPO.
While Importance Sampling (IS) has also been used to address distribution shift in PPO, its usage is fundamentally different from our paper.
PPO uses clipped importance sampling (IS-clip) to improve sample efficiency. Since each batch is typically discarded after single gradient update in standard policy gradient methods, IS allows PPO to reuse the same batch multiple times, extracting the most information from each batch. In contrast, our objective
enforce an effective regularization, prevent them from pushing up the probability of responses that are out of the offline data distribution, mitigating the over-optimization issue. In addition, in PPO, the clipping mechanism acts as a surrogate trust region to prevent the policy from deviating too far from the previous one (which can help mitigate over-optimization), but in fully offline setting (as our paper), this mechanism becomes overly conservative where gradient is zero when the probability ratio falls outside the clipping range, limiting the potential improvement from (Chen et al. 2023 Meng al. 2023]. Our clipping strategy, conversely, mitigates excessively large updates with IS while still providing meaningful updates even when the ratio lies outside the clipping region.
G Theoretical Insight of clipping ratio and KL regularization.
Proposition 2 Assuming supp(To) supp( then the variance of the importance weight
Var
Var
is an upper bound of KL divergence: Not
Nec.
To
KL(To (Te)) S Var
(16)
Teef
The proof straightforward by utilizing the inequality log x 0. This proposition highlights that when regularization is weak (i.e., small B), the policy can deviate significantly from Tht. leading to high variance in the importance weights. This motivates the use of smaller clipping threshold to control variance and stabilize training under such regimes.
Additionally, also conduct the relationship between the regularization parameter 3 and the clipping ratio e. We present the win-rate and KL values for TL:DR summarization dataset below; We observe that increasing the clipping threshold allows for larger policy updates, which in turn leads to higher KI. divergence across all values of 3. Interestingly, at low regularization levels (small (9), using a smaller clipping ratio yields better performance, as it helps prevent over-optimization. Overall, our where smaller clipping threshold should be used for small regularization strength avoid over-optimizaion issue.
experimental results is consistent with Proposition 2
H Computational cost of calculating importance ratio.
The computational overhead of calculating the importance ratios is negligible and results almost the same time complexity DPO. Despite adding the additional importance ratio calculation step to
23

---

Win rate
KL (Tell [Taf
0.01
91.2
92.0
89.7
0.01
15.4
20.1
22.5
33
0.05
92.2
94.5
13
0.05
17.6
18.2
0.1
93.6
94.0
91.4
0.1
15.9
18.8
22.3
0.25
0.5
1.0
0.25
0.5
1.0
log
log
Figure 8: Win rate and KL divergence across different e and B.
the original DPO loss, this ratio can be efficiently computed using intermediate quantities already available in the DPO loss calculation. Specifically, presented in the paper, the importance ratio can be computed in the log-space as follows:
w(y yi) exp (log (y log (y:)) log ( log (yr))
(17)
Hence, IS-DPO only adds small calculation, negligible compared to the much more heavier computations in the vanilla DPO, This implementation also straightforward and simple, requiring only single additional line of code.
To empirically validate this claim, we report the average training time over 100 steps for both DPO and IS-DPO. We use 4 A100-80GB GPUs, batch size of 16, and LLaMA-3.. 3.2-3B as the base model:
IS-DPO
DPO
92.829s + 5.144 92.571s + 4.837
As can be observed, the training time for IS-DPO nearly identical that of the vanilla DPO, confirming that our approach is computationally efficient and negligible in overhead.
I
The necessity of support constraint in IS-DPO.
While the optimal policy under the DPO objective satisfies:
n [ylx)
exp((xx,)))) Z(x)
(18)
Thus shares the same support as Tref- However, prior works RRfailov et al. 2024 Azar et al. 2024 Song et al. 2024 Xu et al. 2024] have shown that DAAs assume that Tref has full support over the entire prompt-response space to achieve n². practice, this assumption rarely holds, as preference datasets only cover a small fraction of the prompt-response space. Consequently, multiple distinct policies can achieve the same global optimum of the DPO objective [Xu et al., 22024] Azar et al.. 2024 Rafailov al.) 20244, including policies that assign probability mass to responses outside the
support of Tee.- This contrasts to the original RLHF framework, which prevents generating responses that are of support via explicit KI. regularization. Thus, our proposed importance sampling method further enforces the support constraint, helping to avoid this failure mode.
Connection to principle of pessimism in Offline RL.
At a high level, the principle of pessimism explicitly subtracts uncertainty-basee penalties such as KL. divergence divergence from the estimated value with the aim preventing overestimation of the value of candidate policies in regions with low data coverage or high uncertainty.

---

IS-DPO also incorporates form of pessimism by assigning lower weights to trajectories that are less likely under the learned policy, effectively penalizing policies that place high probability mass in areas poorly covered by the data. This mechanism aligns conceptually with pessimism- regularization Liu al. 2024a Zhu al. [024]. Moreover, there deeper theoretical connection worth
exploring: the variance of the IS estimator Varm.er
is proportional the chi-squared divergence Weed
between (T0, Wref) This observation suggests promising direction formalize the connection between importance sampling and pessimism. x² divergence has been used in L.M alignment to impose stronger regularization, thereby effectively mitigating over-optimization issues. We believe that bridging this connection more formally, possibly through uncertainty-awazt confidence intervals, a promising avenue for future work.
25


Reviews:
[{"review_id": "MYCWdXJS39", "reviewer": "Reviewer_fRzo", "summary": "* The paper proposes to avoid reward hacking in the offline direct reward optimization setting by regularizing through importance sampling.\n* Theoretically, KL regularization applied in online reward optimization technique is computationally intensive when extended to the direct reward setting as it requires multiple samples to be generated from the trained policy model. This paper demonstrates that importance sampling using a Clipped product of likelihood ratios to weight the direct reward is sufficient to induce the KL-regularized reward training objective. \n* Further, empirical results on popular benchmarks like AlpacaEval and MTBench demonstrate the efficacy of the approach in achieving higher win rates with a lower KL penalty.", "questions": "Adding missing evaluations would improve the soundness of the results.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n* The theoretical analysis of the regularization proposed is sound with valid comparisons with online KL regularized reward learning\n* Empirical results demonstrate comparable or sometime higher win rates at lower KL penalty\n* The sample efficiency in training direct reward learning is maintained while regularizing.\n\nWeaknesses:\n* The statistical significance of the results in Figure 3 and 4 are not clear as many of the baseline approaches are overlapping, with an exception of a low KL penalty (beta) setting where the importance sampling based approach performs better\n* The analysis of the online DPO baseline which is cited as a motivation to approximate is missing in the results. It would be good to understand the headroom in the approximation proposed between this computationally intensive alternative", "was_revised": true, "final_justification": "Update after the author response:\n\nThe soundness of the results are of value as the paper address the issue of overoptimization / reward hacking in direct preference optimization.", "rebuttal": "Thank you for your review and positive feedback on our paper. We appreciate your acknowledgment of the clarity and organization of our work.\n\nPlease find our response to your comments below.\n\n**Q1. The statistical significance of the results in Figures 3 and 4 are not clear as many of the baseline approaches are overlapping, with the exception of a low KL penalty (beta) setting where the importance sampling-based approach performs better.**\n\n**Answer:** Thank you for your comment. We will revise the figures in the camera-ready version to clearly indicate statistical improvements. We'd also like to clarify that our evaluation setup follows prior works [Rafailov et al. 2024, Tang et al. 2024a], where Figures 3/4 illustrate the trade-off between KL divergence and the policy performance for all the methods. We kindly ask the reviewer to disregard the visual overlap in the plots. The key takeaway from these figures is that \"the other approaches suffer from an early convergence problem, while IS-DPO continues to improve or stay flat at later epochs with a significantly lower KL budget.\" (lines 241-243). Consequently, the final win-rate results (summarized below) demonstrate the effectiveness of IS-DPO:\n\n| Method | $\\beta=0.01$| $\\beta=0.05$ | $\\beta=0.1$|\n| -------- | -------- | -------- | - |\n|    IS-DPO  | **91.57**   | 93.35     | **94.8**\n| Length-DPO| 60.25 | 88.67 | 88.83|\n| RPO | 62.37  | 88.25 | 88.22\n| DPO | 62.98 | 89.65 | 91.73\n|$\\chi$-PO | 76.46 | **93.75** | 93.06\n\n\n**Q2. The analysis of the online DPO baseline, which is cited as a motivation to approximate, is missing in the results. It would be good to understand the headroom in the approximation proposed between this computationally intensive alternative. Adding missing evaluations would improve the soundness of the results.**\n\n**Answer:** Thank you for a great suggestion. We have conducted experiments to understand how well the IS-DPO can better approximate the Online-DPO. As we're not allowed to include a Figure, we provide the (reduced) tabular version of the results instead and will add the scaling law curve version of Online-DPO in our camera-ready version. Our findings are below:\n\n**IS-DPO can closely approximate Online-DPO up to a specific KL divergence threshold:** We observe that IS-DPO also follows a similar performance trajectory to Online-DPO in the low to moderate KL divergence regime $\\sqrt{\\text{KL}}\\left(\\pi_\\theta ||\\pi_{\\text{ref}}\\right)\\approx 6$. In this region, both methods show a similar and steady increase in win rate. This suggests that IS-DPO is capable of closely approximating the behavior and performance of Online-DPO within a certain KL \"budget\".\n\n**On-policy benefits from large KL \"budget\"**: While IS-DPO does not show signs of over-optimization, it also does not extrapolate to higher KL regimes, while Online-DPO can find better policies under higher KL regimes. This indicates that on-policy optimization remains beneficial when the optimal policies lie farther from the $\\pi_{\\text{ref}}$, as Online-DPO is able to leverage a larger KL budget.\n\n**Comparison with DPO:** We also provide a comparison between Online-DPO and DPO. Interestingly, we observe that under a low KL budget, DPO appears to outperform Online-DPO $\\sqrt{\\text{KL}}(\\pi_\\theta ||\\pi_{\\text{ref}}) \\approx 4.5$. However, it sharply declines as KL increases, indicating over-optimization, while IS-DPO and DPO effectively utilize a larger KL budget to achieve higher performance.\n\nDPO result:\n|  |  | | | | | |\n| - | - | - | - | - | - | -\n| Win-rate     |   68.8   |    86.9  | 91.4 |86.6| 75.0 |57.6 |\n| $\\sqrt{\\text{KL}}$ |1.70 | 3.10 | 4.51 | 5.91 | 7.32 | 8.72 |\n\nIS-DPO result:\n|  |  | | | | | |\n| -------- | -------- | -------- | - | - | -| -|\n| Win-rate     |  57.15    |   73.6   | 84.0| 90.0 | 92.8 | 92.78| \n| $\\sqrt{\\text{KL}}$ |1.46 | 2.37| 3.28| 4.19| 5.10| 6.02 | \n\n\nOnline-DPO result:\n\n|  |  | | | | | |\n| -------- | -------- | -------- | -| - | -|- |\n| Win-rate     |  56.11   |  77.06    | 88.7 | 94.07 | 94.7 | 93.6\n| $\\sqrt{\\text{KL}}$ |1.726 | 3.17| 4.63|6.08 | 7.53 | 8.25"}, {"review_id": "5n84ojwZFJ", "reviewer": "Reviewer_PDSf", "summary": "This paper addresses the problem of reward over-optimization in offline Direct Alignment Algorithms (DAAs) like DPO. The authors hypothesize that the performance degradation observed in these methods stems from an inadequate implicit KL regularization, which becomes ineffective as the trained policy (π_θ) diverges from the initial reference policy (π_ref). This occurs because the DAA objective is optimized using a static, offline dataset sampled from π_ref, failing to account for the distribution shift during training.\n\nThe proposed solution Importance Sampling DAAs (IS-DAAs), directly tackles this issue. It modifies the standard DAA objective by multiplying the loss for each preference pair by an importance sampling ratio. Extensive experiments on summarization and instruction-following tasks demonstrate that IS-DAA significantly mitigates over-optimization, achieves a better performance-vs-KL-divergence trade-off, and outperforms existing regularization methods, especially under low β (KL penalty) settings.", "questions": "See weakness.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n- The paper tackles a recognized and important issue in LLM alignment: the tendency of offline Direct Alignment Algorithms (DAAs) like DPO to over-optimize and degrade in performance.\n- The proposed method is straightforward, which applies the standard technique of importance sampling to the DPO loss to correct for distribution shift.\n- The authors provide empirical results across several benchmarks that show some improvement over baseline DAA methods, particularly in mitigating performance decline during training.\n\nWeakness:\n- The core proposed by the authors that offline algorithms suffer from distribution shift as the policy moves away from the data-generating policy, is a well-known problem in the field of offline reinforcement learning. The proposed solution, applying importance sampling, is a standard method to tackle this, which has already applied in PPO. As such, the paper's conceptual contribution feels incremental rather than novel, amounting to the application of a known technique to the specific context of DPO.\n- The method's stability and performance rely entirely on the clipping ratio. However, the paper offers no principled way to set this crucial hyper-parameter, aside from a single ablation study shown in Figure 5. This introduces a new, sensitive parameter that a practitioner must optimize via expensive trial-and-error, which counteracts the claimed simplicity of the method and could easily negate its benefits in practice. \n- The paper claims the method has \"minimal computational overhead,\" but this is not thoroughly justified. Calculating sequence-level importance ratios for every sample in a batch adds computational steps that, while not requiring new forward passes, are not free. Considering its added complexity, it is not as simple implied by the paper.", "was_revised": true, "final_justification": "The theoretical connection between IS correction and implicit KL regularization (Theorem 1) is a good contribution I initially overlooked. So I have raised my score.", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Please find our response to your comments below.\n\n**Q1. The core proposed by the authors that offline algorithms suffer from distribution shift as the policy moves away from the data-generating policy, is a well-known problem in the field of offline reinforcement learning. The proposed solution, applying importance sampling, is a standard method to tackle this, which has already applied in PPO. As such, the paper's conceptual contribution feels incremental rather than novel, amounting to the application of a known technique to the specific context of DPO.**\n\n**Answer:** Thank you for your comment. We'd like to highlight that our contributions goes beyond the use of Importance Sampling (IS) to address the distribution shift, due to the following reasons:\n\n- **Foundational Theoretical and Empirical Analysis of IS in DPO**: Our paper provides a theoretical connection of overoptimization and distributional shift between the learned and preference policies in DPO (Section 3.1). Then we propose IS the correction for overoptimization in DPO. However, the most significant analysis of our work is theoreatically demonstrating that IS correction implicitly (and approximately) regularize the DPO's policy in a similar manner as that of minimizing the KL divergence with online samples (see Theorem 1, and Section 5.2's discussion, and also our empirical evidence for online DPO as a response to fRzo's Q2). \nIn summary, our paper provides rigorous theoretical analysis of the overoptimization problem and its solution (via IS) in the context of DAAs, which is the significant contribution beyond simply an application of IS. \n\n\n- **The roles and impact of Importance sampling and clipping are fundamentally different in PPO and IS-DPO**: While IS has also been used to address distribution shift in PPO, its usage is fundamentally different from our paper. PPO uses clipped importance sampling (IS-clip) to improve sample efficiency. Since each batch is typically discarded after a single gradient update in standard policy gradient methods, IS allows PPO to reuse the same batch multiple times, extracting the most information from each batch. In contrast, our objective is to enforce an effective regularization, to prevent the from pushing up the probability of responses that are out of the offline data distribution, mitigating the over-optimization issue. In addition, in PPO, the clipping mechanism acts as a surrogate trust region to prevent the policy from deviating too far from the previous one (which can help mitigate over-optimization), but in a fully offline setting (as in our paper), this mechanism becomes overly conservative-where gradient is zero when the probability ratio falls outside the clipping range $[1-\\epsilon, 1+\\epsilon]$, limiting the potential improvement from $\\pi_{\\text{ref}}$ [4,5]. Our clipping strategy, conversely, mitigates excessively large updates with IS while still providing meaningful updates even when the ratio lies outside the clipping region.\n\nIn summary, these contributions are significant and original, and we hope the response has addressed the reviewer's concerns. We thank the reviewer again for the thoughtful feedback.\n\n**Q2. The method's stability and performance rely entirely on the clipping ratio. However, the paper offers no principled way to set this crucial hyper-parameter, aside from a single ablation study shown in Figure 5. This introduces a new, sensitive parameter that a practitioner must optimize via expensive trial-and-error, which counteracts the claimed simplicity of the method and could easily negate its benefits in practice.**\n\n**Answers:** Thank you for the comment. IS-DPO indeed has a hyperparameter, much similar to prior works, with multiple hyperparameters, tackling over-optimization or distributional shift [1,2,3]. However, our ablation studies in Figure 5 demonstrate the robustness of IS-DPO's hyperparameter -- specifically that IS-DPO maintains strong performance across a reasonable range of clipping values robustness and in Section 4.1, we empirically suggest the use of $\\log \\epsilon = 1.0$ as the preferred choice in practice, which we use for all our experiments.\n\nIn addition, we will now provide a theoretical insight that better understands the relationship between the regularization coefficient $\\beta$ and the clipping parameter $\\epsilon$, which helps provide a principled tuning of $\\epsilon$.\n\n**Proposition:** *Assuming $\\text{Supp}(\\pi_\\theta)=\\text{Supp}(\\pi_{\\text{ref}})$, then the variance of the importance weight $\\mathbb{Var}\\left(\\frac{\\pi_\\theta}{\\pi_{\\text{ref}}}\\right)$ under $\\pi_{\\text{ref}}$ distribution is an upper bound of KL divergence:*\n$$\\text{KL}\\left(\\pi_\\theta || \\pi_{\\text{ref}}\\right) \\leq \\mathbb {Var}\\left(\\frac{\\pi_\\theta}{\\pi_{\\text{ref}}}\\right)$$\n\nThe proof is straightforward; we will include the detailed proof in our camera-ready version. This proposition highlights that when regularization is weak (i.e., small $\\beta$), the policy $\\pi_\\theta$ can deviate significantly from $\\pi_{\\text{ref}}$, leading to high variance in the importance weights. This motivates the use of a smaller clipping threshold $\\epsilon$ to control variance and stabilize training under such regimes.\n\nAdditionally, we also conduct the relationship between the regularization parameter $\\beta \\in\\\\{0.01,0.05,0.1,0.5\\\\}$ and the clipping ratio $\\log\\epsilon \\in\\\\{0.25, 0.5, 1.0\\\\}$. We present the win-rate and KL values below:\n\nWin-rate table:\n| $\\beta$ \\ $\\log\\epsilon$ | 0.25 | 0.5| 1.0|\n| -------- | -------- | -------- | -------- |\n| 0.01     |  91.2    |  92.0    | 89.7|\n| 0.05     |  92.2    |   93.2  |  94.5 |\n| 0.1    |   93.6 |  94.0  |  91.4  | 91.4   |\n|0.5     |  68.9  |   69.7 |   71.0 |    | \n\nKL table:\n| $\\beta$ \\ $\\log\\epsilon$ | 0.25 | 0.5| 1.0 |\n| -------- | -------- | -------- | -------- |\n| 0.01     |  15.4    |  20.1    | 22.5 |\n| 0.05     |  17.6    |   18.2 |  22.2  |\n| 0.1    |   15.9 |  18.8  | 22.3   | \n|0.5     |  2.55  |   2.78 |   3.06 |    | \n\n**$\\epsilon$ as an additional regularization parameter:** We observe that increasing the clipping threshold allows for larger policy updates, which in turn leads to higher KL divergence across all values of $\\beta$. Interestingly, at low regularization levels (small $\\beta$), using a smaller clipping ratio yields better performance, as it helps prevent over-optimization. In contrast, at higher regularization levels (e.g., $\\beta = 0.5$), the stronger regularization larger clipping ratios that are more effective.\n\n\n**Q3. The paper claims the method has \"minimal computational overhead,\" but this is not thoroughly justified. Calculating sequence-level importance ratios for every sample in a batch adds computational steps that, while not requiring new forward passes, are not free. Considering its added complexity, it is not as simple as implied by the paper.**\n\n**Answer**: Thank you for your comment. We'd like to emphasize that the computational overhead of calculating the importance ratios is negligible and results in almost the same time complexity as DPO. Despite adding the additional importance ratio calculation step to the original DPO loss, this ratio can be efficiently computed using intermediate quantities already available in the DPO loss calculation. Specifically, as presented in the paper, the importance ratio can be computed in the log-space as follows:\n\n$$w(\\mathbf y_w, \\mathbf y_l) = e^{\\log\\pi_\\theta (\\mathbf y_w)+\\log\\pi_\\theta(\\mathbf y_l)-\\log\\pi_{\\text{ref}}(\\mathbf y_w)-\\log\\pi_{\\text{ref}}(\\mathbf y_l)}$$ \n\nHence, IS-DPO only adds a small calculation, negligible compared to the much more heavier computations in the vanilla DPO. This implementation is also straightforward and simple, requiring only a single additional line of code.\n\nTo empirically validate this claim, we report the average training time over 100 steps for both DPO and IS-DPO. We use 4 A100-80GB GPUs, a batch size of 16, and LLaMA-3.2-3B as the base model:\n\n| IS-DPO | DPO |\n| -------- | -------- \n| 92.829s $\\pm$ 5.144    | 92.571s $\\pm$ 4.837\n\nAs can be observed, the training time for IS-DPO is nearly identical to that of the vanilla DPO, confirming that our approach is computationally efficient and negligible in overhead.\n\n----\n    1. Disentangling Length from Quality in Direct Preference Optimization. ACL24\n    2. Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer. NeurIPS24.\n    3. Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. ICLR25 Spotlight.\n    4. The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure. AAAI23.\n    5. Off-Policy Proximal Policy Optimization. AAAI23."}, {"review_id": "cq8yhS5njo", "reviewer": "Reviewer_B49u", "summary": "This paper addresses the problem of reward over-optimization in offline Direct Alignment Algorithms (DAAs) like DPO. The authors identify that DAA's implicit KL regularization is only effective when the learned policy is close to the initial reference policy, leading to performance degradation as the model drifts away during training. To mitigate this, the paper proposes Importance Sampling DAAs IS-DAAs, a novel approach that re-weights the standard DAA loss with an importance sampling ratio. This re-weighting aims to better approximate the online learning objective using only offline data, thus enforcing a more effective regularization throughout the training process. To handle the high variance typically associated with importance sampling, the authors propose clipping the importance weights. The paper provides theoretical analysis showing that their method provides an unbiased estimate of the online DPO objective and that its gradient correctly corresponds to the gradient of the KL divergence. Through experiments on TL;DR summarization and instruction-following benchmarks, the authors demonstrate that IS-DAA effectively mitigates over-optimization, particularly under low regularization, and achieves superior performance compared to standard DAA and other baseline methods designed to tackle the same problem.", "questions": "1. Can the authors offer a hypothesis or analysis for why X-PO might be more effective on AlpacaEval, while IS-DPO is stronger on Anthropic-HH? X-PO uses a chi-squared divergence which penalizes large deviations from the reference policy more heavily than the KL divergence implicit in DPO. Does IS-DPO, by more faithfully approximating an on-policy KL-regularized objective, perhaps inherit a specific behavior that is less suited to the data distribution or evaluation protocol of AlpacaEval?\n2. Could the authors provide more practical guidance on setting $\\epsilon$? A key question is its interaction with the main DAA regularization parameter $\\beta$. How should one co-tune $\\beta$ and $\\epsilon$? For instance, if one uses a very high $\\beta$ (forcing the policy to stay close to the reference), does the optimal $\\epsilon$ tend to be larger or smaller? Does the optimal $\\epsilon$ depend on properties of the offline dataset, such as its size or the diversity of the reference policy responses?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Significance: The paper tackles a well-known and highly relevant problem. Reward over-optimization and the performance gap between offline DAAs and online RLHF are significant barriers to creating more robust and reliable aligned models. The community is actively seeking simple and effective solutions, and this paper proposes a strong candidate.\n2. Clarity: The paper is well-written and easy to follow. The authors clearly motivate the problem, starting from the limitations of RLHF, moving to the promise and pitfalls of DAAs, and culminating in a crisp diagnosis of why their implicit regularization fails (Section 2.3). The progression from the standard DAA objective to the proposed IS-DAA objective is logical and well-explained.\n\nWeaknesses:\n\n1. Originality: The primary weakness is the novelty of the proposed technical components. As noted, importance sampling with clipping is proposed in PPO and serves as a cornerstone of modern RL. The paper re-applies this established technique in the context of DAAs. An expert in RL would immediately recognize the objective in Eq. (11) and the clipping in Eq. (13) as analogous to PPO's surrogate objective. The authors could be more explicit about this connection to prior work such as PPO. The contribution is therefore not in inventing a new technique, but in the application, diagnosis, and analysis of an existing one in a new domain.\n2. Quality: The paper introduces a clipping hyperparameter to control the bias-variance trade-off. However, the analysis is limited to a single ablation study (Figure 5). There is little discussion on the sensitivity to this hyperparameter or practical guidance on how to set it. Since clipping fundamentally changes the objective from an unbiased to a biased one, the paper could benefits from a deeper discussion of its impact and how it might interact with the learning rate or the DAA's own parameter.\n3. Quality: The method is motivated by better approximating the online DAA objective. A powerful, albeit computationally expensive, experiment would have been to compare IS-DAA (the offline approximation) directly against an online DPO baseline (which performs actual on-policy sampling). This would have provided a clear measure of how successfully the importance sampling bridges the gap between offline and online training.", "was_revised": true, "final_justification": "My major concern on the novelty of the proposed clipping importance-sampling technique is well addressed. Therefore, I raised my score accordingly.", "rebuttal": "We thank the reviewer for their helpful feedback and will respond to the raised questions below.\n\n**Q1. Discussion between the connection of clipped Importance sampling in IS-DPO and PPO.**\n\n**Answer:** Thank you for your comment. We'd like to emphasize that the contributions of our paper go beyond the mere application of Importance Sampling (IS) in DAAs. We aim to provide a **theoretical understanding of overoptimization**, an important problem, in DAAs; based on this understanding, we then propose **Importance Sampling (IS) as an approximation of the KL regularization** between the reference policy and the training policy under the offline setting. In addition, **the roles of IS in PPO (an online algorithm) and in our IS-DPO (an offline algorithm) are also fundamentally different**. Specifically:\n\n- **Foundational Theoretical and Empirical Analysis of IS in DPO**: Our paper provides a theoretical connection of overoptimization and distributional shift between the learned and preference policies in DPO (Section 3.1). Then we propose IS the correction for overoptimization in DPO. However, the most significant analysis of our work is theoreatically demonstrating that IS correction implicitly (and approximately) regularize the DPO's policy in a similar manner as that of minimizing the KL divergence with online samples (see Theorem 1, and Section 5.2's discussion, and also our empirical evidence for online DPO as a response to fRzo's Q2). \nIn summary, our paper provides a rigorous theoretical analysis of the overoptimization problem and its solution (via IS) in the context of DAAs, which is the significant contribution beyond simply an application of IS. \n\n- **The roles and impact of Importance sampling and clipping are fundamentally different in PPO and IS-DPO**: While IS has also been used to address distribution shift in PPO, its usage is fundamentally different from our paper. PPO uses clipped importance sampling (IS-clip) to improve sample efficiency. Since each batch is typically discarded after a single gradient update in standard policy gradient methods, IS allows PPO to reuse the same batch multiple times, extracting the most information from each batch. In contrast, our objective is to enforce an effective regularization, to prevent them from pushing up the probability of responses that are out of the offline data distribution, mitigating the over-optimization issue. In addition, in PPO, the clipping mechanism acts as a surrogate trust region to prevent the policy from deviating too far from the previous one (which can help mitigate over-optimization), but in a fully offline setting (as in our paper), this mechanism becomes overly conservative-where gradient is zero when the probability ratio falls outside the clipping range $[1-\\epsilon, 1+\\epsilon]$, limiting the potential improvement from $\\pi_{\\text{ref}}$ [4,5]. Our clipping strategy, conversely, mitigates excessively large updates with IS while still providing meaningful updates even when the ratio lies outside the clipping region.\n\nIn summary, these contributions are significant and original, and we hope the response has addressed the reviewer's concerns. We thank the reviewer again for the thoughtful feedback.\n\n**Q2: The sensitivity of clipping ratio $\\epsilon$ and practical guidance how to set it.**\n\n**Answer:**  Thank you for your question. We'd like to clarify that the ablation studies in Figure 5 already provide practical guidance to set $\\epsilon$. Specifically, the studies demonstrate the robustness of IS-DPO's hyperparameter -- specifically that IS-DPO maintains strong performance across a reasonable range of clipping values robustness and in Section 4.1, we empirically suggest the use of $\\log \\epsilon = 1.0$ as the preferred choice in practice, which we use for all our experiments.\n\nNevertheless, we follow the reviewer's suggestion and now also provide a more theoretical insight on the relationship between the regularization coefficient $\\beta$ (DPO's hyperparameter) and the clipping parameter $\\epsilon$. We hope this will provide additional guidance on selecting $\\epsilon$.\n\n**Proposition:** *Consider the two distribution $\\pi$ an $\\pi_{\\text{ref}}$. Assuming $\\text{Supp}(\\pi_\\theta)=\\text{Supp}(\\pi_{\\text{ref}})$, then the variance of the importance weight $\\mathbb{Var}\\left(\\frac{\\pi_\\theta}{\\pi_{\\text{ref}}}\\right)$ under $\\pi_{\\text{ref}}$ distribution is an upper bound of KL divergence:*\n$$\\text{KL}\\left(\\pi_\\theta || \\pi_{\\text{ref}}\\right) \\leq \\mathbb {Var}\\left(\\frac{\\pi_\\theta}{\\pi_{\\text{ref}}}\\right)$$\n\nThe proof is straightforward by utilizing the inequality $\\log x\\leq x-1,\\forall x>0$. We will include the detailed proof in our camera-ready version. This proposition highlights that when regularization is weak (i.e., small $\\beta$), the policy $\\pi_\\theta$ can deviate significantly from $\\pi_{\\text{ref}}$, leading to high variance in the importance weights. This suggests a smaller clipping threshold $\\epsilon$ to control variance and stabilize training under such regimes and vice versa when regularization is strong.\n\nWe also conduct the relationship between the regularization parameter $\\beta \\in\\\\{0.01,0.05,0.1,0.5\\\\}$ and the clipping ratio $\\log\\epsilon \\in\\\\{0.25, 0.5, 1.0\\\\}$. We present the win-rate and KL values below:\n\n**Win-rates**\n| $\\beta$ \\ $\\log\\epsilon$ | 0.25 | 0.5|1.0|\n| -------- | -------- | -------- | -------- |\n| 0.01     |  91.2    |  92.0    | 89.7|\n| 0.05     |  92.2    |   93.2  |  94.5 |\n| 0.1    |   93.6 |  94.0  |  96.67  |   |\n|0.5     |  68.9  |   69.7 |   71.0 |    | \n\n**KL values**\n| $\\beta$ \\ $\\log\\epsilon$ | 0.25 | 0.5|1.0|\n| -------- | -------- | -------- | -------- |\n| 0.01     |  15.4    |  20.1    | 22.5 |\n| 0.05     |  17.6    |   18.2 |  22.2  |\n| 0.1    |   15.9 |  18.8  | 22.3   | \n|0.5     |  2.55  |   2.78 |   3.06 |    | \n\n**$\\epsilon$ as an additional regularization parameter:** We observe that increasing the clipping threshold allows for larger policy updates, which in turn leads to higher KL divergence across all values of $\\beta$. Interestingly, at low regularization levels ($\\beta=0.01$), using a smaller clipping ratio yields better performance, as it helps prevents over-optimization. In contrast, at higher regularization levels (e.g., $\\beta \\in\\\\{0.05, 0.1, 0.5\\\\}$), larger clipping ratios are more effective. Overall, performance remains robust across a wide range of $\\log \\epsilon$ values. Consistent with our ablation studies, we recommend setting $\\log \\epsilon = 1$ as a default choice in practice.\n\n\n**Q3. Comparison with Online-DPO**\n\n**Answer:** Thank you for a great suggestion. As suggested, we conduct experiments to understand how well the IS-DPO can approximate the Online-DPO. We present our findings below:\n\nDPO result:\n|  |  | | | | | |\n| - | - | - | - | - | - | -\n| Win-rate     |   68.8   |    86.9  | 91.4 |86.6| 75.0 |57.6 |\n| $\\sqrt{\\text{KL}}$ |1.70 | 3.10 | 4.51 | 5.91 | 7.32 | 8.72 |\n\nIS-DPO result:\n|  |  | | | | | |\n| -------- | -------- | -------- | - | - | -| -|\n| Win-rate     |  57.15    |   73.6   | 84.0| 90.0 | 92.8 | 92.78| \n| $\\sqrt{\\text{KL}}$ |1.46 | 2.37| 3.28| 4.19| 5.10| 6.02 | \n\n\nOnline-DPO result:\n\n|  |  | | | | | |\n| -------- | -------- | -------- | -| - | -|- |\n| Win-rate     |  56.11   |  77.06    | 88.7 | 94.07 | 94.7 | 93.6\n| $\\sqrt{\\text{KL}}$ |1.726 | 3.17| 4.63|6.08 | 7.53 | 8.25\n\n**IS-DPO can closely approximate Online-DPO up to a specific KL divergence threshold:** We observe that IS-DPO also follows a similar performance trajectory to Online-DPO in the low to moderate KL divergence regime $\\sqrt{\\text{KL}}\\left(\\pi_\\theta ||\\pi_{\\text{ref}}\\right)\\approx 6$. In this region, both methods show a similar and steady increase in win rate. This suggests that IS-DPO is capable of closely approximating the behavior and performance of Online-DPO within a certain KL \"budget\".\n\n**On-policy benefits from large KL \"budget\"**: While IS-DPO does not show signs of over-optimization, it also does not extrapolate to higher KL regimes, while Online-DPO can find better policies under higher KL regimes. This indicates that on-policy optimization remains beneficial when the optimal policies lie farther from the $\\pi_{\\text{ref}}$, as Online-DPO is able to leverage a larger KL budget.\n\n**Comparison with DPO:** We also provide a comparison between Online-DPO and DPO. Interestingly, we observe that under low KL budget, DPO appears to outperform Online-DPO $\\sqrt{\\text{KL}}(\\pi_\\theta ||\\pi_{\\text{ref}}) \\approx 4.5$. However, it sharply declines as KL increases, indicating over-optimization, while IS-DPO and DPO effectively utilize a larger KL budget to achieve higher performance.\n\n**Q4: Hypothesis on why $\\chi$-PO more effect on AlpacaEval.**\n\n**Answer:** Thank you for a great observation. Despite the consistently strong performance of IS-DPO, $\\chi$-PO indeed achieves a slightly (though marginally) better result in AlpacaEval. While we do not have a definitive explanation, we suspect IS-DPO's hyperparameter selection (i.e., $\\epsilon$) may not be optimal for this dataset. A deeper investigation, e.g., why $\\chi$-PO performs well in AlpacaEval, could be valuable, but this requires extensively more experiments and analysis on many benchmarks to confirm this observation. We believe this deserves an independent study beyond the scope of our paper.\n\n----\n    1. Disentangling Length from Quality in Direct Preference Optimization. ACL24\n    2. Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer. NeurIPS24.\n    3. Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. ICLR25 Spotlight.\n    4. The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure. AAAI23.\n    5. Off-Policy Proximal Policy Optimization. AAAI23."}, {"review_id": "Xa5S4nj0Ez", "reviewer": "Reviewer_PaV2", "summary": "This paper highlights the problem of reward over-optimization in Direct Alignment Algorithms (DAAs) such as DPO and IPO, which occurs when models diverge excessively from the reference policy during alignment, leading to suboptimal performance. To mitigate this, the authors propose an adaptive low-variance importance sampling (IS) strategy for offline post-training alignment of large language models (LLMs). Empirical evaluations demonstrate the benefit of the above strategy via mitigating reward over-optimization.", "questions": "Please see weakness section", "limitations": "Please see weakness section", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper uses importance sampling (IS) and intelligent modifications to approximate expectations under the learned policy and mitigate distribution shift. While IS is well-known in the literature, its application here is intelligent and appropriate. It also shows that the regularization can be connected to the KL penalty in RLHF through Theorem 1, which is crucial—though it may require further justification. The work establishes a strong link between loss curvature and KL regularization in RLHF. Empirical analysis shows it outperforms baselines such as RPO, length regularization, and others.\n\nAlthough the paper proposes an interesting approach, certain claims need further clarity. A key issue is regarding the KL regularization to the reference policy, which is a crucial component in the alignment objective. It is not extremely clear in the scenario where there is clipped importance weight being used, how the KL is still ensured mathematically w.r.t the original alignment objective? Theorem 1 provides the details, but an explicit description showing how the importance weight affects the KL balls is crucial? Empirical analysis is also crucial\nSecond, does the unbiasedness of the gradient estimate still hold true with the importance weight and smoothening? It will be helpful to highlight the same. In the importance weight estimation, there can be trajectories generated whose probability is low under the reference policy making the weights unbounded, how the smoothing helps in this regard? Additionally, the Taylor approximation is local and mainly valid around 0, i.e when the policies are close to reference. However, how this helps in scenarios when they are apart which is the main reason of the dist shift issue leading to overoptimization, then the approximation might not hold. Can you provide a detailed discussion regarding the same? Discussion with works that rely on the principles of pessimism needs to be discussed [1,2]. \n\nReferences\n[1]. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf\n[2]. Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer", "was_revised": true, "final_justification": "I keep my scores towards acceptance. There were revisions, but after incorporating them, i believe it can be accepted", "rebuttal": "Thank you for your great comments and efforts in reviewing our paper. Please find our response to your comments below.\n\n**Q1. A key issue is regarding the KL regularization to the reference policy, which is a crucial component in the alignment objective. It is not extremely clear in the scenario where there is clipped importance weight being used, how the KL is still ensured mathematically w.r.t the original alignment objective? Theorem 1 provides the details, but an explicit description showing how the importance weight affects the KL balls is crucial? Empirical analysis is also crucial**\n\n**Answer:** Thank you for your question. We acknowledge that Theorem 1 only provides the details for importance weight. Below, we provide an explanation to clarify how the clipped importance weight affects alignment with the KL regularization. We consider the $\\mu$-squared loss with importance sampling:\n$$\\nabla_\\theta\\mu_{\\text{IS}}(y_1,y_2)=w\\nabla_\\theta\\rho, \\quad \\nabla_\\theta\\mu_{\\text{clip}}(y_1,y_2)=\\min(w, c)\\nabla_\\theta \\rho$$\nwhere $w=\\frac{\\pi_\\theta(y_1,y_2)}{\\pi_{\\text{ref}}(y_1,y_2)},\\rho=\\log\\frac{\\pi_\\theta(y_1)}{\\pi_{\\text{ref}(y_1)}}-\\log\\frac{\\pi_\\theta(y_2}{\\pi_{\\text{ref}(y_2)}}$. As shows in Theorem 1, we can see that $\\mathbb E_{\\pi_{\\text{ref}}}[\\nabla_\\theta\\mu_{\\text{IS}}]=\\nabla_\\theta\\text{KL}(\\pi_\\theta||\\pi_{\\text{ref}})$.\n\n**Clipped IS-DPO ensures alignment with KL regularization**: Notice that the gradient of the clipped objective can be expressed as\n\n$$\\\\nabla\\_{\\\\theta}\\mu\\_{\\text{clip}}=\\nabla\\_\\theta\\mu\\_{\\text{IS}}-\\mathrm{1}\\_{\\{w > c\\}}(w-c)\\nabla\\_\\theta\\rho$$\n\nThis decomposition shows that clipping introduces a bias by down-weighting the influence of samples with large importance weights. However, the clipped gradient still maintains alignment with the importance-weighted gradient, as indicated by the dot product.\n$$\\langle\\nabla_\\theta\\mu_{\\text{clip}}, \\nabla_\\theta\\mu_{\\text{IS}}\\rangle \\geq 0$$\n\nIn effect, clipping acts as a form of regularization that mitigates gradient explosion from high-weight outliers while preserving alignment with the KL regularization.\n\n**Q2. Does the unbiasedness of the gradient estimate still hold true with the importance weight and smoothening? It will be helpful to highlight the same. In the importance weight estimation, there can be trajectories generated whose probability is low under the reference policy making the weights unbounded, how the smoothing helps in this regard?**\n\n**Answer:** We really appreciate this question. We sincerely apologize for the typo in Equation (13) — the operator should indeed be $\\min$ instead of $\\max$. \n\nWhile the clipping operator introduces into the estimator, its purpose is to keep the gradient variance under control. This clipping mechanism is especially important in cases where certain trajectories have very low probability under $\\pi_{\\text{ref}}$. The clipping mechanism will be applied the keep the importance weight bounded to stabilize training.\n\n**Q3. Additionally, the Taylor approximation is local and mainly valid around 0, i.e, when the policies are close to reference. However, how this helps in scenarios when they are apart which is the main reason of the dist shift issue leading to overoptimization, then the approximation might not hold. Can you provide a detailed discussion regarding the same?**\n\n**Answer:** Thank you for your question. You are correct that the Taylor approximation used in Offline DAAs only enforces KL regularization when the policies $\\pi_\\theta$ are close to $\\pi_{\\text{ref}}$. This is the exact objective our our analysis in this section: we aim to show that existing DAAs will only have effective regularization locally (via this Taylor expansion) and become susceptible to overoptimization as the learned policy diverges further.\n\nMore specifically, this approximation shows that DAAs are not well regularized when the learned policy $\\pi_\\theta$ moves away from $\\pi_{\\text{ref}}$. Optimizing Offline DAAs can lead to over-optimization or extrapolation into out-of-support regions of $\\pi_{\\text{ref}}$ $\\left(\\text{KL}(\\pi_\\theta||\\pi_{\\text{ref}}=\\infty\\right)$ [3,4,5,6], and the local regularization used in Offline DAAs is insufficient to prevent these issues compared to standard RLHF.\n\n**Q4. Discussion with works that rely on the principles of pessimism needs to be discussed [1,2].**\n\n**Answer** Thank you for the insightful suggestion. We will include the following discussion, citing the works on principles of pessimism, in the camera-ready version:\n\nAt a high level, the *principle of pessimism* explicitly subtracts uncertainty-based penalties such as KL divergence or $\\chi^2$ divergence from the estimated value with the aim of preventing overestimation of the value of candidate policies in regions with low data coverage or high uncertainty.\n\nIS-DPO(IS) also incorporates a form of pessimism. IS-DPO assigns lower weights to trajectories that are less likely under the learned policy $\\pi_\\theta$, effectively penalizing policies that place high probability mass in areas poorly covered by the data. This mechanism aligns conceptually with pessimism-based regularization [1,2].\n\nMoreover, there is a deeper theoretical connection worth exploring: the variance of the IS estimator is proportional to the chi-squared divergence between $\\pi_\\theta$ and $\\pi_{\\text{ref}}$. This observation suggests a promising direction to formalize the connection between importance sampling and pessimism. $\\chi^2$ divergence has been used in LLM alignment to impose stronger regularization, thereby effectively mitigating over-optimization issues. We believe that bridging this connection more formally, possibly through uncertainty-aware confidence intervals, is a promising avenue for future work.\n\n----\n    [1] Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF. ICML24.\n    [2] Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer. NeurIPS24.\n    [3] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML24 Oral.\n    [4] A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS24.\n    [5] The Importance of Online Data: Understanding Preference Fine-tuning via Coverage. NeurIPS24.\n    [6] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. NeurIPS24.\n    [7] Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. ICLR25 Spotlight."}, {"review_id": "qaIVeb8BmZ", "reviewer": "Reviewer_VYu7", "summary": "This paper proposes added importance weights to the DPO objective to mitigate over-optimization. This is simply done by multiplying the loss of each example by the density ratio of each sample under the current vs reference policy.", "questions": "1. Why does IS-DPO seem to be invariant to $\\beta$? It appears from Figure 4 that no matter the $\\beta$, IS-DPO seems to converge to a sqrt(KL) of 5, while other methods seem to change. Was there possibly a bug here? \n2. Why do the authors choose 2 epochs for training? Is one epoch not more standard for DPO? Would this not increase the overfitting effect in Fig 3?\n3. What is the effect of SFT on IS-DPO, given that without it, the importance weights are incorrect?\n4. What values of $\\beta$ were used for figure 3?", "limitations": "No limitations section is provided -- it would be good to include one!", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n* The method is simple to implement\n* The approach appears to improve performance with minimal changes.\n* The approach is well motivated and explained. I particularly liked the examples in Fig 1 and Sec. 3.2.\n* The paper is sufficiently self-contained -- one can get the required background from reading the work.\n* the chosen baselines appear sufficient to me and the paper focuses on direct alignment algorithms. The paper could be made stronger by considering other baselines outside of DAAs (e.g. GRPO), but I concede that it seems a bit orthogonal at present. \n\n\n## Weaknesses\n\n**Theory and Presentation**\n\n\n* Some of the theory presented is a bit confusing. Why should we consider the taylor expansion of the gradient as in Tang et al. when we could just directly consider the gradient? In the gradient itself (I believe) we do not get this KL term from eq. 8. Moreover, the KL term is only valid at the start of training, and progressively goes away with more updates. To summarize, I am not certain if I understand why the argument from Tang et al. logically holds here. \n\n* Missing assumptions: DAAs do not necessarily require that the policy which produced D is the same as the reference policy $\\pi_\\text{ref}$ -- they can operate over any support. However it has been implicitly assumed by the authors via the importance weights that the sampled responses do come from the reference policy. It seems necessary to highlight this, and point out that it means that SFT on the preference dataset is likely required for IS-DPO to work. \n\n* Theorem 1 is kinda just saying that importance sampling is unbiased.... which is not necessarily a new result and I am unsure if it deserves to be a theorem. It is also strange to state \"the gradient concerning $\\pi_\\theta$ of the square loss is equal to that of the KL when this is again another straightforward application of importance sampling and only holds when $\\rho_\\theta = 0$ at the start of training. That second component should be mentioned again. \n\n* In section 4.3 it is asserted that when supp(P) is a subset of supp(Q), the Q term completely disappears? How do the authors go from P(x)/Q(x) --> P(x)? Would this not instead just be undefined? I thought importance sampling would require the support constraint. This argument was also a bit confusing because a fixed dataset (assumed to be sampled from piref) is used, and if $\\pi^* \\propto  \\pi_\\text{ref} e^r$, then the support of $pi^*$ will be the same as that of pi ref unless the reward function is unbounded.\n\n**Experiments**\n* the experimental results are a bit limited in comparison to several other modern LLM papers, however I understand computational constraints are a factor. HH and TLDR are rather simple datasets at this point. \n* the gains, though positive, are a bit marginal in many cases.\n* The effect on KL evolution appears to be inconsistent (see question).\n* While IS-DPO perhaps exhibits less over-optimization in Fig 3, its performance at different KL budgets appears to be worse? \n\nMinor:\n* missing reference on line 39\n* should note that $\\sigma$ is being used as the logistic function.\n* Line 123 \"going in the opposite direction of the second term\" is ambiguous\n* Line 254 missing reference\n* Line 243 missing reference\n* Line 223: previous approaches *are* \nThe authors should spend some time reading the paper to catch these issues.\n\n## Overall\nOverall, I recommend a borderline accept. The paper has a reasonable idea -- that we can use importance weights to bring offline DAAs closer to online methods. However, it is hard to fully recommend the paper because:\n1. The assumptions of the method and theory are not explicitly stated, and parts of the text around this are confusing.\n2. The experimental results are a little inconsistent (KL evolutions and performance in Fig 3 at fixed budgets)\n\nShould the authors be able to better explain the theory results and inconsistencies in experiments I am willing to raise my score.", "was_revised": true, "final_justification": "My question about the KL-budget experiment seems answered. The authors also clarified a number of points from the draft. I left a comment listing some improvements that I think should be made, namely that Sec. 4.3 should be explained a bit more (tying into the KL budget experiments) and that the SFT assumption should be further clarified.\n\nGenerally, I recommend accepting the paper as its results seem relatively good given a small overhead, and is justified adequately.", "rebuttal": "Thank you for your review and positive feedback on our paper. We appreciate your acknowledgment of the clarity and organization of our work. Please find our response to your comments below.\n\n**Q1. Why should we consider the Taylor expansion? And the KL term is only valid at the start of training**\n\n**Answer:** Thanks for the great question. You are correct that directly computing the gradient does not yield the KL term as shown in Equation 8. In Section 2, our goal is to highlight that the KL regularization term is only meaningful at the beginning of training when using DAAs. Without importance sampling or on-policy sampling, this KL term in DAAs fails to accurately reflect the KL regularization used in RLHF as training progresses with more updates. Hence, by multiplying with the importance ratio, we can enforce more effective regularization and better reflect the KL regularization in RLHF without requiring on-policy samples.\n\n**Q2. Missing assumptions: DAAs do not necessarily require $\\pi_{\\text{ref}}$ to produce $\\mathcal D$ and the necessity of SFT for IS-DPO.**\n\n**Answer**: Thank you for pointing this out. We first apologize for the implicit assumption in our formulation that the policy produced $\\mathcal D$ is the same as $\\pi_{\\text{ref}}$. We will make it more explicit in the final version.\n\nThank you for a great comment on the necessity of (SFT) on the preference dataset for IS-DPO to be effective. We'd like to clarify that this requirement is not unique to IS-DPO but is also essential for DPO and other DAAs [11,12].\n\nNevertheless, we agree that the suggestion of the reviewer can be interesting, thus we also provide experiments exploring how both DPO and IS-DPO perform when trained on preference data not aligned with $\\pi_{\\text{ref}}$. Specifically, we construct a new dataset $\\mathcal{D}\\_{\\text{shift}}$ by sampling responses from final Online-DPO policies, which achieve a high win rate of $\\approx 97\\%$ but exhibit a significant shift from $\\pi_{\\text{ref}}$ $\\left(\\sqrt{\\text{KL}(\\pi_\\theta || \\pi_{\\text{ref}})} \\approx 7.2\\right)$. These responses are then labeled using the golden reward to generate 100k preference samples. We train both IS-DPO and DPO on $\\mathcal{D}_{\\text{shift}}$ with $\\beta=0.05$ and $\\log\\epsilon=0.5$. We present our results below:\n\n| Method | Win-rate | $\\sqrt{\\text{KL}}$ |\n| -------- | -------- | -------- |\n| DPO     | 74.6     | 7.2     |\n| IS-DPO | 77.3 | 3.86\n\nBoth methods achieve lower win rates compared to training on the original dataset, reaffirming the importance of SFT as a crucial warm-up step. Interestingly, IS-DPO achieves a slightly higher win rate than DPO while maintaining significantly lower KL divergence from the $\\pi_{\\text{ref}}$. This suggests that IS-DPO may exhibit more robust to distribution shifts even when preference data is not sampled from $\\pi_{\\text{ref}}$.\n\n**Q3. Theorem 1 just says that importance sampling is an unbiased estimator, which is not a new result, and strange statement.**\n\n**Answer:** Thank you for your suggestions. While the unbiased nature of IS is not new, this theorem, and our paper in general, formalize the impact of distribution shift in DAAs and provide a theoretical analysis (see Section 3.1) to justify the IS correction under DPO objective. Specifically, we show that minimizing the implicit regularization in DPO when applied with importance sampling can also minimize the KL divergence. This connection not only grounds our method in addressing distribution shift but also provides a crucial link between our proposed regularization and KL regularization in RLHF via Theorem 1, as acknowledged by reviewer PaV2. Nevertheless, we will consider the reviewer's suggestion and revise the theorem accordingly to reflect what's known and what's our focus (as in this discussion). \n\n**Q4. The support constraint in Importance Sampling (Sec 4.3)**\n\n**Answer:** Thank you for your question. To further elaborate, we consider the support of a discrete distribution $P$ over sample space $\\mathcal X$ as follows: $\\text{supp}(P)=\\{x\\in\\mathcal X|P(x) > 0\\}$. The IS estimator is defined as $\\mathbb E_{x\\sim Q}\\left[\\frac{P(x)}{Q(x)}f(x)\\right]=\\sum\\limits_{x\\in\\text{supp}(Q)}Q(x)\\frac{P(x)}{Q(x)}f(x)=\\sum\\limits_{x\\in\\text{supp}(Q)}P(x) f(x)$. When we assume $\\text{supp}(P) \\subseteq \\text{supp}(Q)$, we can see that the IS estimator is unbiased, $\\sum\\limits_{x\\in\\text{supp}(Q)}P(x) f(x) =\\sum_{x\\in\\text{supp}(P)}P(x) f(x)=\\mathbb{E}_{x\\sim P}\\left[f(x)\\right]$.\n\nWhile it's true that the optimal policy under the DPO objective satisfies $\\pi^* \\propto \\pi_{\\text{ref}} e^{r}$, and thus shares the same support as $\\pi_{\\text{ref}}$. However, prior works [1,2,3,4] have shown that DAAs assume that $\\pi_{\\text{ref}}$ has full support over the entire prompt-response space to achieve $\\pi^*$. In practice, this assumption rarely holds, as preference datasets only cover a small fraction of the prompt-response space. Consequently, multiple distinct policies can achieve the same global optimum of the DPO objective [2,3,4], including policies that assign probability mass to responses outside the support of $\\pi_{\\text{ref}}$ [1, 3]. This contrasts to the original RLHF framework, which prevents generating responses that are of support $\\pi_{\\text{ref}}$ via explicit KL regularization. Thus, our proposed importance sampling method further enforces the support constraint, helping to avoid this failure mode.\n\n**Q5. Limited experiments with HH and TLDR are simple datasets.**\n\n**Answer:** Thank you for your feedback. We employ HH and TLDR, as they are widely adopted standard benchmarks in the alignment literature [4,5,6,7] for evaluation of DPO and its variants. Consequently, our work should be evaluated on these benchmarks for a fair evaluation with the existing baselines; evaluating on additional datasets should indeed incur significant computational constraints. \n\nNevertheless, please note that we have also conducted additional experiments on the UltraFeedback dataset in the paper, where we evaluate IS-DPO using popular benchmarks such as AlpacaEval 2.0 and MT-Bench, designed to measure alignment with human preferences in more practical settings. These results (Section 4.2) further support the effectiveness of IS-DPO beyond HH and TLDR. We hope this extended evaluation addresses your concerns regarding the scope of our experimental validation.\n\n**Q6. The invariance of IS-DPO to $\\beta$.**\n\n**Answer:** Thank you for your great comment. We confirm that there is no bug in Fig. 4 of IS-DPO. We observed that $\\text{KL}(\\pi_\\theta,|, \\pi_{\\text{ref}})$ during training typically remains within the range of 22.87 to 27.06. As discussed in Section 4.3, this behavior arises because the importance ratio in IS-DPO gradually decreases throughout training. As the importance weights diminish, resulting in an implicit early stopping effect.\n\nThis mechanism provides two key benefits. First, a low importance ratio implies that the current policy $\\pi_\\theta$ assigns very low likelihood to certain responses, and further updating on such samples can lead to over-optimization [5, 9]. IS-DPO naturally avoids this issue by down-weighting such updates. Second, a low importance weight may signal that on-policy sampling is needed, which is especially beneficial in iterative learning settings [10].\n\n**Q7.  Why 2 epochs for training instead of 1 epoch?**\n\n**Answer:** Thank you for your question. Prior work [9] has shown that DPO can suffer from over-optimization even before completing a single epoch, with further training resulting in performance degradation. Motivated by this, our work focuses on settings where over-optimization is a particular concern. In the Controlled Summarization task, we deliberately train for 2 epochs to highlight the robustness of IS-DPO under prolonged training and weak regularization (i.e., low $\\beta$); this setting is also consistent with prior studies [5, 6]. To ensure fair comparisons with standard setups, we still follow the standard 1-epoch configuration used in Instruction Following tasks [4, 7, 8] to demonstrate IS-DPO effectiveness in aligning with human preferences.\n\n**Q8. The values of $\\beta$ in Fig. 3? Why IS-DPO exhibit worse performance at different KL budgets** \n**Answer:** Thank you for your questions. We consider a range of $\\beta\\in\\{0.01, 0.05, 0.1\\}$ to consider different KL values in Fig 3. Viewing KL as a resource to be spent, we observe that while DPO exhibits better performance under a low KL budget, IS-DPO tends to consume more of the effective KL budget, ultimately achieving a higher peak win rate. Interestingly, we found that this phenomenon also exhibits in Online-DPO and previous work [8,13].\n\n----\n    [1] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML24 Oral.\n    [2] A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS24.\n    [3] The Importance of Online Data: Understanding Preference Fine-tuning via Coverage. NeurIPS24.\n    [4] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. NeurIPS24.\n    [5] Disentangling Length from Quality in Direct Preference Optimization. ACL24.\n    [6] Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. ICLR24 Spotlight.\n    [7] REBEL: Reinforcement learning via regressing relative rewards. NeurIPS24.\n    [8] Direct Language Model Alignment from Online AI Feedback. CORR24\n    [9] Learning Dynamics of LLM Finetuning. ICLR25 Oral.\n    [10] Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences. CoRR24.\n    [11] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS23 Oral. \n    [12] ORPO: Monolithic Preference Optimization without Reference Model. ACL24.\n    [13] Understanding the performance gap between online and offline alignment algorithms. CoRR24"}]

Author rebuttal:

Meta-review: This paper considers the problem of reward over-optimization in offline direct alignment algorithms like DPO. The main idea behind their approach is to take standard DAAs like DPO and multiple the loss by a clipped importance weight to address distribution shift between the trained policy \pi_theta and \pi_ref. The authors provide promising empirical results for this approach on standard RLHF datasets, and show via some simple calculations that the method can be interpreted as implicitly enforcing KL regularization.

Reviewers initially were mixed, with many additional questions around experiments and derivation. However, after the discussion period, they were uniformly fairly positive and recommended acceptance. I strongly encourage the authors to include all of the additional derivations and experiments that came up in discussion.
Decision: Accept (poster)