Title: Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: direct preference optimization

human preference alignment

Regularization
Abstract: Recently, Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO) have emerged as alternatives to the standard Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs) with human values. 
Surprisingly, while DAAs do not use a separate proxy reward model as in RLHF, their performance can still deteriorate over the course of training -- an over-optimization phenomenon found in RLHF where the learning policy exploits the overfitting to inaccuracies of the reward model to achieve high rewards.
One attributed source of over-optimization in DAAs is the under-constrained nature of their offline optimization, which can gradually shift probability mass toward non-preferred responses not presented in the preference dataset. This paper proposes a novel importance-sampling approach to mitigate the distribution shift problem of offline DAAs.
This approach, called (IS-DAAs), multiplies the DAA objective with an importance ratio that accounts for the reference policy distribution. IS-DAAs additionally avoid the high variance issue associated with importance sampling by clipping the importance ratio to a maximum value. Our extensive experiments demonstrate that IS-DAAs can effectively mitigate over-optimization, especially under low regularization strength, and achieve better performance than other methods designed to address this problem.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes added importance weights to the DPO objective to mitigate over-optimization. This is simply done by multiplying the loss of each example by the density ratio of each sample under the current vs reference policy.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* The method is simple to implement\n* The approach appears to improve performance with minimal changes.\n* The approach is well motivated and explained. I particularly liked the examples in Fig 1 and Sec. 3.2.\n* The paper is sufficiently self-contained -- one can get the required background from reading the work.\n* the chosen baselines appear sufficient to me and the paper focuses on direct alignment algorithms. The paper could be made stronger by considering other baselines outside of DAAs (e.g. GRPO), but I concede that it seems a bit orthogonal at present. \n\n\n## Weaknesses\n\n**Theory and Presentation**\n\n\n* Some of the theory presented is a bit confusing. Why should we consider the taylor expansion of the gradient as in Tang et al. when we could just directly consider the gradient? In the gradient itself (I believe) we do not get this KL term from eq. 8. Moreover, the KL term is only valid at the start of training, and progressively goes away with more updates. To summarize, I am not certain if I understand why the argument from Tang et al. logically holds here. \n\n* Missing assumptions: DAAs do not necessarily require that the policy which produced D is the same as the reference policy $\\pi_\\text{ref}$ -- they can operate over any support. However it has been implicitly assumed by the authors via the importance weights that the sampled responses do come from the reference policy. It seems necessary to highlight this, and point out that it means that SFT on the preference dataset is likely required for IS-DPO to work. \n\n* Theorem 1 is kinda just saying that importance sampling is unbiased.... which is not necessarily a new result and I am unsure if it deserves to be a theorem. It is also strange to state \"the gradient concerning $\\pi_\\theta$ of the square loss is equal to that of the KL when this is again another straightforward application of importance sampling and only holds when $\\rho_\\theta = 0$ at the start of training. That second component should be mentioned again. \n\n* In section 4.3 it is asserted that when supp(P) is a subset of supp(Q), the Q term completely disappears? How do the authors go from P(x)/Q(x) --> P(x)? Would this not instead just be undefined? I thought importance sampling would require the support constraint. This argument was also a bit confusing because a fixed dataset (assumed to be sampled from piref) is used, and if $\\pi^* \\propto  \\pi_\\text{ref} e^r$, then the support of $pi^*$ will be the same as that of pi ref unless the reward function is unbounded.\n\n**Experiments**\n* the experimental results are a bit limited in comparison to several other modern LLM papers, however I understand computational constraints are a factor. HH and TLDR are rather simple datasets at this point. \n* the gains, though positive, are a bit marginal in many cases.\n* The effect on KL evolution appears to be inconsistent (see question).\n* While IS-DPO perhaps exhibits less over-optimization in Fig 3, its performance at different KL budgets appears to be worse? \n\nMinor:\n* missing reference on line 39\n* should note that $\\sigma$ is being used as the logistic function.\n* Line 123 \"going in the opposite direction of the second term\" is ambiguous\n* Line 254 missing reference\n* Line 243 missing reference\n* Line 223: previous approaches *are* \nThe authors should spend some time reading the paper to catch these issues.\n\n## Overall\nOverall, I recommend a borderline accept. The paper has a reasonable idea -- that we can use importance weights to bring offline DAAs closer to online methods. However, it is hard to fully recommend the paper because:\n1. The assumptions of the method and theory are not explicitly stated, and parts of the text around this are confusing.\n2. The experimental results are a little inconsistent (KL evolutions and performance in Fig 3 at fixed budgets)\n\nShould the authors be able to better explain the theory results and inconsistencies in experiments I am willing to raise my score.\n\nQUESTIONS:\n1. Why does IS-DPO seem to be invariant to $\\beta$? It appears from Figure 4 that no matter the $\\beta$, IS-DPO seems to converge to a sqrt(KL) of 5, while other methods seem to change. Was there possibly a bug here? \n2. Why do the authors choose 2 epochs for training? Is one epoch not more standard for DPO? Would this not increase the overfitting effect in Fig 3?\n3. What is the effect of SFT on IS-DPO, given that without it, the importance weights are incorrect?\n4. What values of $\\beta$ were used for figure 3?\n\nLIMITATIONS:\nNo limitations section is provided -- it would be good to include one!", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper highlights the problem of reward over-optimization in Direct Alignment Algorithms (DAAs) such as DPO and IPO, which occurs when models diverge excessively from the reference policy during alignment, leading to suboptimal performance. To mitigate this, the authors propose an adaptive low-variance importance sampling (IS) strategy for offline post-training alignment of large language models (LLMs). Empirical evaluations demonstrate the benefit of the above strategy via mitigating reward over-optimization.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper uses importance sampling (IS) and intelligent modifications to approximate expectations under the learned policy and mitigate distribution shift. While IS is well-known in the literature, its application here is intelligent and appropriate. It also shows that the regularization can be connected to the KL penalty in RLHF through Theorem 1, which is crucial—though it may require further justification. The work establishes a strong link between loss curvature and KL regularization in RLHF. Empirical analysis shows it outperforms baselines such as RPO, length regularization, and others.\n\nAlthough the paper proposes an interesting approach, certain claims need further clarity. A key issue is regarding the KL regularization to the reference policy, which is a crucial component in the alignment objective. It is not extremely clear in the scenario where there is clipped importance weight being used, how the KL is still ensured mathematically w.r.t the original alignment objective? Theorem 1 provides the details, but an explicit description showing how the importance weight affects the KL balls is crucial? Empirical analysis is also crucial\nSecond, does the unbiasedness of the gradient estimate still hold true with the importance weight and smoothening? It will be helpful to highlight the same. In the importance weight estimation, there can be trajectories generated whose probability is low under the reference policy making the weights unbounded, how the smoothing helps in this regard? Additionally, the Taylor approximation is local and mainly valid around 0, i.e when the policies are close to reference. However, how this helps in scenarios when they are apart which is the main reason of the dist shift issue leading to overoptimization, then the approximation might not hold. Can you provide a detailed discussion regarding the same? Discussion with works that rely on the principles of pessimism needs to be discussed [1,2]. \n\nReferences\n[1]. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf\n[2]. Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer\n\nQUESTIONS:\nPlease see weakness section\n\nLIMITATIONS:\nPlease see weakness section", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of reward over-optimization in offline Direct Alignment Algorithms (DAAs) like DPO. The authors identify that DAA's implicit KL regularization is only effective when the learned policy is close to the initial reference policy, leading to performance degradation as the model drifts away during training. To mitigate this, the paper proposes Importance Sampling DAAs IS-DAAs, a novel approach that re-weights the standard DAA loss with an importance sampling ratio. This re-weighting aims to better approximate the online learning objective using only offline data, thus enforcing a more effective regularization throughout the training process. To handle the high variance typically associated with importance sampling, the authors propose clipping the importance weights. The paper provides theoretical analysis showing that their method provides an unbiased estimate of the online DPO objective and that its gradient correctly corresponds to the gradient of the KL divergence. Through experiments on TL;DR summarization and instruction-following benchmarks, the authors demonstrate that IS-DAA effectively mitigates over-optimization, particularly under low regularization, and achieves superior performance compared to standard DAA and other baseline methods designed to tackle the same problem.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Significance: The paper tackles a well-known and highly relevant problem. Reward over-optimization and the performance gap between offline DAAs and online RLHF are significant barriers to creating more robust and reliable aligned models. The community is actively seeking simple and effective solutions, and this paper proposes a strong candidate.\n2. Clarity: The paper is well-written and easy to follow. The authors clearly motivate the problem, starting from the limitations of RLHF, moving to the promise and pitfalls of DAAs, and culminating in a crisp diagnosis of why their implicit regularization fails (Section 2.3). The progression from the standard DAA objective to the proposed IS-DAA objective is logical and well-explained.\n\nWeaknesses:\n\n1. Originality: The primary weakness is the novelty of the proposed technical components. As noted, importance sampling with clipping is proposed in PPO and serves as a cornerstone of modern RL. The paper re-applies this established technique in the context of DAAs. An expert in RL would immediately recognize the objective in Eq. (11) and the clipping in Eq. (13) as analogous to PPO's surrogate objective. The authors could be more explicit about this connection to prior work such as PPO. The contribution is therefore not in inventing a new technique, but in the application, diagnosis, and analysis of an existing one in a new domain.\n2. Quality: The paper introduces a clipping hyperparameter to control the bias-variance trade-off. However, the analysis is limited to a single ablation study (Figure 5). There is little discussion on the sensitivity to this hyperparameter or practical guidance on how to set it. Since clipping fundamentally changes the objective from an unbiased to a biased one, the paper could benefits from a deeper discussion of its impact and how it might interact with the learning rate or the DAA's own parameter.\n3. Quality: The method is motivated by better approximating the online DAA objective. A powerful, albeit computationally expensive, experiment would have been to compare IS-DAA (the offline approximation) directly against an online DPO baseline (which performs actual on-policy sampling). This would have provided a clear measure of how successfully the importance sampling bridges the gap between offline and online training.\n\nQUESTIONS:\n1. Can the authors offer a hypothesis or analysis for why X-PO might be more effective on AlpacaEval, while IS-DPO is stronger on Anthropic-HH? X-PO uses a chi-squared divergence which penalizes large deviations from the reference policy more heavily than the KL divergence implicit in DPO. Does IS-DPO, by more faithfully approximating an on-policy KL-regularized objective, perhaps inherit a specific behavior that is less suited to the data distribution or evaluation protocol of AlpacaEval?\n2. Could the authors provide more practical guidance on setting $\\epsilon$? A key question is its interaction with the main DAA regularization parameter $\\beta$. How should one co-tune $\\beta$ and $\\epsilon$? For instance, if one uses a very high $\\beta$ (forcing the policy to stay close to the reference), does the optimal $\\epsilon$ tend to be larger or smaller? Does the optimal $\\epsilon$ depend on properties of the offline dataset, such as its size or the diversity of the reference policy responses?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of reward over-optimization in offline Direct Alignment Algorithms (DAAs) like DPO. The authors hypothesize that the performance degradation observed in these methods stems from an inadequate implicit KL regularization, which becomes ineffective as the trained policy (π_θ) diverges from the initial reference policy (π_ref). This occurs because the DAA objective is optimized using a static, offline dataset sampled from π_ref, failing to account for the distribution shift during training.\n\nThe proposed solution Importance Sampling DAAs (IS-DAAs), directly tackles this issue. It modifies the standard DAA objective by multiplying the loss for each preference pair by an importance sampling ratio. Extensive experiments on summarization and instruction-following tasks demonstrate that IS-DAA significantly mitigates over-optimization, achieves a better performance-vs-KL-divergence trade-off, and outperforms existing regularization methods, especially under low β (KL penalty) settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles a recognized and important issue in LLM alignment: the tendency of offline Direct Alignment Algorithms (DAAs) like DPO to over-optimize and degrade in performance.\n- The proposed method is straightforward, which applies the standard technique of importance sampling to the DPO loss to correct for distribution shift.\n- The authors provide empirical results across several benchmarks that show some improvement over baseline DAA methods, particularly in mitigating performance decline during training.\n\nWeakness:\n- The core proposed by the authors that offline algorithms suffer from distribution shift as the policy moves away from the data-generating policy, is a well-known problem in the field of offline reinforcement learning. The proposed solution, applying importance sampling, is a standard method to tackle this, which has already applied in PPO. As such, the paper's conceptual contribution feels incremental rather than novel, amounting to the application of a known technique to the specific context of DPO.\n- The method's stability and performance rely entirely on the clipping ratio. However, the paper offers no principled way to set this crucial hyper-parameter, aside from a single ablation study shown in Figure 5. This introduces a new, sensitive parameter that a practitioner must optimize via expensive trial-and-error, which counteracts the claimed simplicity of the method and could easily negate its benefits in practice. \n- The paper claims the method has \"minimal computational overhead,\" but this is not thoroughly justified. Calculating sequence-level importance ratios for every sample in a batch adds computational steps that, while not requiring new forward passes, are not free. Considering its added complexity, it is not as simple implied by the paper.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-03"}

comments: SUMMARY:
* The paper proposes to avoid reward hacking in the offline direct reward optimization setting by regularizing through importance sampling.
* Theoretically, KL regularization applied in online reward optimization technique is computationally intensive when extended to the direct reward setting as it requires multiple samples to be generated from the trained policy model. This paper demonstrates that importance sampling using a Clipped product of likelihood ratios to weight the direct reward is sufficient to induce the KL-regularized reward training objective. 
* Further, empirical results on popular benchmarks like AlpacaEval and MTBench demonstrate the efficacy of the approach in achieving higher win rates with a lower KL penalty.

STRENGTHS AND WEAKNESSES:
Strengths: 

* The theoretical analysis of the regularization proposed is sound with valid comparisons with online KL regularized reward learning
* Empirical results demonstrate comparable or sometime higher win rates at lower KL penalty
* The sample efficiency in training direct reward learning is maintained while regularizing.

Weaknesses:
* The statistical significance of the results in Figure 3 and 4 are not clear as many of the baseline approaches are overlapping, with an exception of a low KL penalty (beta) setting where the importance sampling based approach performs better
* The analysis of the online DPO baseline which is cited as a motivation to approximate is missing in the results. It would be good to understand the headroom in the approximation proposed between this computationally intensive alternative

QUESTIONS:
Adding missing evaluations would improve the soundness of the results.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-08