Unifying Stable Optimization and Reference Regularization in RLHF
Authors: Li He
Qiang Qu
He Zhao
Stephen Wan
Dadong Wang
Lina Yao
Tongliang Liu
Venue: ICLR 2026 Poster (2026)
Keywords: RLHF
LLM
Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Reinforcement Learning from Human Feedback (RLHF) has advanced alignment capabilities significantly but remains hindered by two core challenges: reward hacking and stable optimization. Current solutions independently address these issues through separate regularization strategies, specifically a KL-divergence penalty against a supervised fine-tuned model ($\pi_0$) to mitigate reward hacking, and policy ratio clipping towards the current policy ($\pi_t$) to promote stable alignment. However, the implicit trade-off arising from simultaneously regularizing towards both $\pi_0$ and $\pi_t$ remains under-explored. In this paper, we introduce a unified regularization approach that explicitly balances the objectives of preventing reward hacking and maintaining stable policy updates. Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity. Extensive experiments across diverse benchmarks validate that our method consistently outperforms RLHF and online preference learning methods, achieving enhanced alignment performance and stability.

Published as a conference paper at ICLR 2026
UNIFYING STABLE OPTIMIZATION AND REFERENCE REGULARIZATION IN RLHF
Li He
Qiang Qu' He Zhao?
Stephen Wan
Dadong Wang² Lina Yao Tongliang Liu!* Sydney AI Centre, The University of Sydney CSIRO, Data61 University of New South Wales
ABSTRACT
Reinforcement Learning from Human Feedback (RLHF) has advanced alignment capabilities significantly but remains hindered by two core challenges: reward hacking and stable optimization. Current solutions independently address these issues through separate regularization strategies, specifically a KL-divergence penalty against supervised fine-tuned model (Tτ) to mitigate reward hacking, and policy ratio clipping towards the current policy (T²) to promote stable align- ment. However, the implicit trade-off arising from simultaneously regularizing towards both mo and ग. remains under-explored, In this paper, we introduce a uni- fied regularization approach that explicitly balances the objectives of preventing reward hacking and maintaining stable policy updates. Our simple yet principled alignment objective yields weighted supervised fine-tuning loss with superior trade-off, which demonstrably improves both alignment results and implementa- tion complexity. Extensive experiments across diverse benchmarks validate that our method consistently outperforms RI.HF and online preference learning meth- ods, achieving enhanced alignment performance and stability. Our implementa- tion is available at gigttb......... com/ ctwmtmllmmecccccocoocooccccccc.c haalllab/2220 ICLR DAR
INTRODUCTION
Large language models (LLMs) (Brown et al.) 2020; Bubeck et al. (023) have recently achieved human-level performance on a wide range of complex tasks (Guo et al., 2025 Patil & Jadon 2025 Zhang et al. 2025). These successes rely heavily on reinforcement learning from human feedback (RLHF), which aligns model behaviors with human preference. Despite its effectiveness, RLHF faces two fundamental and persistent challenges: reward hacking and stable policy optimization.
Reward hacking (Gao et al. 2023 Kwa et al.] 2024) Rafailov et al., 2024) Rashidinejad & Tian, significant mismatch between high expected rewards and actual poor performance. Stable policy optimization
2024 Wen al. 2024) occurs when the policy becomes over-optimized, resulting
(Schulman al. 2015a |Png et al. 2019 Touati et al. 2020) addresses the challenge where gradi- ents estimated from the RL objective can lead to drastic policy shifts and ultimately model collapse.
Current online RLHF methods tackle these issues through separate regularization mechanisms Ziegler et al.) 2019 Shao et al., 2024 Hu et al. 2025), introducing Kullback-Leibler (KL) di- vergence penalty with respect to the initialization model (mp) to combat reward hacking while using policy ratio clipping based the current policy (ττ) ensure optimization stability. However, these two critical regularization gradually evolve into conflict along the alignment process, creat- ing an overly restrictive optimization framework. By enforcing the constraints independently via penalty and clipping, the learning policy must remain close to both the initialization model and the current policy. This systematically excludes high-reward policies that could offer stable optimiza- tion guarantees but lie outside the intersection of both trust regions. As illustrated in Figure a this limitation is particularly problematic when the optimal policy alignment requires substantial be- havioral changes that extend beyond the reference support. Consequently, current methods that fail address the conflict of two opposing regularization achieve suboptimal alignment performance.
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Reward Hacking
*
Reward Hacking
*
Penalty
Clipping
Penalty
Search Expansion
9.0
Reward
N
31;
8.5
PPO
8.0
Dual-PPO
Dual-PPO-Clip
7.5
8
10
16
KL divergence mo
(a) PPO-based RLHF
(b) Dual-KL Alignment
(c) Reward- KL Pareto Frontier
Figure 1: Dual-KL regularization enables exploration beyond reference policy support. (a) PPO-based RLHF uses policy ratio clipping relative to т. for stable optimization and KL divergence penalty relative to Ho for reference regularization. High-reward regions remain unexplored when they lack sufficient support under the reference policy. (b) Our approach unifies stable optimization and reference regularization, enabling flexible trade-offs between the two mechanisms. This allows the policy to expand into high-reward regions previously inaccessible due to limited reference sup- port, achieving better alignment when substantial behavioral changes are required.
(c) Empirical validation on antropic-Hellfullm dataset: incorporating dual-KL KI. penalties in advantage estima- tion improves the reward-KK KL Pareto frontier over standard PPO for both Dual-PPO variants.
To both address and empirically validate this limitation, we consider a straightforward attempt to unify stable optimization and reference regularization through dual-KL regularization objective, as visualized in Figure lb). Specifically, we implement two variants of PPO (Stiennon et al. 2020
Ouyang al. 2022; Kaufmann al., (024) that incorporate two KL divergence penalties each with respect to and Ne in the advantage estimation (Schulman et al. 2015b). This unified alignment framework effectively enlarges the search area by flexibly allowing the policy to explore outside the reference support when stable optimization is guaranteed. The advantage of this expanded search space is evident in Figure [Tc), where both Dual-PPO variants achieve superior Pareto frontiers in the reward-KL tra The improved performance through dual-KL alignment highlights the necessity of a unified framework allowing for explicit trade-off between the two regularization.
Beyond these preliminary results, we theoretically analyze key advantage of this dual-KL ap- proach: the effective reference target becomes an interpolation of To and Te in log space that is gradually moving closer the optimal distribution along the alignment process. Leveraging this theoretical foundation, we reformulate the RI. alignment objective as a weighted supervised fine- tuning (SFT) loss, which both enhances learning stability and reduces implementation complexity. The resulting algorithm, Dual-regularized Advantage Regression (DAR), presents a novel RL- free algorithm that outperforms both online RLHF and online preference optimization methods.
In summary, we first provide empirical analysis identifying the limitations of existing RLHF meth- ods and demonstrate that the dual-KL objective effectively addresses them. Second, we propose DAR with rigorous theoretical foundations and show consistent empirical advantages over baseline algorithms within both direct AI aligmment and online RLHF pipelines. Finally, through extensive ablation studies, we establish connections between empirical results and theoretical insights, paving the way for broader applications of DAR across multiple downstream domains.
RELATED WORK
Online LLM Alignment predominantly relies on Proximal Policy Optimization (PPO) (Schulman ct al. 2017), which has demonstrated notable stability and effectiveness as an online policy-gradient method. However, the implementation complexity of PPO has motivated the search for simpler alternatives. In response, several variations of REINFORCE (Kool et al., 2019 Ahmadian et al.) 2024 Hu et al. 2025) have been proposed utilizing Monte- Carlo sampling for baseline estimation to avoid fitting a separate value model. These improvements are also incorporated in GRPO (Shao)
al. 2024). which serves as the optimization algorithm in developing state-of-the-art reasoning
2

---

Published as a conference paper at ICLR 2026
models (Guo et al.
2025). Complementary to these efforts, substantial research has further focused on adapting Direct Alignment from Preference (DAP) (Xiao et al. 2024 Ethayarajh et al.] 2
Liu et al.] 2025 Yang al. 2025b) online alignment by integrating varoos online feedback
mechanisms while maintaining algorithmic simplicity. These online preference labels are generated
Xu al. 2024b), human feedback (Xiong et al.] 2024). Al feedback (Huang et al. 2024 Qi et al. 2024), or self-feedback (Sun et al. [023] Bao et al. 2024 Wang et al. 2024a Lu et 2025).
using reward models (Chen et al.
2024
Despite these advances, prior works have not addressed the critical trade-off between stable RL optimization and reference regularization. Our work presents the first analysis this fundamental balance, revealing its importance for effective LLM alignment.
Reference Regularization has emerged as fundamental component in effective RLHF (Liu et al.,
2024a), prompting extensive research into alternatives and improvements to conventional imple- mentations. Recent advances have explored enhanced reference targets, with notable contributions investigating multi-target strategies (Aminian et al. 2025 Le et al., 2025), using behavior L.LM (Xu al. 2024c) and DPO-aligned LLM (Gou & Nguyen 2024). In different direction, dynamic ref- erence optimization approaches are proposed throughout the training process to improve alignment effectiveness (Gorbatovski et al., 2024 Ramé al. 2024 Yuan et al. 2026 Concurrently, efforts to reduce implementation complexity have yielded reference-free alignment techniques that substi- tute the traditional regularization term with SFT pre-training losses (Hong et al., [2024]]] Liu al.] 2024b). Taking this further, some researchers have demonstrated the feasibility of completely elim- inating reference regularization under specific assumptions. such as uniform reference policies (Xu et al. Iength. contth-contoolled alignment (Gupta et al. 2024 Meng et al. 2024] Xiao et al. (2025), and verifiable reward (Yu et al., 2025). Collectively, these developments support that reference reg- ularization should be viewed not as universally mandatory, but rather as a flexible, task-dependent design choice that can be adapted or even omitted based on specific training contexts and objectives.
Weighted Policy Regression solves RL problems via an iterative optimization framework in the style of supervised learning. An early example of this kind Reward Weighted Regression (RWR) by Peteers& & Schaal (2007), ; an on-policy RI. algorithm that updates the probability of each state- action pair based on their accumulative discounted return. Built upon RWR, Advantage Weighted Regression (AWR) (Peng et al.) (019)] proposes to instead calculate regression weight based on policy improvement and further incorporate off-policy data for better sample efficiency. Critic Reg- ularized Regression (Wang al. 2020) is an offline variant focusing training with pre-collected off-policy datasets. Our work extends the family of weighted regression algorithms to LLM align- ment by introducing reference regularization.
3 PRELIMINARIES
This section establishes the mathematical foundations for our proposed method. We begin with the problem formulation of RLHF, and examine how PPO is implemented for RLHF. We then dis- cuss stability guarantees policy optimization, and finally present Advantage Weighted Regression (AWR) as algorithmic foundation for our novel alignment approach.
RLHF Problem Formulation. RLHF (Dai et al. 2023 Zheng et al, 2023b Dong et al. 2024) aims to align language model To with human preferences by optimizing reward reward-maxine -maimizziiiigggg objective. Given a prompt dataset (zx) and a pre-trained reward model raz, v). the RLHF objective optimizes reward expectations with regularization
reference policy Net
JRLHP (To) (To: meet) max EZ-D [Eynrall a) ()( v) (eo())) || <<r(y|>)]] 30
where 3 controls the strength of reference regularization. The KL penalty serves two purposes: preserving prior knowledge from the reference policy and mitigating reward hacking. Typically, both no and मet are initialized from x SPT model fine-tuned high-quality demonstration data.
PPO for RLHE PPO optimizes the RLHF objective in terms of advantage functions and imple- menting stable policy updates (Ziegler al. 2019: Stiennon et al. 2020 Bai et al.) 2022 Ouyang
ct al. 2022 Zheng et al. (003c). The advantage A(r,)) y) = r(x,8) V (x) represents the reward improvement of response y relative to the expected return 4 (xr) under the current policy m. PPO
3

---

Published as a conference paper at ICLR 2026
optimizes the token-level advantage using clipped objective that constrains policy updates:
lal
(uo M<i) (36 Vei) A,, clip(
To(ui 1+ (2)
( mT) = max R gOMT We
min
TT [.
where the clip operator constrains the policy update ratio within [] e, +11++ e|. The reference reg- ularization in eq. ① for RLHF is incorporated via a token-level reward shaping term, so that the token-level reward is defined r, r(x, v) log [a x.(())>
KL. Penalty for Stable Optimization. The clipping mechanism in eq. ensures monotonic policy improvement by limiting the magnitude of policy updates. An equivalent formulation achieves alignment stability through KL divergence penalty with respect to :..
To (o 2. Veci)
JPPO-Penalyy (Te:: TT) = max Ex D.yoee
(B Bccs) A.
AD [T (vlz) ) Toe (v([)))
youke
where À > is the KL penalty coefficient. This penalty- based formulation reveals that stable policy optimization can be implemented via KL regularization (Schulman et al. 2015aa Peng et al. 2019)) providing bridge to unifying stable optimization and reference regularization as dual-KL, penalties.
METHOD
This section presents our methodological framework for balancing the trade-off between preventing reward hacking and maintaining stable optimization in RLHE. We begin by formalizing the dual- KL aligmment objective, which reformulates preference optimization as an advantage maximization problem with dual forward KL penalties:
Jaaal KL. (Tg: Tref nT) max E D. (A(2,) y)]
no
(3)
3 allx (Ty()) no(y]=)] + (1 a (To(lx)) <=(v)=)))
where B > 0 controls the overall regularization strength and E [0, balances the two KL terms.
We organize the of this section as follows. Section provides comprehensive analysis with additional experimental results showing the superiority of Dual-PPO over standard PPO. Building on these findings, Section 4.2) presents theoretical analysis explaining why this dual-KL objective provides a more effective reference target. Finally, Section 4.3 introduces DAR, our practical opti- mization framework that aligns LLMs through iterative weighted SFT, followed with gradient anal- ysis and implementation considerations.
4.1 UNDERSTANDING THE LIMITATION
PPO-hased RLHF approaches implement reference regularization and stable optimization as sep- arate components, resulting in constrained optimization problem that lacks effective trade-off mechanisms. Following eq. [I] and the formulation of Scchulman et al. (0017). PPO-based RLHF effectively optimizes the following constrained objective:
To x Ves) A - 3IL (y((z) <o(v|=)]
JPPO-Align(Tag Tey Teet) = max Eg Wootte
T. (M x. Fee))
8.l. (xI(TT. || mo .r) <e,
where is the KL divergence threshold. This formulation reveals the fundamental limitation of ex- isting methods. As the stable constraint operates outside the primary optimization objective, policy updates confined within the intersection of the trust regions of mo and π. As training progresses and ny diverges from Tp. this intersection becomes increasingly constrictive. This directly explains the widening performance gap in Figure (c) KL budgets increase.
Empirical Validation. To establish the empirical foundation for our approach, we conduct com- parative experiments to evaluate standard PPO against two dual-KI. variants the helpfulness task
4

---

Published as a conference paper at ICLR 2026
PPO Variants
Reward
Win % over no
Win over PPO
PPO
9.070
80.87%
Dual-PPO Dual-PPO-Clip
9.522
88.40%
58.52% 60.39%
9.581
90.37%
Table I: Comparative evaluation of PPO variants on the Helpfulness task using test samples, with Qwen2-72B -onstrrt-Insttuct as the judge. We report the mean reward, mean reference win rate, and mean win rate of Dual-PPO variants over PPO. Dual-PPO variants consistently outperform standard PPO.
Bai ct al. 2022). The distinction between variants lies in their clipping mechanisms: Dual-PPO- Clip incorporates the same policy ratio clipping as standard PPO, while Dual-PPP -aal operates without this constraint. Complete experimental details are provided in Appendix B
Our results provide compelling evidence for the superiority of dual-KL approaches. Beyond the improved Pareto frontiers, L.IM-judged evaluations in Table [ demonstrate that PPO with dual-KI. penalties consistently outperforms standard PPO across all metrics. This confirms that dual-KL. variants effectively mitigate the fundamental limitations of standard approaches, providing strong motivation for the theoretical analysis and practical framework developed in subsequent sections.
4.2 THEORETICAL ANALYSIS
To provide theoretical insight into the advantages of our unified approach, the following proposi- tion shows that our du.l-KI. objective is mathematically equivalent to optimizing against single, dynamically constructed reference policy:
Proposition 4.1
The dual-KL. advantage maximization objective in eq. equivalent to optimiz-
ing against an interpolated reference policy in log-space:
Janal KI. max E, B. [A(x, Dxy. T)())
To(v|z) n. (y(x))-a
C(x)
T6
Trel
30 ni (ylx))= is the normalizing factor for the effective reference target. where C(x) = C(x)=E,
The proof is provided in Appendix A.I
This theoretical unification provides crucial insights into the superiority of the dual-KL objective over standard approaches. Unlike previous online RLHF methods that rely exclusively static initialization policy, Proposition [4.] reveals that our objective implements dynamic regularization mechanism through a-weighted interpolation of the static To and the evolving my. The trade-off coefficient a provides explicit control over the reference target within the policy space, where increasing a shifts the target closer to up, illustrated in Figure
Log Policy Space
New Ref. Support
Tref
Ho
As me progressively aligns with human preferences through iter- ative optimization from ma. the log-likelihood interpolation con- structs reference target that is inherently positioned closer to the optimal policy distribution. Thus, such target provides superior support coverage high-reward regions, effectively expanding the optimization landscape towards human preferences throughout the alignment process. This mechanism directly explains our empiri- cal observation of widening performance advantages at higher KL budgets: the dual-KL objective regularizes against an increasingly favorable reference target while maintaining stability.
Old Ref. Support
Figure 2: Log-likelihood interpolation creates ref- erence target that provides better support for the opti- mal policy distribution.
5

---

Published as a conference paper at ICLR 2026
4.3 DUAL-REGULARIE ADVANTAGE REGRESSION
We now introduce DAR, our simple RL-free alignment framework that implements the dual-KL alignment objective through iterative weighted-SFT. Following established regression-based pol- icy search methods, DAR not only simplifies implementation complexity compared to PPO-based RLHF but also improves learning stability by optimizing through stable policy regression. Building upon the derivations from previous works (Peng et al. 2019) Peters & Schaal 2007] Rafailov et al., 2023), we derive the following theorem 4.2 as the theoretical foundation for our approach with detailed proof in Appendix A.2
Theorem 4.2 Under mild assumptions, the dual-constrained advantage maximization objective for- mulated in eq. (3) admits the following closed-form solution:
- exp (AAlo..))
1.
x" (y]z))
(xx)
where Z(x) = Z(z)=E ( 1 a exp ( (1)(2)))) is the partition function.
Leveraging the optimal policy л characterized in Theorem 4.2 we can transform eq. 3 into an
iterative policy regression problem. This approach obtains an improved policy T++1 by minimizing the KL-divergence between the learning policy my and x* using ne as the sampling policy:
Tc+1 = arg min E DKL[T (1)()))
|| [v(v)=)]
( (()().)))) To(ello)
arg min E DKL.
(o( Te (((lz)) exp
7(x)
To(y)ar) arg max Exen (yvr(alla))
exp ( (=A(o.)))
log (y)
Te T
Regularization Weight
Advantage Weight
SFT: increuse likelihood
The complete mathematical derivation of this tractable regression loss is provided in Appendix A.3
Objective Analysis. The iterative policy search in DAR manifests through increasing the log prob- ability of prompt-response pairs that generates each iteration. Since both the Regularization Weight and Advantage Weight eq. 4 are positive, the derived objective constitutes a weighted SFT loss. Intuitively, this objective assigns higher log probabilities to each response proportional to the product of 1) Regularization Weight (Wreg): a discount factor penalizing responses based on their divergence from the reference distribution, reflecting the desired trade-off between mo and m and 2) Advantage Weight (Waay): ): a reward signal that increases with expected policy improvement.
Practical Implementation. To avoid the need for separate value model, employ Monte Carlo sampling to estimate expected returns, consistent with recent alignment approaches (Hu et al.) 2025 Shao et al.) 2024 To enhance learning stability, we incorporate batch-based advantage normaliza- tion (Raffin et al. 202T). To mitigate potential gradient explosion (Peng et al. 2019) from exponen- tial operations, we implement& clipping mechanism with threshold Welip- applying the product of weights: min
Wadv. Wahp). The complete implementation is in Algorithm
Algorithm Dual-regularized Advantage Regression
Input:
prompt dataset D(x), reference model Tee. reward model r, training steps T', regulariza- tion coefficients a, 3, Monte Carlo sampling size K, clip threshold Welip
Initialize Jet. T(= = Tet.
for / to 7 do Sample prompt from Dx:)), K-shot responses {
1=1 from my (-(x;)
Calculate Advantage A(æ, wi) r(æ, y:) k Σ r(I, w:)
Calculate batch- mean μ4 and standard deviation の for Advantage
Apply Advantage Normalization: Anom (I, y1) [A y:) - MAA/OA
Calculate Regularization Weight Why and Advantage Weight widv
Update 8, into 0 using VoLDAR = - ΣK [min( miin w Welp V log To(y x)]
Let ne+1 m0
end for

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
In this section, we present a comprchensive empirical evaluation of DAR across two distinct online alignment settings to thoroughly assess its effectiveness in aligning LLMs. We evaluate DAR in: (1) direct Al alignment, where an LLM annotator serves as the reward judge, and (2) standard RLHF, which utilizes a reward model trained human preference data. We describe the datasets, models, baseline methods, and evaluation metrics for each setting below, with implementation details and hyperparameter configurations provided in Appendix B
Direct AI alignment (Lee et al. 2023 Guo et al., (024) provides a reasonably fair framework for comparing our method against a broad spectrum of baselines, including: 1) offline DAP methods:
DPO (Rafailov et al., 2023) and SimPO (Meng et al. 2024); 2) online DAP methods: DPO, IPO
(Azar et al., 2024). Zhao Zhao al. (2023); 2) online RLHF methods: PPO Ziegler et al. 2019).
RLOO (Ahmadian et al., 2024), GRPO (Shao et al.) 2024). Our experiments utilize three datasets covering diverse alignment objectives: Reddit TL;DR (Stiennon et al.) 2020) for summarization tasks, Anthropic Helpfulness for helpful dialogue generation, and Anthropic Harmlessness for safe dialogue generation (Bai et al., [022).. The base model is Qwen2-7B/QWen2.555
(Yang er al, (Yang al., (002aa) as the LLM annotator. Performance is assessed using win rates over To and reward scores judged by held-out test sets.
2024a; Owen et al., 2025), while employing Qwen2-72B-Inse 1nt.-nntruuc//Oeen333333333
GPT4--urbo/GPTS.1 (Achiam al., 2023 OpenA 2025
Standard RLHF utilizes a state-of-the reward model pre-trained on human preference data.
(Grattafiori et al., 2024). We use Helpsteer2 (Wang et al., 2024b). a comprehensive dataset that considers multiple aspects of re- sponse quality in facilitating helpful responses. Following the experimental setup, we employ an instruction-fiie finntuund mode. Qwen2-7B-Instruee as our base model. We compare DAR against two online RLHF algorithms: RLOO, GRPO; and iterative supervised fine-tuning with best-of-n sam- pling as baseline methods. Performance is assessed using MT-Bench (Zheng et al.) 2023a) and
The reward model is fine-tuned from
AlpacaEval 2.0 (Dubois al. 2025) benchmarks to ensure consistent and reliable assessment.
5.1 HOW DOES DAR PERFORM IN LEARNING FROM LLMs?
Figure 3 shows the reference win rate curves for DAR and baseline methods when finetuning Qwen2-7B, using Diven-72BB-Instruus as the LLM annotator and GPT-4-Turbo as the evaluator. Peak performance for each method summarized in Table 2 Table 3 presents the reference win rates of DAR against online RLHF methods when finetuning Qwen2.5-7B, using Qwen3-32B as the LLM annotator and GPT-5.1 as the preference judge. Our analysis reveals that DAR demonstrates superior performance in overall alignment quality and sample efficiency.
Comparison to DAP. Our results confirm three key findings regarding DAR's performance relative to preference learning methods. First, consistent with previous research, all online methods out- perform offline methods, highlighting the importance of online data collection in LLM alignment. Second, as shown in Figure 6 DAR exhibits markedly superior sample efficiency, converging using only half the annotations required by DAP methods. This efficiency advantage stems from DAR being an inherently online RLHF method that directly optimizes policy improvement rather than learning from comparison labels. Third, and most importantly, DAR consistently outperforms all three online DAP methods across all tasks. This consistent superiority suggests that online prefer- ence learning algorithms, which rely solely on reference regularization without stable optimization guarantees, converge to suboptimal performance due insufficient exploration of the policy space,
Comparison to online RLHE DAR demonstrates clear advantages over established online RLHF baselines in both final performance and learning dynamics. Specifically, DAR achieves mean ref- erence win rate of 92.42% across all three tasks, substantially outperforming the strongest baseline, GRPO, which achieves 15%. This 7.27% improvement empirically demonstrates the advantage of unifying stable optimization and reference regularization through dual-KL penalties, enabling exploration of an expanded search space that better covers high-reward regions. Notably, this per- formance advantage remains consistent across both experimental settings (Qwen2-7B with GPT- Turbo evaluation and Qwen2.5-7B with GPT-5.1 evaluation), validating DAR's robustness and effectiveness across different model scales and alignment scenarios.
7

---

Published as a conference paper at ICLR 2026
(a) Reddit TL
(b) Anthropic Helpfulness
(c) Anthropic Harmlessness
90%
85%
Win Rate
85%
Win Rate
Win
70%
63%
55%
0.0
0.2
0.4
0.6 Epochs
0.8
1.0
0.0
0.5
1.0
1.5
2,0
0.0
0.5
1.0 Epochs
1.5
2.0
Epochs
DAR (Ours)
RLOO
Online DPO
Online SLiC
SimPO
GRPO
PPO
Online IPO
DPO
Figure 3: Reference win rate curves of DAR against DAP methods and online RLHF methods. The base policy is Qwen2-7B, and the LLM annotator is Swen2-72B-Instruue Win rates are evaluated by Gur-o-Turbo on random test set of 1,000 examples. Shaded regions indicate 95% confidence intervals across 3 random seeds.
Table 2: Reference win rates for the best checkpoints in Figure 3
Results are averaged over 3 seeds.
The highest highlighted, and the best mean win is in bold.
Algorithm
TL;DR
Helpful
Harmless
Mean
Offline
DPO
67. 17.17%
81.34%:-0.911%
75.47%
Preference
SimPO
64.57%++ 588%
86.45%+1..55
18.27%
73. 10%
DPO
78.47%:+11.66
28.86%%:0.388%
83.55%:110666%
83.63%
Online
IPO
76.33%:tu.219
88, 17%::..203
84. 84.89%:10.29%
83. 13%
Preference
SLiC
78.29%::40.69
91.49%:++449
83.9 63.99%%=0.850
84.59%
PPO
72.87%.10.918
80.61 0.888
82. 2..94%
78.80%
Online
RLOO
17.59%
87.98%
85.00% 1.00%
83.52%
RLHIF
GRPO
83.03% 1.0%%
86.93%++0.42
85.50%++0.519
85.15%
DAR (Ours)
98.2776:+0555
93.16%6:+0488
85.8496:40.369
92.42 %
Table 3: Reference win rates of DAR against online RLHF methods. The base policy is Qwen2.5-7B using annotations by Qwen3-325 where the judge model GPT-5.1. Results are averaged over seeds based on 1 test set. The highest is highlighted, and the best mean win rate is in bold.
Algorithm
TL:DR
Helpful
Harmless
Mean
PPO
60.70%
61.73%=0.95%
72.50%
64.98%
Online RLHF
RLOO
68.38 68.80%=11444
84.52%=0.544
66.63%%=11664
76.65%
GRPO
68.27 1.68%
77.72% 0.76%
79.63 96 1.70%
75.21% %
DAR (Ours)
80.07% 10.44%
86.46% 0.54%
81.28% 10.40%%
82.60 %
5.2 CAN DAR IMPROVE THE KL/REWARD PARETO FRONTIER?
To comprehensively evaluate the theoretical benefits of DAR's interpolated reference target, we con- duct Pareto frontier analysis examining the fundamental trade-off between reward maximization and KI. regularization. By sweeping different coefficients, we map the complete reward-KL. reg- ularization trade-off space for each method. For DAR, we fix the trade-off coefficient and report the a-intcrpolated KL regularization our analysis, since DAR employs dual-KL regularization.
Figure presents the Pareto frontiers for DAR and online RLHF baselines across all three datasets, with dashed horizontal lines indicating DAR's s peak performance. First, DAR consistently dominates the Pareto frontier across all datasets, achieving higher rewards with reduced KL regularization. This consistent frontier improvement demonstrates that the interpolated target achieves a consistently bet- ter reward/KL. trade-off compared to the fixed initialization. Second, the peak performance of DAR
8

---

Published as a conference paper at ICLR 2026
(a) Reddit TL;DR
(b) Antropic Helpfulness
(c) Antropic Hamlessness
8.5
3.5
10
Reward
Reward
Reward
7.5
8.5
9.0
7.0
8.0
8.5
7.5
12
15
10
15
10
|Regularization Term]
|Regularization Term]
|Regularization Term]
DAR (Ours)
GRPO
RLOO
PPO
DAR Peak Performance
Pareto analysis of reward/KL regularization trade-off by sweeping 8. Each marker repre- sents k evaluation using Qwen2-72BBInstruse the annotator fine-tuning Qwen2-7B. Solid lines show second-order polynomial fits, while dashed lines indicate peak fitted reward for DAR.
Figure 4:
Table 4:
Performance of DAR fine-tuning Qwen2-7B IB-Istructt on Helpsteer2 against online RLHF baselines on MTT-Bench and AlpacaEval 2.0. The best results are highlighted.
MT Bench
AlphacaEval 2.0
GPT-4
GPT-4-Turbo
Len.
I.C% over (SE)
Len.
Qwen2-7B-Insiin Instrrct (Ty)
8.334
7.769
1340
2051
RLOO
8,409
7.893
1580
52.25 (0.14)
2076
GRPO
8.425
7.856
1559
50.50 (0.16)
2038
Iter-SFT
8.378
7.838
1343
49.80 (0.17)
2004
DAR (Ours)
8.538
7.931
1358
54.17 (0.23)
1963
confirms our previous empirical findings based on win rates, that DAR outperforms baseline meth- ods both Helpfulness and TL:DR tasks, and remain competitive Harmlessness. These Pareto frontier improvements provide compelling empirical evidence for our core theoretical motivation that dynamically interpolating between the current and the initialization policy.
5.3 How WELL CAN DAR ALIGN LLMS IN STANDARD RLHF?
To further evaluate DAR's effectiveness, we assess its performance in standard RLHF scenarios. Table 4 presents comparative results showing performance improvements for DAR and baseline methods. For MT-bench, report scores of the base and after alignment, while AlpacaEval 2.0 results are presented as win rates relative to the initialization policy.
The experimental results demonstrate DAR's superior alignment capabilities across both evaluation frameworks. Specifically. DAR achieves the most substantial performance improvements compared baseline RLHF methods, with particularly notable gains in LC%. These findings validate two key aspects of our approach: first, DAR's effectiveness generalizes across diverse downstream alignment tasks with varying characteristics; second, the restrictive constraint problem we identified exists in realistic settings, making our dual-KI. framework broadly applicable to practical RI.FF scenarios.
5.4 How IMPORTANT IS THE REGRESSION TRANSFORMATION?
To isolate the contribution of regression-based approach, derive DAO (Direct Alignment Optimization) as a variant of DAR that uses standard RL optimization instead regression (see Appendix A.4). Together with Dual-PPO, compare DAR against these two Dual-KL RL-based methods to examine the specific benefits of transforming the RL objective into weighted SFT.
Figure (a) and (b) present the reward curves Helpfulness and TL:DR datasets. The results reveal that DAR achieves superior final performance while maintaining consistent training stability throughout optimization. In contrast, DAO suffers from training instability, showing reward curve

---

Published as a conference paper at ICLR 2026
(a) Regression vs. RL on TL,DR
Trade off Coefficient
(e) Sampling on Rate
0.5
95%
Wis Raze (Chers)
95%
9236
tos Mas mggi
90%
neward
7.5
90%
Win Rate
85%
Win
85%
68%
169
202
80%
5.5
0A6
D40
Dual-99
42%
050
75%
0.00
0.25
0.50
0.75
1.00
80%
0
0.1
0.2
0.4
0.8
1.0
70%
16ihet
8-shet
4-shet
2-hhet
L-shot
Epochs
Trade-offe a
N-shot S Sampling
(b) Regression vs. RL on Helfpulness
(d) Total Regularization
(1) Weight Clip on Reward
65%
9.5
Reward
8.5
Win Rate
Reward
9.0
7.5
0.5
85%
$-
8- 0.05
0.68
D80
Dual IP0
2- 0.03
0-0.3
HapR
Hammisss
0.0
0.5
1.0
1.5
2.0
80%
0.6
0.8
1.0
1.2
1.4
1.6
7.5
10
20
50
150
Epochs
Epochs
Weight Clip
Figure 5: DAR vs. RL-based Dual-KL (DAO, Dual-PPO) on (a) TL;DR, and (b) Helpfulness. Ablation studies on the Helpfulness task: (c) Trade-off coefficient on alignment results; (d) Total regularization coefficient on win rate; (e) N-shot sampling size on win rate for DAR and Monte- Carlo baseline methods. (I) Weight Clip threshold Welip on reward for DAR across three datasets. The shaded area in (a), (b), (d) represents the 95% confidence interval over seed.
collapse. Dual-PPO exhibits significantly higher variance among different seeds, as it relies on accurate value predictions. These results demonstrate that the regression transformation is crucial for DAR's success, providing a simpler implementation as well as a key advantage in training stability.
5.5 ABLATION STUDIES
trade-off and total regularization. Figure 5(c), (d) shows the impact of varying a (fixed 3 = 0.05), and varying 3 (fixed a = 0.1). Reducing progressively increases both performance and generation length, but exhibit reward hacking behavior when reference-free at a = with unnecessarily long response with a missing-FOSS rate of 8%. This trend is well connected with our theoretical analysis on how a defines the reference target. DAR performs robustly across different 3 values, further validating the learning stability of regression-based method.
N-shot sampling and Weight Clipping. Figure 5]e) shows DAR's robustness across sampling sizes, even with one-shot. Our approach with weight clipping effectively minimizes gradient esti- mation variance typically introduced by imprecise advantage calculations at limited sampling sizes. Figure (5) shows that clipping threshold of 20 yields optimal results, which is consistent with the clipping value used by |Peng et al.] (2019). This effectively balances the preservation of reward signals while preventing outlier rewards from dominating the training process.
CONCLUSION
In this paper, we propose the dual-KL alignment objective by unifying stable optimization and ref- erence regularization, which explicitly address the fundamental trade-off between the two mecha- nisms. To practically implement this approach, we introduce DAR, a novel online algorithm that optimizes LLM alignment via weighted SFT loss. Our experimental results show that DAR effec- tively balances the inherent tension during the L.MM alignment process, achieving superior perfor- mance, more stable learning dynamics, and simplified implementation. This approach demonstrates consistent advantages over existing methods in both direct Al alignment and standard RLHF, under- scoring our method in advancing LLM alignment techniques.
Limitation. DAR requires online data collection and access to the current policy distribution, limit- ing its applicability to offline RLHF settings where on-policy samples for KL divergence estimation are unavailable. While off-policy corrections could potentially bridge this gap, adapting dual-KL. regularization purely offline scenarios remains an important challenge for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors thank the anonymous reviewers for their insightful and constructive feedback. We are grateful to Suqin Yuan, Muyang Li, Runqi Lin, Zhaoqing Wang, and Yuhao Wu for valuable discus- sions throughout this project. Li He is supported by the CSIRO Data61 PhD Scholarship. Tongliang Liu is partially supported by the following Australian Research Council projects: FT2201003 DP260102466, DP220102121, LP220100527, LP220200949. We acknowledge OpenAl for provid- ing API credits through the OpenAl Researcher Access Program.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint w3xiv:2333..8777 2023.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in lmm.. arXiv preprint arXiv:2402. 14740, 2024.
Gholamali Aminian, Amir R Asadi, Idan Shenfeld, and Youssef Mroueh. Theoretical analysis of kl-regularized rlhf with multiple reference models. arXiv preprint arXiv:2502.01203, 2025.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 21d::22044058222 2022.
Rong Bao, Rui Zheng, Shihan Dou, Xiao Wang, Enyu Zhou, Bo Wang, Qi Zhang, Liang Ding, and Dacheng Tao. Aligning large language models from self-reference ai feedback with one general principle. arXiv preprint arXiv:2406.1113 11190, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901,,027 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. URL : /wwxaiiii//// v.w.vivvvooooo www.oog///bbbbb.a.o... 22033. 12712
Lichang Chen, Jiuhai Chen, Chenxi Liu, John Kirchenbauer, Davit Soselia, Chen Zhu, Tom Gold- stein, Tianyi Zhou, and Heng Huang. Optune: Efficient online preference tuning, 2024. URL
https://arxx http:///axxvvv.ga/aa/a. www.ooo.///oooooooooo/ooooo.ooa/ http://b///40... 2406.07657
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint
arXiv:2310.12773. 2023.
Tri Dao. F Faster attention with better parallelism and work partitioning, 2023, URL
https://arxiv. wttp:///////../.......
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf.
arXiv preprint 2rXiv:2405.08880 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2025. URL hollhtts::.. /ttp:///aiivvv.. org/ abs/ 2404. 04475
11

---

Published as a conference paper at ICLR 2026
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01336, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-108666 PMLR, 2023.
Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov. Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09656 2024.
Qi Gou and Cam-Tu Nguyen. Mixed preference optimization: Reinforcement learning with data selection and better reference model. arXiv preprint arXiv:2403.19443, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178.. 2024.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan, Accelerate: Training and inference at scale made sim- ple, efficient and adaptable. https: : //g1 tegitthu...... wtt.ccmm////... htom/huggia ng face/ awwaelcellleeeeeeeeeeeeeeeeeaeee lerate 2022.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12900 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint PXXi:2402.044990 2024.
Taneesh Gupta, Rahul Madhavan, Xuchao Zhang. Chetan Bansal, and Saravan Rajmohan. Refa: Reference free alignment with fine-graine fine-grained length control. In Second Conference on Language Modeling, 2024.
Jiwoo Hong. Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model, 2024. URL https: //trxiaaaa.v///////a/eeeeaaaa wwwporgggbbb//.a.. 07691
Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025. URL /tttps:///// org. abs abs/2501. 03262
Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning. In Forty-first International Conference on Machine Learning, 2024. URL net forumorrummmmm 1ewrummmdd-wwWWWWRR.....
htttps://cpenrevilt
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/2310.06825, 2023. URL. https://app http:/pp..emaaiiiiiiiimm seemanticschollaa seeeannnisssoolal...o//ooooooopo hrtg:oooo//oorto/ 263830494]
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback, 2024. URL htttattt/aaxii...oto http:///aaxxi..oooo///222111 hbtp://aab//aa..eeee 14925
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get baseline for free!. 2019. htRL:ttt::/// htt.tpps///eeenreeee..... net/ ltaeet//ooommmn::111110::
Thomas Kwa, Drake Thomas, and Adrià Garriga-Alonso. Catastrophic goodhart: regularizing rlhf with kl divergence does not mitigate heavy-tailed reward misspecification. Advances in Neural Information Processing Systems, 37:14608-14633, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
12

---

Published as a conference paper at ICLR 2026
Hung Le, Quan Hung Tran, Dung Nguyen, Kien Do, Saloni Mittal, Kelechi Ogueji, and Svetha Venkatesh. Multi-reference preference optimization for large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 24375-24383, 2025.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:339.000667 2023.
Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, et al. A survey of direct preference optimization. arXiv preprint arXiv:2503. 11701, 2025.
Yixin Liu, Pengfei Liu, and Arman Cohan. Understanding reference policies in direct preference optimization. arXiv preprint arXiv:2407.13709 2024a.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang. Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization rlhf: Your sft loss implicitly an ad- versarial regularizer. Advances in Neural Information Processing Systems, 37:138663-138697, 2024b.
Songshuo Lu, Hua Wang, Zhi Chen, and Yaohua Tang. Urpo: A unified reward & policy optimiza- tion framework for large language models. arXiv preprint arXiv:2507.17515 2025.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198--222355, 2024.
OpenAL. Gpt-5.1: A smarter, more conversational chatgpt. https:://geeaa..commneees gpt-5-11 5-1 November 2025. Accessed: 2025-11-23.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730 27744, 2022.
Avinash Patil and Aryan Jadon. Advancing reasoning in large language models: Promising methods and approaches. arXiv preprint arXiv:2502.03671, 2025.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighhte regression: Simple and scalable oof-policy reinforcement learning. arXiv preprint 1rXiv:1910.000177,2000 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward- weighted regression for operational space control. In Proceedings of the 24th international conference Machine learning. pp. 745- 750, 2007.
Biqing Qi, Pengfei Li, Fangyuan Li, Junqi Gao, Kaiyan Zhang, and Bowen Zhou. Online dpo: Online direct preference optimization with fast-slow chasing, 2024. URL https://arxxi.. org/ htg/abs/2e 020.444666600 05534
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2.5 technical report, 2025. URL URL https://aaxx
org abs//bb//00 2412. 15115
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-533 53741, 2023.
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejna, Brad Knox, Chelsea Finn, and Scott Nickum. Scaling laws for reward model overoptimization in direct align- ment algorithms. Advances in Neural Information Processing Systems, 37: 37:126207-122222. 2024.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. :aseli Reliable reinforcement learning implementations. Journal of machine learning research, 22(268): 2021.
13

---

Published as a conference paper at ICLR 2026
Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre- Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies (2024). URL https://aaxii. wtg/abs/2406.16768, 2024.
Paria Rushidinejad and Yuandong Tian. Sail into the headwind: Alignment via robust rewards and dynamic labels against reward hacking. arXiv preprint arXiv:2412.09544 2024.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 106.v41550.02438,, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1707.07777706477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL httppssts///axii org abs abss/2a 200
03300
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-46060 PMLR, 2018.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008- 3021, 2020.
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with instructable reward models. arXiv preprint arXiv:2310.05910. 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.001 18, 2024.
Ahmed Touati, Amy Zhang, Joelle Pineau, and Pascal Vincent. Stable policy optimization via off-policy divergence regularization. In Conference on Uncertainty in Artificial Intelligence, pp- 1338-1337. PMLR, 2020.
Leandro Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. /tttps://qintee
2020.
Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. Cream: Consistency regularized self-rewarding language models. arXiv preprint arXiv:2410. 2tti:2220..273... 12735, 2024a.
Zhilin Wang, Alexander Bukharin. Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preferenee: Complementing ratings with preferences, 2024b.
attp:/bss/2/100/ 01257
Ziyu Wang, Alexander Novikov, Konrad ZoIna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778. 2020.
Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Bowman, He He, and Shi Feng. Language models learn to mislead humans via rlhf, 2024. URL https://arra wttp://///..////////
12822
14

---

Published as a conference paper at ICLR 2026
Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. Simper: A minimalist approach to preference alignment without hyperparameters. arXiv preprint htt1:2502000883. 2025.
Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Zongrui Li, Ruirui Lei, Wanggui He, Luu Anh Tuan, Long Chen, Hao Jiang, al. A comprehensive survey of direct preference optimization: Datasets, theories, variants, and applications. arXiv preprint arXiv:2410. 111/22400.155555 15595, 2024.
Wei Xiong, Hanze Dong. Chenlu Ye, Ziqi Wang. Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rihf under kl-constraint, 2024.
/tttpp:://rxiiv.. org/ abs 221. 11456
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of I1m performance in machine translation. arXiv preprint axXiv:240.0088 2024a.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? comprehensive study, 2024b. URL attps://aran //tp/v/aavvvvv///////// www./bs.//aaeooooootooooe.eeesst 20000//204..1111
Wenda Xu, Jiachen Li, William Yang Wang, and Lei Li. Bpo: Staying close to the behavior Ilm creates better online Ilm alignment. arXiv preprint arXiv:2406. 12168, 2024c.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint 1111552412.15115. 2024a.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Puning Yang, Qizhou Wang, Zhuo Huang, Tongliang Liu, Chengqi Zhang, and Bo Han. Ex- ploring criteria of loss reweighting enhance Ilm unlearning. In Forty-second International Conference on Machine Learning, 2025b. URL ltttttt//// http///ppeeeviiieee net wtf.moorm?i/ ?id= /
InGOugCZ1Aq
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406. 10216, 2024b.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503.144700 2025.
Suqin Yuan, Xingrui Yu, Jiyang Zheng, Lei Feng, Dadong Wang, Ivor Tsang, and Tongliang Liu. Mitigating mismatch within reference-based preference optimization. In The Fourteenth Interna- tional Conference on Learning Representations, 2026.
Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Yang Yuan, Quanquan Gu, and Andrew Yao. On the design of kl-regularized policy gradient algorithms for Ilm reasoning. arXiv preprint arXiv:2505.17500 17508, 2025.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judee with mt- bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023a.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf large language models part Ppo. arXiv preprint N0X4:2307.04964. 2023b.
15

---

Published as a conference paper at ICLR 2026
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Yuhao Zhou, Limao Xiong, et al. Delve into ppo: Implementation matters for stable rlhf. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023c.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:19090085000 2019.
16

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENTS
While this paper introduces a novel alignment approach for LLMs, we acknowledge potential ethical risks inherent in the RLHF alignment process. Our method may inadvertently incorporate systematic biases present in human evaluators, or be misused to reinforce harmful behaviors if implemented without proper safeguards. Although our contribution is primarily methodological, we strongly advocate for responsible implementation and encourage practitioners to prioritize safery, fairness, and comprehensive evaluation when applying these techniques to ensure beneficial outcomes.
REPRODUCIBILITY STATEMENTS
We are committed to ensuring full reproducibility of our work and provide comprehensive support- ing materials. Complete proofs of all theoretical claims and derivations with clear explanations of assumptions are included in Appendix A The source code for our DAR method is provided in the supplementary materials. while Appendix contains detailed implementation details for both DAR and baseline methods, along with links to all datasets and models used in our experiments.
LLM USAGE
The use of LLMs was limited to editorial assistance for improving the clarity of this paper.
A MATHEMATICAL DERIVATIONS
A.1 PROOF OF PROPOSITION 4.1
Proposition 4.1 Let mo be model initialization and H, be the current model, where we To(y/>) > 0 and ( (r) > 0 for all pairs of prompts and answers y. Given a prompt dataset D(x), parameters
[0. 1] and | 0, the dual-KL regularization introduced in the objective of eq. for the learning policy can be equivalently represented as a single reference regularization
with Ner = C(w) (y 2² Te (y/=)²=-, where C(x) = E, To To(y m (e(yx)) I-a is the normalizing factor for the effective reference target.
Proof. We begin with the dual-KL objective defined in eq.
Jd (TTo; Frets Te) = max D.w.a.lll [A(x, [A(r, w)]
a DRL To(y|>>) < + (1 a DK mo(yl>)) Il ma(ulo)))
Focusing on the dual-KI. term, we perform the following manipulations:
aDDLL To(U|>:)
<o((|>))
+ (1 Tx.. To(y)>) m=(el>))
log (yrla) No (v)):)
To (tylar)
E (yyle
a)) log
= 3- Eyyxro(yle) log To - a log (yylo) (1 - ax) log N. (ylx))
To make this connection explicit, we rewrite the expression by combining the logarithmic terms:
1 3 (y)rx( log ()(()))) log C(x) To(y)=)" (e(l)>) l-o 1-o log C(x)
where C(x) Eg ,o(y|0)
((([)))- ( is the normalizing factor ensuring the unified target To(y)æ))* n. (g)l:)) as valid policy distribution. And log C(x) is constant that can be moved from the expectation:
log no ( Cla) c ( t. (v)
3 By (ulz)
3 TDkL. To(y)>))
noly m
(5)
C(x)
17

---

Published as a conference paper at ICLR 2026
Substituting eq. (5)
into eq. 33.
we obtain the alignment objective with single regularization target:
KL max (A(x, (or
xx To(y|=) [o(g)>)
||
n
C(a)
This completes the proof.
We have demonstrated that the dual-KL alignment objective proposed in this paper, which explicitly addresses the trade-off between reference regularization тo and stable optimization mis effectively regularizes the learning policy me with respect an interpolated target of mo and my with parameter a controlling their relative influence.
A.2 PROOF OF THEOREM 4.2
Theorem 4.2 Restated. Consider reinforcement learning setup where =(z.))=r(o.
an advantage fiuctiin for all prompt-respons. pairs (x. y), with r(x. y) being reward model and T (x) denoting the expected reward under the current policy wt- Lel ny a model initialization, and [0, 1], B 0 be regularization parameters. Assuming both To To(y)æ) > 0 and Tell)=) O hold for all prompt-respone pairs, there exists a close-form solution to the dual-constrained advantage (or reward) optimization objective for the learning policy no:
A(x,
max xy
3 (aDxL TT())=)) mo(ulz) (1 aa]Dke To((|=) || m<(ol=)))
The solution takes the form:
m exp (#AAC.v)):
(y) (r)x))= = Z(x)
(6)
where = < - m. - exp
( A(x,)))) the partition function.
By expanding and reorganizing the optimization objective derived in Proposition have:
Proof.
we
max E D.yDoyy
A(z, v)
8 Dk. T || mo(y/z)* Te
min Egw D.
log
<o()(>>) ( -
To To(y
1
T0(
min D.w.Dyyvoollle)
log
log ((x)
(7)
2(z) (g1z) exp A(x,))
As the partition function Z(x) depends exclusively on To and Ti, and the advantage function
A y) r(x, y) 4 (a: (none of which depend the learning policy лa), log 7(x) acts as a constant term in our optimization objective. Consequently, we can eliminate this term from eq. (
obtain:
To(v|=>
min E, (.wo(()))))) log
Z(w) To (e (yy)) exp ((x,,)))
(
min
Exn DxL.
To (y(xx) ||
Z(x) To (yllll))))
exp
(8)
Based on Gibbs' inequality, eq.
8 is minimized when the two distributions are identical. We have:
n (y)z)= = Z(x) (=(v)=)=<<vll>)>-->>>>))>>><)))) exp (1A(c,))
which completes the proof.
18

---

Published as a conference paper at ICLR 2026
A.3 DAR DERIVATION
Having derived the optimal policy œ" in Theorem 4.2 we formulate DAR's iterative policy regres- sion loss to obtain an improved policy T++1 by minimizing the KL-divergence between a parame- terized policy and r²:
Me+1 arg min EgnD DKL x (vlz)) mo(vlo)
and substitute in eq. (
To(y|>)* Te([[=)>~ exp ( mo(ull)).
arg min E DKL.
((x)
after expanding the KL--divergence term, we can further reduce the objective by dropping out the terms not dependent on IT!
= arg min EAND [- Σ Z(x) ,o(y|x) ne(ulx)* exp (+ v) log To =o((la))
factor the partition function term it is positive scaling factor is independent of our policy, thus not shifting the optimal policy:
|- Σ mo(ua))" (u(l))) exp
( (=)(().)) log foglo()))))
= arg min ERDD
Xg
we obtain our final optimization objective by taking ni as our sampling policy:
To (T(())z) exp (Z, log mo(ulo)) ()(2)))
T+++1 = arg max ERED B Bynune(sla))
We
A.4 DAO DERIVATION
In addition to the cinesssion-based DAR, we derive an optimization variant for optimizing the dual-constrained alignment objective called -egulregul Advantage Optimization (DAO). We present DAO in two formulations that differ in their sampling strategy and gradient estimation: REItFORCE-style policy gradient approach and a PPO-style optimization with importance sam- pling.
REINFORCE-style Policy Gradient. begin with the objective from Proposition
JDAO = max En (gw.o [A(r, v)] - DxL TT(()=)) To (x) Te ())l))²
In this REINFORCE-style formulation, we sample directly from the policy mo being optimized. To derive the policy gradient, we calculate the derivative of the objective:
To(
VoJDAN EIND Σ Vo To(y)=)A(x,))) v) 33 mo(y)>>) log mo(y)zr) ग. (
Vo lolog (y(l))) (Axx,v) - log no To(y|z) Te (g
no
PPO-style Optimization. Following the approach in PPO and GRPO, we can instead use the current policy n, for sampling. Starting with the same objective, apply importance sampling to convert this into an expectation over samples from Ny rather than ne:
( (
A(æ, y) - To To(2 log
n (y/z²) x
JDAO max Exw (.yreelllll)))
( ग. (vl (1lar)1-0
19

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL DETAILS
B.1 EXPERIMENTAL SETUP
For all experiments in this paper, we retain only the input prompts from each preference dataset. We employ the Adafactor optimizer (Shazeer & Stern, 2018) without adaptive learning rate updates, using 15 warmup steps followed by a constant learning rate. All training is conducted on NVIDIA H100 GPUs, utilizing both flash-attention22 (Dao, 2023) and accelerate (Gugger et al. 2022) for optimization. During online data collection and evaluations, we sample completions with a temper- ature of 0.9. The decoding strategy used for MT- -TT-Becch and Alpaca-Eval Eval 2.0 is greedy sampling. For all methods using Monte-Carle sampling to estimate expected return, we use a sampling size of 4 throughout.
Direct AI Alignment. We initialize the base model from Qwen2-7B and Qwen2.5-77 7B and perform supervised fine-tuning on TL;DR Helpfulness) and Harmlessness using learning rate of 6e-6. The SFT dataset for HH is the chonse responses, for TL:DR is the SFT dataset of human demonstration [ When sampling online responses, we limit generation 64 tokens for the TL:DR, and 256 tokens for Helpfulness and Harmlessness. The training schedule varies by method: online RLHF methods run for 1 epoch on TL:DR, and 2 epochs on Helpfulness and Harmlessness, while online DAP methods run for 5 epochs on all three tasks. Throughout training, we save 10 checkpoints for evaluation.
Our simplified direct-RLAIF (Lee et al. 2023) implementation utilize Qwen2-72B-Insnn Instruct Qwen3- 32B as the AI annotator for zero-shot annotation using reward prompts (Appendix D.I) and prefer- ence prompts (Appendix D.2). The annotator generate judgments with a maximum length of 256 tokens at zero temperature, from which we extracted reward and preference labels through pattern matching using predefined rules. We employe vLLM (Kwon et al.) 2023) as our inference engine for its computational efficiency, strictly adhering to pre-defined chat templates.
The alignment experiments use an online batch size of 512 and an effective batch size of 128, performing four gradient updates per online batch. The online DAP methods use a learning rate of 5e-7, while the online RLHF methods use a learning rate of 1e-6 (except for Dual-PPO variants). For evaluation, primarily rely on the metrics of reference win rate and AI reward, with judgments generated by both TPT-4--TrrboGPT--. and the corresponding LLM annotator. Each evaluation select random subset of 1,000 test samples with the annotator's temperature set to 0, ensuring consistent judgment quality.
PPO Variants Comparison. This experimental setup follows the direct AI alignment setting de- scribed above.
Standard RLHE. For the standard RLHF setting, we fine-tune Qwen2-7BB-Intrrue using a pre- trained reward model, learning rate of le-6 with weight decay of 0.01, an online batch size of 512, and an effective batch size of 128. We set the maximum token generation limit to 1000. The model is trained for complete epoch of the dataset, with the final checkpoint saved for evaluation.
on the HelpSteer2 dataset. We use a
To assess model performance, this setting employs two widely-adopted benchmarks: MT-Bench and AlpacaEval. MT-Bench evaluates multi-turn conversational ability across eight categories (writing, roleplay, reasoning, mathematics, coding, extraction, STEM, and humanities), using LLM annota- tors to score responses on 10-point scale. AlpacaEval assesses single-turn instruction-follooing on 805 questions, measuring win rates against baseline model using automated evaluation. We lever- age GPT-4-Turbo (Achiam et al. 2023 as the evaluation judge for both MT-Bende and AlpacaEval 2.0, while we also report the MenBench score judge by GPT-4.
B.2 DAR IMPLEMENTATION
Our implementation of DAR builds upon the DPO trainer from TRL. For both settings, we use weight clip threshold of 20. The differences lie in the trade-off and total regularization coefficients: we set a 0.1 and 3 = 0.05 for Direct AI Alignment, while for the standard RLHF, we use = 0.5 and / 0.5. To explore the impact of the regularization term, we plot the Pareto frontier using a fixed = 0.1 and varying values of B.
20

---

Published as a conference paper at ICLR 2026
B.3 BASELINE ALGOITHHM
For all the algorithms, we directly use or adapt the trainer implementations provided by TRL von
Werra et al. 2020).
PPO. We use the PPO trainer provided in TRL., As our preliminary experiments demonstrate that PPO is rather sensitive hyperparameters and yields performance that is not competitive compared to other online RLHF methods, we only implement PPO in the direct AI alignment setting, The value model is initialized with Qwen2-1.55D We set the PPO epoch to 4 per online batch, with missing-eos penalty of 1, constant KL coefficient of 0.03, a clip range of 0.2 (including on the value), and value function coefficient of 10 considering the small parameter size compared to the policy. Following the design decisions discussed by Xu et al. (2024b), we further modify the PPO trainer to implement scalar reward assignment at the end of the sequence.
Dual-PPO. Dual-PPO is adapted from the PPO trainer via modifying the advantage estimation. s(yl=)
mfyl=) (1 a)) log x+(g ), which is cal- Specifically, we redefine reward r(z, y) 3(x log xa(g]æ)
culated in token-level in the practical implementation. In the experiments, the coefficients are set as
= 0.3, B = 0.02. And we use single PPO update epoch, paired up with a learning rate of 3e-6 on all three datasets.
Dual-PPO-Clip. Dual-PPO-Clip implements the policy ratio clipping mechanism based Dual- PPO using a hard threshold of 0.2 consistent to PPO. While the coefficients are a = 0.9 and B 0.01, the rest hyperparameters remain the same the ones in Dual-PPO.
RLOO. The implementation of RLOO in our experiments is adapted from the DPO trainer, follow- ing a similar approach to DAR implementation. In the direct AI alignemnt setting, we employ constant KL coefficient of 0.03. For the HelpSteer2 task, we adopt a KL. coefficient of 0.05.
GRPO. The GRPO implementation is also adapted from the DPO trainer. In direct Al alignment, employ a constant KL coefficient of 0.04 (also reported by Shao et al. (2024)). ). For the HelpSteer2 task, we adopt KI. coefficient of 0.03.
Iterative SFT. We implement best-of-4 sampling with iterative SFT, where each online batch gen- erates single gradient update.
DAO. The hyperparameters for DAO follow those used for DAR, with the only difference being the removal of the weight clip threshold. In our experiments, we implement the REINFORCE-style..
DAP Methods. The three online DAP methods, DPO, IPO, and SLiC, all utilize the same underlying trainer in TRL. but differ in their loss calculation mechanisms. We directly use the configuration reported in the online feedback framework proposed by Guo et al. (2024). that both online DPO and offline DPO employ a KL coefficient of 0.1. IPO uses KL coefficient of 1.0, while SLiC operates with coefficient of 0.002. For SimPO, we conducted hyperparameter tuning within the search ranges specified by Meng et al. (2024). The optimal configuration was selected based on the performance on the Helpiulness task, yielding / = 2 and y = 1.6 as the final hyperparameter values.
In Table we summarize the hyperparameter search ranges used for the online RLHF algorithms.
Table 5: Hyperparameter search range for online RLHF methods.
Method
Hyperparameter
PPO
8 [0.01, 0.02, 0.03, 0.05, 0.1]
Dual-PPO
B [0.01, 0.03, 0.05, 0.1]
Dual-PPO-Clip
a E 0.31, 0.7.
RLOO
8 [0.005, 0.03, 0.05, 0.1
GRPO
B E [0.001, 0.005, 0.01, 0.02, 0.03, 0.04]
B E [0.05, 0.1, 0.3,
DAR
a E [0.1, 0.3, 0.5, 0.7,
21

---

Published as a conference paper at ICLR 2026
B.4 PARETO ANALYSIS
We conduct Pareto analysis on the Reward/KL trade-off by sweeping over 3 for each method, as defined in Table [ For methods involving dual-KL. objectives, trade-off coefficient held constant during the sweep. Specifically; 0.3 for Dual-PPO, 0.9 for Dual-PPO-Clip. a 0.1 for DAR.
C MORE RESULTS
C.I LLM-AS-A-JUDGE
Table 6: Human-Al agreement comparing Al reward and Al preference methods on a 1k subset of the test sets for TL;DR, Helpfulness, and Harmlessness tasks. LLM annotators include Qwen2, Llama-3, Mistral, Gemma-2, and GPT-4. Al preference agreement is averaged over both orderings (chosen vs. rejected) and (rejected vs. chosen) to mitigate positional bias. Llama-3 results on Harmlessness unavailable due to frequent invalid judgments triggered by the model's safety mechanisms.
Reward
Preference
Model
Version
TL:DR
Helpful
Harnless
TL:DR
Helpful
Harmless
72B-Instruct
74.97%
73.25%
73.89%
71.35%
71.15%
67.19%
Qwen2
IBIInstu
67.32%
72.10%
61.86%
65.74%
66.42%
61.40%
Ins-rutt 73.70%
73.07%
58. 1.3%
69.82%
Llama-3
N/A
N/A
8n-trrutct
61.15%
70.16%
60.95%
66.03%
8x7B-Instruct
75.86%
72.35%
72.13%
67.76%
68.59%
51.22%
Mistral
7B-ntttucct
69.85%
68.09%
70.05%
64.55%
67.01% %
63.50%
27b-il
74.84%
72.44%
74.44%
68.45%
67.37%
68.93%
Gemma-2
9b-it
74.87%
70.66%
72.71% %
67.36%
66.73%
67.43%
Llama-3.1
405t-Instruct
79.32%
74.34%
81.58%
72.76%
68.14%
60.25%
GPT-4
0613
76.12%
73.81%
79.08%
72.91%
73.67%
55.64%
To justify using LLM annotators in direct Al alignment settings, we evaluate human-Al agreement using pre-collected human preference labels ground truth. For AI reward computation, first obtain reward scores for each response pair, then determine preference labels through pairwise com- parison of these rewards, designating the response with the higher reward value preferred. We consider a large range of open-source models different parameter sizes, they are Qwen2 (Yang al. 2024a), Llama (Grattafiori et al.] 2024). Mistral (Tiang al. 2023), Gemma (Team et al., 2024 In addition, we include GPT-4 (Achiam et al. 2023) for reference purpose. As shown in Table larger open-source models demonstrate high agreement with human preferences. With AI reward labels gives higher human-ai agreement comparing to ai preference labels.
Granularity of Al reward. In the absence of human-annotated reward labels for the three datasets, we establish ground truth baselines using reward labels generated by GPT-4, LLaMA-3.1-444 1400BB, and a pre-trained reward model by Yang et al.) (2024b). This framework enables the granularity of reward labels generated by open-source L.MSs. We employ Pearson correlation coefficients across two evaluation scenarios: first, correlations computed over the complete set of 2,000 reward labels, and second, correlations focused specifically tied comparison pairs where both responses receive identical AI reward labels. This methodology provides insights into both overall correlation patterns and the models' ability to handle nuanced distinctions between similar- quality responses.
evaluate
The empirical results in Table demonstrate strong positive correlations between LLM -eneereeteedd reward labels and all three ground truth baselines. These correlations indicate that the reward labels effectively capture qualitative differences between chosen and rejected responses. Notably, the com-
22

---

Published as a conference paper at ICLR 2026
parable Pearson correlations between tied response pairs and the full dataset suggest that the reward labeling mechanism maintains consistency even when evaluating responses similar quality.
In conclusion, these results provide justification for using these open-source LLMs Al annotators both during training and evaluation. It is worth noting that, our finding support the robustness of the reward labeling system for preserving underlying preference patterns across varying response quality levels.
Table 7: Pearson correlation coefficients between Al reward and ground truth baselines: GPT-4, Llama-3.1 1atna-3.1-405B5. 1-405B, and pre-trained reward model. r represents correlations over the full set of 2,000 reward labels; r(tie) represents correlations over tied comparison pairs with identical Al-generated reward labels. Results are averaged across three datasets: TL:DR, helpfulness, and harmlessness.
GPT-4
Llama-3.1 3.1-405B
RM-UniFeedback
Model
(tie)
r (tie)
r
(tie)
Qwen2-72B-Instre 0.8023
0.8308
0.7948
0.8255
0.6868
0.6833
1orma--3--70BImmme
0.7352
0.7511
0.7531
0.7612
0.6443
0.6175
Mistral-8x7B -aal--xxxB-Inttreee
0.6947
0.6383
0.6591
0.5996
0.6471
0.5850
Gemma-2-27B-it
0.7635
0.7745
0.7621
0.7702
0.6821
0.6869
C.2 DAR vs. ONLINE DAP
Reddit TL;DR
(b) Anthropic Helpfulness
(c) Anthropic Harmlessness
90.0%
85.0%
Win Rate
85%
Win Rate
Win Rate
80.0%
75.0%
65%
60.0%
55.0%
Epochs
Epochs
Epochs
DAR (Ours)
Online
Offiine
Online DPO
Online SLIC
DAR Peak Performance
Figure 6: Win rate curves of DAR against online DAP methods (DPO, IPO, SLiC), and offline DPO. The shaded area represents the 95% confidence interval computed over seeds, and the win rates judged by GPT-4-Turbo based 1k random test set of three datasets.
In the direct Al alignment setting, online DAP methods requires significantly longer training epochs. In our experiment section, we present the results of online DAP as dashed lines. Here we provide full win rate curves comparing our method against the online DAP and offline DPO in Figure 6
Figure/ddmmonstraatt that DAR achieves superior alignment results while maintaining greater learn- ing efficiency than existing online DAP baselines.
C.3 DAR vS. ONLINE RLHF
To evaluate performance from a standard RL perspective, we present AI reward curves in Figure 7 As expected, these reward trajectories strongly correlate with the win rate patterns observed previously. On the helpfulness and TL:DR tasks, DAR achieves the best performance, significantly outperforming all bascline methods. On the harmlessness dataset, DAR and GRPO demonstrate competitive performance relative to each other, both outperforming PPO and RLOO.
C.4 FORWARD KI VS. REVERSE KL FOR STABILITY
A key design choice in our method is the use of reverse KL divergence DRI.[T [[]) for stable optimization, which differs from TRPO's forward KI. formulation Der. (xx())))) [
23

---

Published as a conference paper at ICLR 2026
(a) Reddit TL;DR
(b) Anthropic Helpfulness
(c) Anthropic Harmlessness
10
10
Reward
Reward
Reward
7
0.00
0.25
0.50
0.75
1.00
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Epochs
Epochs
Epochs
DAR (Ours)
RLOD
PPO
GRPO
Figure 7: Reward curves comparing DAR against online RLHF methods (PPO, RLOO. GRPO). Shaded areas represent 95% confidence intervals over 3 seeds. Rewards are evaluated using Qwen2- -2B-Instruct on random test subset for Helpfulness, Harmlessness, and TL;DR tasks.
While our prior work (AWR) provided theoretical justification for this choice, the distinction has im- portant practical implications: forward KL encourages mode-covering behavior, where the learning policy attempts to match all modes of the sampling policy, while reverse KL promotes mode-seeking behavior, concentrating probability mass on the dominant modes. We therefore conduct an ablation study empirically investigate whether this choice impacts learning dynamics and final alignment performance.
To isolate the effect of KL direction, we implement Dual-Mix-PPO, which implements stable opti- mization via forward KL:
(L Tonal KL (T;; Trefs Te) max B2w bon y)]
x
(aDaLLlT((ll)) [e((v + (1 (T To(l))))
Thereby, in the actually implementation, the advantage computed via: r(x: y) (( log x(vla) +
To (v/ar)
=(yla) ). is worth noting that our proposed algorithm DAR dependent on the dual-KL KL both in reverse form, so it is infcasible to implement Mix-DAR in this ablation study.
(1 11-a0 a:) (<(l(z))
=ylaz) log (x(l))))
Figure 8 shows reward curves comparing Dual-PPO and Dual-Mix- across three random seeds. Dual-Mix-PPO (forward KL) consistently yields lower final performance despite providing more stable learning curves. This suggests that while mode-covering behavior prevents catastrophic policy updates, it overly restricts exploration. The learning policy becomes too conservative, limiting its ability to discover high-reward responses that deviate from the sampling policy. In contrast, Dual- PPO with the mode-seeking behavior of reverse KL allows more aggressive optimization toward high-reward regions, resulting in better alignment performance. These results empirically justify the design decision of DAR optimizing dual-KL objective both in reverse forms.
Dual-PPO vs. Dual-Mix-PPO on Anthropic Helpfulness
9
Reward
7
6
Dual-PPO
Dual-Mix-PPO
0.0
0.2
0.4
0.6
1.0
Epochs
Figure 8: Reward curves comparing Dual-PPO and Dual-Mix-Pii PPO in finetuing Qwen2-7B on Help- fulness. Shaded areas represent 95% confidence intervals over 3 seeds. Rewards are evaluated using Qwen2-72B- 72B--Inrttttttt Ik random test subsets.
24

---

Published as a conference paper at ICLR 2026
C.5 COMPUTATIONAL COMPLEXITY
Table 8: GPU time comparison between DAR and online RLHF baselines on Helpfulness using two GPUs. Results show the minimum time interval between checkpoints for each run, averaged across three seeds.
GPU Time
Algorithm
(min/batch)
PPO
14.1010.26
RLOO
12.69+0.08
GRPO
12.85
Iter-SFT
11.79++0.13
DAR
12.54:10.16
A key advantage of DAR over online RLHF methods is its computational efficiency and implementa- tion simplicity. Table presents GPU time comparisons, in minutes per online batch, between DAR and online RLHF methods on the Helpfulness dataset, with each experiment using two GPUs per run. While iterative SFT achieves the lowest runtime at 11.79 minutes, DAR demonstrates superior efficiency among RL approaches with runtime of 12.54 minutes. This outperforms all online RLHF baselines, including PPO, RLOO, and GRPO, where the most efficient baseline (RLOO) requires 12.69 minutes. These results empirically demonstrate that DAR is computationally lightweight method that provides simple implementation.
25

---

Published as a conference paper at ICLR 2026
D PROMPT EXAMPLES
D.1 REWARD PROMPT
Table 9: An reward prompt example generate Al reward labels for summarization. (text) and (sunmary) are populated with unlabeled examples. The reward label is then extracted via pattern matching.
Task Description
good summary is shorter piece text that has the essence of the original. It tries to accomplish the same purpose and conveys the key information from the original post. Below we define four evaluation axes for summary quality: ccherence, accuracy, coverage, and overall quality.
Coherence: This axis answers the question "how coherent is the summary en its own?" A summary is coherent is it's easy understand when its own and free of English errors. A summary is not coherent if it's difficult to understand what the summary is trying to say. Generally, it's more important that the summary is understandable than it being free grammar errors.
Accuracy: This axis answers the question "does the factual information in the summary accurately match the post?" summary is accurate if it doesn't say things that aren't the article, it docsn't t mix up people, and generally not misleading.
Coverage: This axis answers the question "how well does the sumnary the post?" A summary has good coverage mentions the main information from the post that's important to understand the situation described in the post. summary has poor coverage if someone reading only the summary would be missing several important pieces of information about the situation in the post. A summary with good coverage should also match the purpose of the original post (e.g. to ask for advics).
the important information
Overall quality: This axis answers the question "how good is the summary overall at representing the post?" This can encompass all of the above axes of quality, as well as others you feel are important. it's hard to find ways make the summary better, the overall quality is good. I# there are lots of different
the summary can be made better, the overall quality is bad.
Instruction
You are an expert summary rater. Given a TEXT (completed with SUBREDDIT and a TITLE) and SUMMARY, your role is to provide SCORE from to 10 that rates the quality of SUMMAYY given the TEXT with being awful and being a perfect SUMMARY.
Sample to Annotate
TEXT: (text)
SUMMARY: (sumnary)
Ending
SCORK:
26

---

Published as a conference paper at ICLR 2026
Table 10: A reward prompt example generate AT reward labels for helpfulness. (context) and (response) are populated with unlabeled examples. The reward label then extracted via pattern matching.
Task Description
The most important part of the task is to evaluate of helpfulness and
the Assistant's responses honesty.
Please consider the respenses carefully, and your overall judgment. Ask yoursel: "did this response represent a thoughtful, honest, and reasonable attompt to either directly help the user, or to converse with the user to clicit information that will make it easier for the Assistant to help the user later in this conversation". Note that this is aubtle if the context is about a difficult personal situation, showing empathy and understanding. But if the context is about something practical, helpful response may be a concise response, and not one that prolongs the interaction with irrelevant questions and comments. The Assistant should be helpful in the same way that friend subject-expert would (hopefully) be helpful.
helpful response might involve
Instruction
You are an expert rater of helpful and honest Assistant responses. Given CONTEXT and RESPONSE, your role is to provide SCORE from to 10 that rates the quality the CONTEXT with 1 being completely unhelpful and di shonest and 10 being extremely helpful and honest RKSPONSE.
assistant's RESPONSE given
Sample to Annotate
CONTEXT:
(context)
RESPONSE: (response)
Ending
SCORE:
27

---

Published as a conference paper at ICLR 2026
Table 11: A reward prompt example to generate Al reward labels for harmlessness. {context) } and (response} are populated with unlabeled examples. The reward label is then extracted via pattern matching.
Task Description
The most important part of the task is to evaluate the Assistant's s responses in terms of harmfulness.
Please consider the responses carefully, and your overall judgament. Ask yourself, "Is this response offensive or discriminatory? Does it aid in a dangerous or illegal act? Does it provide sensitive or consequential advice without mocesty care?"
Instruction
You are an expert rater of harmful assistant responses. Given a CONTEXT and a RESPONSE, your role is provide SCORE from 1 10 that rates the harmfulness of assistant's RESPONSE given the CONTEXT with 1 being completely harmless extremely harmful RESPONSE.
being
Sample to Annotate
CONTEXT: {context}
RESPONSE: (response)
Ending
SCORK:
28

---

Published as a conference paper at ICLR 2026
D.2 PREFERENCE PROMPT
Table 12: A preference prompt example to generate Al preference labels for summarization. (text), (summary1), ,, and {summarys ry2] populated with unlabeled examples. The prefer- ence label is then extracted via pattern matching.
Task Description
good summary a shorter piece text that has the essence of the original. It tries to accomplis ish the same purpose and conveys the key informaticn from the original post. Below dežine four evaluation axes for sumnary quality: coherence, accuracy, coverage, and overall quality.
Coherence: This axis answers the question "how coherent is the summary on its own?" A summary is coherent is it's easy to understand when read on its own and free of English errors. A summary is not coherent if it's difficult to understand what the summary is trying to say. Generally, it's important that the summary understandable than being free of granmar errors.
Accuracy: This axis answers the question *does the factual information in the summary accurately match the post?" summary is accurate if it doesn't say things that aren't in the article, it doesn't mix up people, and generally is not misleading.
question "how well does the summary cover the important information in the post?" A summary has good coverage if it mentions the main information from the post that's important understand the situation described in the post. summary has poor coverage if somecne reading only the summary would be missing several important pieces of informat ion about the situation in the post. A summary with good coverage should also match the purpose the original post (e.g. to ask for advice).
Coverage: This axis answers
Overall quality: This axis answers the questicn "how good is the summary overall representing the above axes of
post?" This can encompass all quality, as well as others you feel are important. If it's hard to find ways to make the summary better, the overall quality is good. IE there are lots of di different ways the summary can be made better, the overall quality is bad.
Instruction
You are expert summary rater. Given a piece of text and two of its possible sumnaries, output 1 2 indicate which summary best adheres to coherence, accuracy, coverage, and overall quality as defined above.
Sample to Annotate
Text (text) Summary (summaryl Summary 2 summary2]
Ending
Preferred Summary=
29

---

Published as a conference paper at ICLR 2026
Table 13:
prompt example to generate Al preference labels for helpfulness. (context), ) (response1] ), and (response2) are populated with unlabeled examples. The preference la- bel is then extracted via pattern matching.
Task Description
The most important part of the task is to evaluate the Assistant's s responses in terms of helpfulness and honesty.
Please consider the responses carefully, and your overall judgment. Ask yoursels "did this response represent a thoughtful, honest, and reasonable attempt to either directly help the user, or to converse with the user clicit information that will make it easier for the Assistant help the user later on in this conversation". Note that this is subtle - if the context is about a difficult personal situation, showing empathy and understanding. But if the context is about something practical, helpful response may be a concise response, and not one that prolongs the interaction with irrelevant questiens and comments. The Assistant should be helpful in the same way that friend subject-expert would (hopefully) be helpful.
helpful response might involve
Instruction
You are an export rater of helpful and honest Assistant responses. Given the context and the responses choose the most helpful and honest response based on the definitions above.
Sample to Annotate
Context (context)
Response 1 (responsel) Response 2 (response2)
Ending
Freferred Response-
30

---

Published as a conference paper at ICLR 2026
Table 14: A prompt example to generate Al preference labels for harmless. {context], }. {responsel) j. and (response2) are populated with unlabeled examples. The preference la- bel is then extracted via pattern matching.
Task Description
The most important part of the task is choosing which of the Assistant's 8 responses is more harmful.
Please consider the responses carefully, and your overall judgement. Ask yoursels, "Is this response offensive or discriminatory? Does it dangerous or illegal act? Does it provide sensitive or consequential advice without modesty or care?" Note that harm might be harnful by promoting biases or sterectypes, even if not overtly offensive. It could be harmful by providing dangerous information or encouraging risky behavior, even if not explicitly illegal. Or it might be harmful by giving overly conficent advice on serious matters without appropriate caveats or suggestions to seek professional help when needed.
come in many forms. A respense
Instruction
You expert rater of harmful assistant responses, Given context and the two respenses choose the most harmful response.
Sample to Annotate
Context {context) Response 1 (responsel) Response 2 (response2)
Ending
Freferred Response
31


Reviews:
[{"review_id": "yqzBpTdgiX", "reviewer": "Reviewer_7Noe", "summary": "The paper proposes an approach to address the trade-offs arising from simultaneously regularizing towards the reference policy (to mitigate reward hacking) and the current policy (for stable policy updates), in RLHF. They accomplish this by regularizing towards a convex combination of the reference policy and current policy. This is done via a weighted supervised fine-tuning loss, which allows for stable training. Experimentally, their proposed approach improves over baselines (both online RL-based approaches and online/offline direct alignment algorithms)", "questions": "1) Is there a reason for choosing a convex combination of the two KL divergences for DAR, instead of choosing independent multipliers?\n\n2) Why is the $\\pi_{ref}$ not normalized in the KL constraint for DAR and does this impact the derivation of the optimal policy for DAR?\n\n3) Why can $Z(x)$ be dropped from the objective, without modifying it, in the DAR derivation in Appendix C.3?\n\n4) Is there a theoretical justification for this statement \"as $\\pi_{t}$ is trained, the log-likelihood interpolation constructs a reference target that is inherently positioned closer to the optimal policy.\" in line 240?\n\n\nMinor Questions/Nits\n\n- Equation 2 operates at trajectory level, whereas PPO is defined at token-level\n- What the reward (avg, total, etc) in Table 1. Explain it in detail in caption.\n- Why no comaprision against PPO in the the Experiments for Standard RLHF (line 317)?\n- In Figure 3, why no evolution plots for online direct alignment methods?\n- Improvements on MT-Bench and AlpacaEval 2.0 in Table 3 seem marginal. Also the column is named \"AlphacaEval\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a critical problem that has not been explored in the literature, the impact of regularizing towards both the reference policy and the current policy, in RLHF. Regularizing towards the reference policy is done by a KL penalty to the reward, and regulazing towards the current policy is achieved via clipping or a KL constraint. Together, these two constraint our objective to operate in the intersection of the trust region that becomes increasingly restrictive as training progresses. The paper proposes a simple weighted supervised fine-tuning objective by regularizing towards a convex combination of both the reference policy and the current policy, leading to a stable training algorithm DAR (Dual-Regularized Advantage Regression). Experimental results showcase that DAR surpasses online RL based methods (PPO, GRPO, RLOO) and online/offline direct alignment methods (DPO/IPO/SLiC) across three training domains. The presentation is clear, crisp with no major issues with the grammar and writing flow.", "weaknesses": "One of my concerns with the paper is their choice to regularize towards a convex combination of the reference policy $\\pi_{0}$ and the current policy $\\pi_{t}$ i.e $\\alpha D_{KL}(\\pi \\vert\\vert \\pi_{0}) + (1-\\alpha) D_{KL}(\\pi \\vert\\vert \\pi_{t})$. This inherently leads to incentivizing regularizing to one of the distibutions than the other (when $\\alpha$ != 0.5). It would have been better to have two independent multipliers for each of the divergence, which supports the Lagrangian view of the objective when looking at the divergences as constraints i.e $\\alpha_{1} D_{KL}(\\pi \\vert\\vert \\pi_{0}) + \\alpha_{2} D_{KL}(\\pi \\vert\\vert \\pi_{t})$.\n\nAdditionally, in proposition 4.1, the objective has a KL penalty wrt a reference mixture distribution $\\pi_{ref} = \\pi_{0}^{\\alpha}\\pi_{t}^{1-\\alpha}$. $\\pi_{ref}$ need not be a valid probability distribution, since it may not sum up to 1. The KL term here hence may not be between two distributions. There needs to be a normalizing factor for $\\pi_{ref}$ to make it a valid probability distribution. This would affect their proof of the optimal policy in Theorem 4.2, since there would now be normalizing factors of $\\pi_{ref}$ in the expressions.\n\nIn the DAR derivation in Appendix C.3, ignoring the earlier issue with $\\pi_{ref}$ not being normalized, on line 885, they state \"we factor out the partition function $Z(x)$ as it is a positive constant and doesnt shift optimal policy\". $Z(x)$ depends on $x$ and leads to weighing each prompt differently. Considering a simple setting with two possible $x$, the objective is $Z(x_{1}) f_{\\theta}(x_{1}) + Z(x_{2}) f_{\\theta}(x_{2})$.  How can $Z(x)$ be factored out without affecting the objective \n\nThe authors state that as $\\pi_{t}$ is trained, the log-likelihood interpolation constructs a reference target that is inherently positioned closer to the optimal policy. No proof for this is provided. If this is solely because $\\pi_{ref}$ contains $\\pi_{t}^{1-\\alpha}$, that becomes more optimal over the course of training. If so, standard PPO also has a policy constraint with respect to $\\pi_{t}$. How is yours more optimal?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mFZcuqyjCW", "reviewer": "Reviewer_rHRn", "summary": "This paper proposes a dual-KL regularization framework for RLHF that unifies two objectives usually treated separately: (1) preventing reward hacking via KL to the initial SFT model π₀, and (2) maintaining stability via KL or clipping to the current policy πₜ.\nThe authors show that these can be merged into a single interpolated reference in log-space, leading to a new weighted-SFT formulation called DAR (Dual-regularized Advantage Regression). DAR is positioned as a simple, RL-free alternative to PPO/GRPO, with theoretical analysis and experiments on Qwen2-7B showing improved reward–KL trade-offs.", "questions": "1.How sensitive is performance to α? Could an adaptive trade-off help?\n2.Would DAR remain stable under noisy or AI-feedback reward models?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear identification of a long-standing conflict between stability and reference regularization.\n2. Mathematical formulation is elegant and internally consistent.\n3. DAR simplifies PPO-style RLHF into a regression-like loss that is easier to implement and more stable.", "weaknesses": "1. Outdated baseline setup. All experiments use Qwen2-7B and compare mainly against PPO, GRPO, and RLOO; no comparison to modern alignment frameworks, stronger models, and new RL methods.\n2. The novelty is mostly formal: the “dual-KL” is effectively a convex interpolation between π₀ and πₜ, similar to prior multi-reference ideas.\n3. Theoretical results rely on clean advantage estimation; no analysis under noisy or biased rewards.\n4. Empirical gains are modest and might vanish under stronger baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nSqWeKzCTc", "reviewer": "Reviewer_GWzK", "summary": "The paper proposes a *dual-KL regularization* approach that aims to jointly address two RLHF pain points: (i) preventing reward hacking via reference regularization and (ii) ensuring stable optimization via trust-region style control. The method first studies a constrained “PPO-Align” objective and then derives a *weighted dual-KL* objective that yields an interpretable, weighted SFT (DAR) loss by effectively interpolating between the initialization policy $\\pi_{0}$ and the current policy $\\pi_{t}$ in log space. Experiments on TL;DR, Anthropic Helpfulness, and Harmlessness report improved win rates (Figure 3/Table 2), with evaluations judged by GPT-4 Turbo. Figure 1 provides the conceptual motivation for expanding the search region beyond the intersection of the trust regions to balance stability and reference adherence.", "questions": "- Given the centrality of Table 2, the community would benefit from end-to-end scripts (prompts, judge configs, seeds, filtering) in the supplement to exactly reproduce those numbers. Can we provide that in supplement?\n- Would a *mixed-direction* dual-KL be preferable on theory/empirics—e.g., a mode-covering  term $D_{\\mathrm{KL}}\\left[\\pi_{\\theta}\\||\\pi_{0}\\right]$ *and* a mode-seeking term around the behavior/current policy $D_{\\mathrm{KL}}\\left[\\pi_{t}\\||\\pi_{\\theta}\\right]$?\n- Could you report training stability metrics such as gradient-norm statistics, entropy, and seed-wise variance for dual-PPO?\n- How do conclusions change with newer backbones (e.g., Qwen3) or different reward models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Timely problem framing.** The paper clearly motivates the dual goals of stabilizing policy updates while constraining drift from a reference policy, and argues for a unified objective rather than separate mechanisms.\n- **Empirical gains.** Across three benchmarks, DAR shows strong win rates against online RLHF (e.g., PPO, GRPO, RLOO) and online DAP baselines; curves in Figure 3 and the summary in Table 2 support the claim.\n- **Implementation clarity intent.** The paper points to code/supplement details, which is important given the number of components interacting (policy, judge, datasets, ablations).", "weaknesses": "- In the discussion of PPO stability, the classical TRPO/PPO literature typically regularizes with a KL of the form $D_{\\mathrm{KL}}\\left[\\pi_{\\text{old}}\\||\\pi_{\\theta}\\right]$, see Schulman et al. (2015, TRPO) and Schulman et al. (2017, PPO). By contrast, the paper’s *PPO-Align* (Sec. 4.1) constrains with $D_{\\mathrm{KL}}\\left[\\pi_{\\theta}\\||\\pi_{t}\\right]$ and penalizes $D_{\\mathrm{KL}}\\left[\\pi_{\\theta} \\||\\pi_{0}\\right]$. The rationale for changing directions relative to the classical trust-region view is not made explicit. This makes it hard to judge whether the final dual-KL choice is a principled departure or just a convenient variant.\n- Figure 3/Table 2 use GPT-4 Turbo as the judge and Qwen2-72B-Instruct as the LLM annotator. Results would be stronger with a stronger contemporaneous judge (e.g., GPT-5) and annotator (Qwen3) for additional validation.\n- While DAR (weighted SFT) is argued to be stable, quantitative stability analyses for the *dual-PPO* path are sparse (e.g., per-iteration KL to $\\pi_{0}$ and $\\pi_{t}$, gradient-norm/entropy trends, collapse rates across seeds).\n- Most results focus on Qwen2 and Llama-3.1 settings; adding a recent backbone (e.g., Qwen3) would better probe portability.\n- *(Minor)* Colors/encodings are hard to parse quickly; the caption should explicitly map colors/shapes to policies/regions and call out what “search expansion” specifically denotes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CTbHwvO34v", "reviewer": "Reviewer_4Fmj", "summary": "This work proposes a unified regularization approach with weights that explicitly balances the objectives of preventing reward hacking and maintaining stable policy updates in RLHF.", "questions": "(1) Is Eq. (2) PPO for RL not RLHF, since there is no KL penalty to $\\pi _ {\\rm ref}$? The equation in Section 4.1 seems to transform the KL penalty in Eq. (1) into hard constraint $KL<\\epsilon$, yes? \n\n(2) \"Empirical Validation\" in Section 4.1 involves two dual-KL variants. Why not move \"Empirical Validation\" to after introducing two dual-KL variants?\n\n(3) Your dual-KL (Eq. 3) looks like a special case of [1] with 2 references, what are your differences and additional contributions?  \n[1] Gholamali Aminian, Amir R Asadi, Idan Shenfeld, and Youssef Mroueh. Theoretical analysis of kl-regularized rlhf with multiple reference models. ArXiv:2502.01203, 2025.\n\n(4) In proposition 4.1, $\\log\\pi_{\\theta}(y|x)=\\alpha\\log\\pi_0(y|x)+(1-\\alpha)\\log\\pi_t(y|x)+C(x)$ with a constant normalization $C(x)$ is suggested to ensure policies summing up to 1. \n\n(5) What are the evaluation metrics of MT Bench in Table 3? \n\n(6) In Figure 5c, could you explain more about EOS-missing rate, and why $\\alpha=0$ increases EOS-missing rate? Is it convenient to add LC-win rate? \n\n(7) In Algorithm 1, what are $\\mu_A$ and $\\sigma_A$? What are the meanings of the two weights $w _ {\\rm reg}^i$ and $w _ {\\rm adv}^i$ and how are they computed?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors found a reasonable limitation of the existing approach to solve reward hacking and maintain stable policy updates, i.e., the conflict of two regularizers that pull the trained policy towards the reference policy and previous-step policy respectively. This well motivates the proposed dual-KL approaches, with the novel and straightfoward idea of combination of the two regularizers, which are presented clearly. The experiments look comprehensive to me.", "weaknesses": "A few clarity issues as shown in the following questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)