KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity
Authors: Gholamali Aminian
Amir R. Asadi
Idan Shenfeld
Youssef Mroueh
Venue: NeurIPS 2025 poster ()
Keywords: Alignment
multiple reference model
RLHF
Sample complexity
Primary area: reinforcement_learning
TL;DR: We provide theoretical analysis for forward and reverse KL-regularized RLHF under multiple reference models.

Abstract: Recent methods for aligning large language models (LLMs) with human feedback predominantly rely on a single reference model, which limits diversity, model overfitting, and underutilizes the wide range of available pre-trained models. Incorporating multiple reference models has the potential to address these limitations by broadening perspectives, reducing bias, and leveraging the strengths of diverse open-source LLMs. However, integrating multiple reference models into reinforcement learning with human feedback (RLHF) frameworks poses significant theoretical challenges, where achieving exact solutions has remained an open problem. This paper presents the first \emph{exact solution} to the multiple reference model problem in reverse KL-regularized RLHF. We introduce a comprehensive theoretical framework that includes rigorous statistical analysis and provides sample complexity guarantees. Additionally, we extend our analysis to forward KL-regularized RLHF, offering new insights into sample complexity requirements in multiple reference scenarios. Our contributions lay the foundation for more advanced and adaptable LLM alignment techniques, enabling the effective use of multiple reference models. This work paves the way for developing alignment frameworks that are both theoretically sound and better suited to the challenges of modern AI ecosystems.

KL- KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity
Gholamali Aminian' The Alan Turing Institute
Amir R. Asadi Statistical Laboratory
London, UK gaminian@turing ac. uk
University of Cambridge, UK asadi @statslab.. cam ac uk
Idan Shenfeld Massachusetts Institute Technology USA ddanshen@mit... edu
Youssef Mroueh
IBM Research USA mroueh@us. ibm. com
Abstract
Recent methods for aligning large language models (LLMs) with human feedback predominantly rely single reference model, which limits diversity, model overfitting, and underutilizes the wide range of available pre-trained models. Incor- porating multiple reference models has the potential address these limitations by broadening perspectives, reducing bias, and leveraging the strengths diverse open-source LLMs. However, integrating multiple reference models into rein- forcement learning with human feedback (RLHF) frameworks poses significant theoretical challenges, where achieving exact solutions has remained open problem. This paper presents the first exact solution to the multiple reference model problem in reverse KL-regularized RLHF. We introduce comprehen- sive theoretical framework that includes rigorous statistical analysis and provides sample complexity guarantees. Additionally, we extend our analysis forward KL-regularized RLHF, offering new insights into sample complexity requirements multiple reference scenarios. Our contributions lay the foundation for more advanced and adaptable LLM alignment techniques, enabling the effective use of multiple reference models. This work paves the way for developing alignment frameworks that are both theoretically sound and better suited to the challenges of modern ecosystems.
1 Introduction
Large language models (LLMs) have revolutionized natural language processing (NLP) by demon- strating remarkable capabilities in understanding and generating human language. Powered by vast datasets and advanced neural architectures, these models have set new benchmarks across various NLP tasks, including machine translation and conversational agents. Despite these advancements, aligning LLMs with human values and preferences remains critical challenge. Such misalignment can lead to undesirable behaviors, including the generation of biased or inappropriate content, which undermines the reliability and safety of these models [Gehman et al., 2020].
Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal framework for addressing alignment challenges in LLMs. By fine-tuning LLMs based on human feedback, RLHF steers models towards more human-aligned behaviors, enhancing truthfulness, helpfulness, and
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

harmlessness while maintaining their ability to generate accurate and high-probabilliy outputs [Wirth al., 2017, Christiano et al., 2017]. In RLHF, heward-based methods use a trained reward model to evaluate (prompt, response) pairs. These methods treat the language model as policy that takes prompt x and generates a response y conditioned on x; optimizing this policy to generate responses with maximum reward. Typically, reference policy (usually the pretrained model before fine-tuning) is used as a baseline to regularize training, preventing excessive deviation from the original behavior.
An inherent limitation of most works on LLM alignment is their reliance on a single reference model [Wang et al.. 2024b]. First, this restricts the diversity of linguistic patterns and inductive biases available during training. In that, it is over restrictive potentially leading to model that inherits the limitations or cultural biases of a single pretrained source. Second, such an approach is inefficient in utilizing the wealth of pre-trate pre-trained models available in modern AI ecosystems, which excel in different domains and capture unique nuances, leaving valuable collective intelligence untapped. Therefore, incorporating multiple LLMs reference models produces a model that reflects the characteristics of all reference models while satisfying human preferences. This approach is particularly relevant as the open-source community continues to release diverse pre-trained and fine-tuned LLMs of varying scales, trained on a wide range of datasets [Jiang et al., 2023, Penedo et al., 2023].
A solution is to extend the RLHF training to utilize multiple reference models. While RLHF with multiple reference models has demonstrated practical utility [Le et al., 2024], its theoretical underpinnings remain largely unexplored. A critical gap in current understanding is the lack of an exact solution for reverse KL-regularized RLHF when incorporating multiple reference models. This theoretical limitation has prevented the study of sample complexity of bounds on both optimality and sub-optimality gaps in the reverse KL-regularized framework. Addressing this problem crucial for advancing the alignment of LLMs with human preferences in increasingly complex and diverse settings.
In this work, we provide the solutions for RLHF with multiple reference models when regularized via Reverse KL divergence (RKL) forward KL divergence (FKL). In addition, we provide a statistical analysis of these scenarios. Our main contributions are as follows:
We propose a comprehensive mathematical framework for reverse KL-regularized RLHF with multiple reference models and provide the exact solution for this problem and calculate the maximum objective value.
We provide theoretical guarantees for the proposed multiple reference models scenario under reverse KL-regularization. In particular, we study the sample complexity of reverse KL-regularized RLHF under multiple reference models.
We also study the multiple reference models scenario under forward KL-regularized RLHF and analyze its sample complexity.
2 Related Works
Multiple References: Inspired by model soups [Wortsman et al., 2022], Chegini et al. [2024] propose reference soup policy, achieved by averaging two independently trained supervised fine- tuned models, including the reference model. However, their approach lacks theoretical guarantees, particularly regarding applicability to alignment tasks. More recently, Le al. [2024] introduced the concept of multiple reference models for alignment. Due to the challenges in deriving a closed- form solution for the RLHF objective under multiple refereneing constraints, the authors proposed a lower-bound approximation. In this work, we address this gap by deriving the closed-form solution for the multiple reference model scenario under reverse KL.-regularization
Several works have studied the theoretical underpinnings of reverse re-rellladd RLHF, particularly in terms of sample complexity [Zhao al., 2024, Xiong et al., 2024, Song et al., 2024, Zhan et al., 2023, Ye et al., 2024]. Among these, Zhao et al. [2024] analyze reverse KL-regularized RLHF, demonstrating the effect of reverse KL-regularization and establishing an upper bound on sub-optimality gap with (L1/n) sample complexity (convergence rate) where represents the size of preference dataset. More detailed comparison with these works provided in Section 7. However, to the best of our knowledge, the RLHF framework incorporating multiple reference models has not yet been studied.
Theoretical Foundation of RLHF:
'The sample complexity provides insight into how quickly bounds converge the dataset size increases.

---

Forward KL-regularization and Alignment: The forward KL-regularization for Direct Preference Optimization (DPO) proposed by Wang et al. [2024a]. The application of forward KL-regularization for alignment from demonstrations is shown in [Sun and van der Schaar, 2024]. The forward KI.- regularization in stochastic decision problems also studied by Cohen [2017]. To the best of our knowledge, the forward Kl.rregularized RLHF is not studied from a theoretical perspective.
3 Preliminaries
Upper-case letters denote random variables (e-g., 7), lower-case letters denote the realizations of random variables (e.g., 2), and calligraphic letters denote sets (e.g., 2). All logarithms are in the natural base. The set of probability distributions (measures) over space X' with finite variance is denoted by P(X). The KL-divergence between two probability distributions on IR with densities p(x) and q(x), so that ((x) > 0 when p(x) > 0, is (L(p)l4) := JR p(x) log(p(z)/q(at) (a(x)) (with 0/0 = 0). The entropy of a distribution p(z) is denoted by (p) = - SRa p(x) log(p(z)).
Notations:
We define the Escort and Generalized Escort distributions [Bercher, 2012] (a.k.. normalized geometric transformation).
Definition 3.1 (Escort and Generalized Escort Distributions). Given discrete probability measure P defined on set A, and any À 2 0, we define the escort distribution (P)* for all A as
(P) = (P(a)) (PEA
Given two discrete probability measures P and Q defined on a set A, and any X E [0, 1}, we define the generalized escort distribution (P, x)* as the following tilted distribution:
P*(a)Qt>Ma)
(P) = ΣZEA PA(x)
Next, we introduce the functional derivative, see Cardaliaguet et al. [2019].
Definition 3.2. [Cardaliaguet et al., 2019] A functional U P(R") + admits a functional
derivative there is a map 6t P(R") R" RR which continuous P(R")) and, for all m,m' )( it holds that
U(m') Um 1 =f-² [
8U (ma, a) (m' - (dd dA,
5m
where mA m 1(m' m).
We also define the sensitivity policy Ty (r(()x), which is a function of reward function r(x, y). with respect to the reward function
(r) lim Ar
(1)
Ör
Ar
4 Problem Formulation
Following prior works [Ye et al., 2024, Zhao et al., 2024], we consider the problem of aligning policy with human preferences. Given input (prompt) the finite space of input texts, the policy E ПI, where the set of policies, models a conditional probability distribution (yy|z) over the finite space of output texts y y. From given n and we (-12).
X' which is samples from p(x), is
can sample an output (response)
Preference Dataset: Preference data is generated by sampling two outputs (y, y/)[z from nhe the reference policy (model), and presenting them to an agent, typically a human, for rating to indicate which one is preferred. For example, y > y' denotes that is preferred to y'. A preference dataset is then denoted as = {v," 3/. x 1n 1==1= where n the number of data points, y., and denote the preferred (chosen) and dispreferred (rejected) outputs, respectively.
We assume that there exists true model of the agent's preference p"(y > y (r)). which assigns the probability of y being preferred to y' given based on the latent reward model which is unknown.

---

4.1 RLHF from One Reference Model
Using the dataset our goal is to find policy that maximizes the expected preference while being close to a reference policy Tref- In this approach, Bradley- Terry model Bradley and Terry [1952] is employed as the preference model, p3 > y [>) (ro (r y) (z.23''))
where a denotes the sigmoid function and rg : X' x - is reward model parameterized by 0, which assigns scalar score to indicate the suitability of output y for input In Christiano al. [2017], the reward model is trained on D to maximize the log-likelihood (MLE) estimator:
n
CR(0, = Σ log (ro (22, (l)) re re( v()).
Given trained reward model rg(2, y) where 0 arg maxage (n D), consider the regularized optimization objective which is regularized via reverse KL-regularized or forward KL- regularized.
Reverse KL-regularized RLHF: crucial component RLHF is the use of reference model to compute a Reverse Kullback-Leibler (KL) divergence penalty. This penalty ensures that the process does not deviate excessively from the original model, mitigating the risk generating nonsensical responses [Ziegler et al., 2019]. The reverse KL-regularized optimization objective for (7 > 0) can represented as:
maxEy [2, - KL (T(([)))||=
(3)
7
Note that the solution of (3) is,
<0(x)) Tree(y) (ylx)) exp(1ra(x, y))
(xx
where Z(x) = Ey Ey (e) [exp(<ra(2; Y)) is the normalization factor. Similarly, we can define
Tp- (y(zz) using ra- (x, y) instead of (z ( y) in (4). This RLHF objective is employed to train LLMs such Instruct-GPT Ouyang al. [2022] using PPO Schulman al. [2017].
We define J(Ta- (-1=)) = Ey (1o)) (x. Y)] (a.ka. value function") and provide an upper bound on optimal gap,
J (Tx) (T(x)) (t)) (1z)) (TT
(5)
Furthermore, inspired by [Song et al., 2024, Zhao et al., 2024), we consider the following RL.F objective function based on the true reward function,
Jy (Tror ( (-Ix), (-(=))) = By Ey (Y,x)] 1 (e)
(6)
As studied by Zhao al. [2024], Song et [2024], Zhan al. [2023], we also aim to study the following sub-optimality gap,
[ (.12), T (+xz) = (m - (-1x)) (x)
(-(+z))
(7)
Forward KL-regularized RLHF:
Inspired by [Wang et al., 2024a], can consider the forward KL-regularized optimization objective as,
1 y max [ra(x: r)] KL (TT)
(8)
As discussed in [Wang et al., 2024a], this optimization problem has an implicit solution given by:
*g(Ul=)= = <
v))
'We can also consider ([6x)())(((((((X))) All of our results also holds for expected version of value function.

---

where (z)z) is normalization constant ensuring that ʃy (1) ( 1. Some properties of Zz(x) are discussed App. E. Similar to (5) and (7), for forward KL-regularized RLHF, we can define,
(- (-)))) = Ey (x) (n) (Y,z)) KL ( [T
(10)
(12 5] (+)) == ( (-1z)) - J, (Tret (122), (-)()=
(11)
4.2 Assumptions
For our analysis, the following assumptions are needed. Assumption 4.1 (Bounded Reward). We assume that the true and parametrized reward functions, ro- (I, v) and (., y), are non-negative functions and bounded by Rmax-
Assumption 4.2 (Finite Class). We assume that the reward function class, R, finite, [R] < oc.
The assumption of bounded rewards (Assumption 4.1) and Finite class (Assumption 4.2) are common in the literature [Song et al., 2024, Zhan al., 2023, Zhao et al., 2024, Chang et al., 2024, Xiong et al., 2024]. More discussion regarding these assumptions are provided in App. C.
Coverage conditions play fundamental role in understanding the theoretical guarantees of RLHF algorithms. We first introduce the most stringent coverage requirement, known as global coverage [Munos and Szepesvári, 2008]:
Assumption 4.3 (Global Coverage). For all policies π, we require maxz,y=o()>00 Ree (v]2) S CGo.
(yla)
where सeर denotes the reference model and CGC IR." is a finite constant.
A key implication of Assumption 4.3 is that it requires substantial coverage: specifically, for any prompt v and token sequence y in the support p. we must have (y)r()
Cac
While global coverage has been extensively studied in the offline RL literature [Uehara and Sun, 2021, Zhan et al., 2022], it imposes strong requirements that may be unnecessarily restrictive for RLHF. A key insight from recent work [Zhao et al., 2024, Song et al., 2024] is that RLHF algorithms inherently employ reverse KL-regularization, which ensures learned policies remain within neighborhood of the reference model. This observation motivates a more refined coverage condition:
Assumption 4.4 (Local Reverse KL-ball Coverage). Consider Erkl *(ylw) (x)()))))
and any policy satisfying
Eg (T. (: (x:) (e S Erkl, we require max z,=())=()))> depends on the KL threshold Ezkl-
S Ceaas where Ce R."
Similar to Assumption 4.4, we consider the forward KL-ball coverage assumption.
Assumption 4.5 (Local Forward KL-ball Coverage). Consider Ena < cc and any policy satisfying Egza [T ret (-]==)]](( End, we require maxx,y=(()>00 ( depends on the KL threshold Ega-
((yla) < Ceaas where Cenu
R
The local reverse or forward KL-ball coverage condition offers several advantages. Focusing only on policies within a reverse KL-ball of the reference model provides sharper theoretical guarantees while imposing weaker requirements. This localization aligns naturally with RLHF algorithms, which explicitly constrain the learned policy's 's divergence from the reference model. For any fixed reference model Tref, the reverse forward KL local coverage constant is always bounded by the global coverage constant: max(Ca.., Cepu) I Cgc. This follows from the fact that KL-constrained policies form a subset of all possible policies.
5 RLHF from Multiple Reference Models via Reverse KL divergence
In this section, inspired by Le et al. [2024], we are focused on situations involving K reference policies { &ref f
i=1 where the latent reward model among all reference policies is the same. All proof details are deferred to Appendix D.
5

---

5.1 Exact Solution of RLHF under multiple reference models via RKL
Inspired by [Le al., 2024], our objective can be formulated as a multiple reference models RLHF objective,
K Oi O. (T
max
E Y~u(-1a)
(r(
Y)
(12)
where 0g are weighting coefficients for each reference policy and Σ a; = 1. This objective was explored in previous studies, leading enhancements in pure RL problems Le et al. [2022].
However, addressing this optimization problem in LLMs through reward learning and RL finetuning pose similar challenges to (3), Our goal is to derive a closed-form solution for the multi-reference RLHF objective in (12). Note that in [Le et al., 2024, Proposition 1J, a lower bound on RLHF objective in (12) is proposed, and the solution for this surrogate objective function is derived as follows.
Trof (y/1))
IL (y)z))
exp (1r(2) (2r(2.)):
(13)
7(xx
where (eer(y(
and (x) = Ey Tret (y (r) exp (ar(2r, (x))
Teaf
(v]=)
In contrast, in the following theorem, we provide the exact solution of the objective function for the multiple reference model (12).
Theorem 5.1. Consider the following objective function for RLHF with multiple reference models,
K
max E (-1((
rT- (x, Y)]
Σ
where a; - and as E f)r E [K]. Then, the exact solution of the multiple reference model objective function for RLHF is,
Ha (ylxr) (slx) 7(x)
x (v/x)
exp (yro- (I.8)).
(14)
where
I1 K
K or (v(z), vEyi=1 i=1
(arre((/zz)
Fa(x) ΣI
(15)
(x)
and ((x)
Eg ((rl()) exp (1r0-(2,))) (2,8)). The maximum objective
value is
log E, I1 (ett,((/z))) exp (,r(x,1))
Note that this result does not rely on the assumptions stated in Subsection 4.2 and in fact holds in greater generality. Using Theorem 5.1, consider the following optimization problem for the multiple reference models scenario.
Ey (re Y)]
(16)
where A. (y)æz is defined in (15) generalized escort reference policy. The algorithm of reverse KL-regularized RLHF with two reference models is presented in App. A.
5.2 Main Results for RLHF via RKL
In this section, we provide our main theoretical results for the RLHF algorithm with multiple reference models based on reverse KL-regularization. Using the convexity of reverse KL divergence, we can provide an upper bound on the sub-optimality gap. Furthermore, holds under Fa. as reference policy with Ca.cu First, we can derive the following upper bound on the sub-optimality gap of the RLHF algorithm with multiple reference models.
assume that Assumption 4.4

---

Theorem 5.2. Under Assumption 4.1, 4.2 and 4.4, the following upper bound holds on the sub- optimality gap with probability least (1 for 8 E (0, 1/2).
( ())()))))))))))))))))))))())), E aa 128 maxx R log( (g(()))6))
max
(17)
12
Using Theorem 5.2, we can provide the upper bound the optimal gap under the RLHF algorithm.
Theorem 5.3. Under Assumption 4. I, 4.2 and 4.4, there exists constant C' > such that the following upper bound holds optimality gap of reverse KL-regularized RLHF with probability at least (1 (1-6) for 5 E (0, 1/2),
log(( + C' 2
2C 2. log(|R) /8)
[TT (12), (-2(1z)) I aa 128c 12
ARmax R2
mmax
n Remark 5.4 (Sample Complexity). We can observe sample complexity of ((1/n) for the sub- optimality gap and (1/v/n) for the optimality gap from Theorem 5.2 and Theorem 5.3, respectively.
RLHF from Multiple Reference Models via Forward KL Divergence
In this section, inspired by [Wang et al., 2024a], we extend the RLHF from multiple reference models based on reverse KL-regularization Le al. [2024] forward KL-regularization. Similar, to
Section 5, we are focused on situations involving K reference policies {
(Te i=1
where the latent
reward model among all reference policies is the same. All proof details are deferred to Appendix
6.1 Solution of RLHF under multiple reference models via FKL
Inspired by [Le et al., 2024, Wang et al., 2024a], our objective can be formulated as multiple reference models RLHF objective,
max
Y~T((Z)) [r(,,
E
(18)
k where are weighting coefficients for each reference policy and Σ 3 1. This objective was explored in previous studies, leading to enhancements in pure RL problems Le et al. [2022]. However, addressing this optimization problem in LLMs through reward learning and RI. finetuning poses similar challenges to (3). Our goal is to derive closed-form solution for the multi-reference RLHF objective in (18).
We now provide the implicit solution of the RLHF with multiple references. Theorem 6.1. Consider the following objective function for RLHF with multiple reference models,
[ro- (x, x)]
3,KL (<T)
max You
(
where ΣK1 8 1 and B, E (0. 1) for [K]. Then, the implicit solution of the multiple reference models objective function for RLHF is.
TB.r (y/2
# (y/x) (2(xx) rg- (I, 3))
(19)
where 7g Etn (y)lz) and (xx) is the solution Jyey #7. (y/x) 1.ror a given X.
Using Theorem 1, we can consider the following optimization problem for forward KL-rreullrized RLHF under multiple reference model scenario.
Ey-n(11a) (z, - KL ( ([]])
(20)
where n.. (y)x) is defined Theorem 6. as weighted reference policy. The algorithm of forward KL-regularized RLHF with two reference models is presented in App. A.
7

---

Table 1: Comparison of Various Works in Theoretical Foundation of RLHF: Key features include support for RKL sub-optimality gap, RKL optimality gap, FKL sub-optimality gap, and FKL optimality gap and their Sample Complexities for each scenario.
RKL Sub-optimality Gap
RKL Optimality Gap (Sample Complexity)
Sub-optimality Gap (Sample Complexity)
FKI. Optimality Gap (Sumple Complexity)
Work
(Sample Complexity)
Song al. [2024]
O0l//))
x
Zhao al. (2024)
(11/n)
x
X
X
Chang et al. [2024]
O(1/v/)
x
X
Xiong et [2024]
00//m)
Our Work
((1/m)
00//a)
001//))
001//))
6.2
Main Results for RLHF with FKL
This section presents our core theoretical analysis of forward KL-regularized RLHF under the multiple reference model setting. We begin by leveraging KL divergence's convex properties to establish an upper bound on the sub-optimality gap. Throughout this section, we consider
0 (ulz)) ([Zg(=r)rg(<an and Tg. (y|æ) 7(Zg- (2)-ra- (m.3))) Furthermore, we assume that
(vl=)
Assumption 4.5 holds under (t(lz) reference policy with CB First, derive upper bound for the sub-optimality gap in the multiple reference forward KL-regularized RLHF setting.
Theorem 6.2. Under Assumption 4.1, 4.2 and 4.4, the following upper bound holds on the sub- optimality gap with probability at least (1 - 6) for d E 1),
log((Rl/8) n
J (1xx), #f(1))) - B. 2mmar Remax
(21)
Using Theorem 6.2, provide the upper bound on the optimal gap under the multiple reference forward KL-regularized RLHF setting.
Theorem 6.3. Under Assumption 4. 4.2 and 4.4, the following upper bound holds on optimality gap of the multiple reference forward KL-regularized RLHF algorithm with probability at least (1 5) for (0,1)
log( [R|/8) n
max (Ilog(C s.) log(7 + 1))
[ (1x), #2(1z)) (1x),)((11)) < 16Cp e2 2umax Rmnx
7
Remark 6.4 (Sample Complexity). Choosing n, we have sample complexity O(1/ vT) on optimality gap from Theorem 6.3. We can also observe the sample complexity of (L/ for the sub-optimality gap.
7 Discussion
In this section we provide theoretical comparison with single reference model in terms of sample complexity, Rmax and coverage constant. We also extend our framework to DPO. Further discussion, c.g., coverage assumption and comparison of RKL with FKL, are provided in App. G.
Our theoretical results extend to the single-reference model setting, enabling comparison with existing work in this domain. The RKL- regularized RLHF framework and its sub-optimality gap have been investigated by Song et al. [2024] and Zhao et al. [2024], who established sample complexity bounds. Song al. [2024] derived sub-optimality gap sample complexity of ((1//n), which Zhao et al. [2024] later improved to O(1/n), demonstrating the effectiveness of RKL. regularization. Note that, in [Zhao et al., 2024], is shown that when the emror tolerance is sufficiently small, the sample complexity follows an O(1/e) relationship. This corresponds to (11/n,, where n represents the dataset size. comparison with [Zhao et al., 2024], proposed approach based on functional derivative and convexity of KL
Theoretical Comparison with Single-Reference Models:

---

07
08
Cuu (P0)
MIOO
Pooeant (ivern kmed)
Chass
MERPIII
04
0.3
01
0.0
0.2
0.5
0.8
10
0.
02
0.4
0.5
0.5
1.0
a8hha
abhh
(a) Mean and 95% CI for @ performance on (b) Mcan normalized reward and 95% CI on the GSM8K using policy gradient algorithms.
UlraFeedback dataset, using offline RLHE,
Figure 1: In both online and offline RI., our analytical RKL objective outperforms both the MRPO approximation and single reference objective (a 0).
divergence. Our approach is more general and can be applied to the forward KL-regularized RLHF framework. There also some works similar algorithms to RLHF. Additionally, Chang et al. [2024] proposed an algorithm integrating offline and online preference datasets in RLHE, analyzing its optimality gap sample complexity under RKL regularization. The general reverse KI.-regularized RLHF framework under general preference models is studied by Xiong et al. [2024] and sample complexity of O(1/ (n) for sub-optimality gap is derived. To the best of knowledge, the sample complexity of the optimality gap and sub-optimality gap for forward KL-regularization have not been studied in the literature. Furthermore, in [Huang et al., 2024], KL-divergence and ddvveeegee are considered as regularizers, and the sample complexity on optimality gap for x ~DPO are studied. We summarized our comparison with different works related the theoretical study of RLHF in Table 1.
Comparison in terms Rmax and coverage constant: In Table 1, we compared different methods in terms of their sample complexity bounds. Regarding the dependency Rmaxx observe that all existing bounds for RL.HF with RKL regularization scale as ((exp( Rmma)) [Song et al., 2024, Zhao et al., 2024, Chang al., 2024, Xiong al., 2024]. This exponential dependency arises directly from Lemma B. reflecting the inherent non-linearity introduced by the sigmoid function in the Bradley-Terry model. Additionally, concerning the coverage constant, the upper bounds under RKL regularization scale as O(Ca.saa), highlighting the significant impact of coverage parameters on optimal and suboptimal regret bounds.
Extension to DPO: Our current results for reverse KL-regularized RLHF and forward KL-regularized RLHF can be extended to the DPO framework [Rafailov et al., 2023]. In particular, can derive the following DPO function for reverse KL-regularized under multiple reference models scenario using Theorem 5.1,
RKL π DDO,0
arg max TIE
log o
{o(= log(
(y (22.) ) Ta ref (y (ar.)
log(
Te (vilzz)) ) Ta. (vf (z;))
(22)
For forward Kegrearlarieed DPO, we can combine Theorem 6.1 with the approach outlined in [Wang al., 2024a], to derive the DPO function,
[o ( Te (V/l=z) TT. (v." [x.)) To (o.) To (y) (x+))
FKL. arg max N
Σ log
(23)
DPO,8
Furthermore, we derive optimality gap under bounded implicit reward assumptions in App. F.
Experiments
To support our theoretical findings, we conducted two sets of experiments: one using online policy gradient algorithm, and another using an offline RI.FF algorithm. Together, these experiments are designed to cover the primary use cases of Kl--constrained RI. optimization in the L.M post-training setting. Our experiments address two goals:
1. Evaluating the benefits of using multiple reference models versus single reference.
2. Comparing exact analytical solution the approximation proposed by Le et al. [2024].

---

Online RL. Since our theory applies to general KL-constrained RL not only to settings with learned reward models, as in standard RLHF we ran an experiment on the GSM8K dataset [Cobbe et al., 2021] using GRPO [Shao et al., 2024], a policy gradient method. This setup uses solution verifier as the reward model, avoiding complications from learned rewards and letting us focus on the effect of multiple reference models during training. We trained the instruction-tuned 0.5B model from the Qwen 2.5 family [Yang et al., 2024], and used the 1.5B math-speccillzzd model from the same family as second reference. For each value of E {0.0, 0.3, 0.5, 0.7, 1.0). for FKL we consider 3 3=0 a, we trained models using the following regularization: (1) Normalized geometric mean as in multi-reference RKL objective, (2) Arithmetic mean an approximation of our multi-reference FKL objective, and (3) MRPO approximation [Le et al., 2024] of the multi-reference RKL objective.
Offline RL. This experiment compares our exact analytical solution to MRPO [Le et al., 2024] in an offline RLHF setting. We trained the instruction-tuned 0.5B Qwen 2.5 model using the UltraFeedback dataset [Cui et al., 2023], with the 1 1.5B Qwen 2.5 model as the second reference. This can be seen as a combination of knowledge distillation [Gu et al., 2023] and RLHF Evaluation was performed using the Skywork-Rewee Beward-Llama-3. 1.lamaa.---8B-.. 1..1-8--0... reward model [Liu et al., 2024]. Here again we compared three training algorithms: (1) DPO [Rafailov et al., 2023] using the normalized geometric mean of reference policies, (2) DPO based on FKL divergence proposed by [Wang et al., 2024a] using the arithmetic mean, and (3) MRPO version of DPO [Le et al., 2024].
To validate that our algorithm works at larger scale, we also applied it to the Qwen 2.5 7B model. We trained this model on the UlrrFeedbacck Cui al. [2023] dataset, and evaluated the trained model's win rate against the preferred answer using GPT-4o as LLM-as-a-Judge. We follow the standard protocol of first performing SFT on the dataset before the DPO step. As a second reference, we used Qwen 2.5 14B Instruct. Due to the increased compute demands, we only experimented with a = 0.5.
Table 2: Win rate against the preferred answer from the Ultrafeedback dataset. Combining both references leads to a substantial gain in performance.
Model
Win rate
Base model (Qwen 2.5 7B)
8.6%
SFT model
43.4%
DPO (single reference SFT model)
56.4%
DPO (single reference - 14B model)
59.8%
Ours (DPO with both references)
66.1 %
For more details on the experimental setting and discussion on the computational aspects of using multi-reference, see Appendix H.
Conclusion and Future Works
This work develops theoretical foundations for two Reinforcement Learning from Human Feedback (RLHF) frameworks: reverse KL-regularized and forward KL-regularized RLHF. We derive solutions for both frameworks under multiple reference scenarios and establish their sample complexity bounds. Our analysis reveals that while both algorithms share identical sample complexity for the optimality gap, the reverse Kl.-regularized RLHF achieves superior sample complexity for the sub-optimality gap.
The main limitation of our work lies in the assumption of bounded reward functions where some solutions are proposed to solve this limitation in App C. Promising directions for future work include: (a) extending analysis to multiple-referente sub-Gaussian reward functions; (b) following [Wang et al., 2024a], investigating
KL-regularized RLHF with unbounded rewards or
reference RLHF regularized by general f-divergences; (c) following [Sharifnassab et al., 2024], extending the analysis to preference models beyond the Bradley-Terry (BT) model; (d) following [Xu et al., 2025), combining our approach with doubly robust preference-optimizzaion algorithms to mitigate model misspecification; and (e) extending inference-time algorithms (e.g., [Mroueh, 2024, Beirami et al., 2024, Aminian et al., 2025]) to the multiple-reference setting using our approach.
10

---

Acknowledgment
The authors would like to thank Ahmad Beirami for his valuable comments and insightful feedback on our work. Gholamali Aminian acknowledges the support of the UKRI Prosperity Partnership Scheme (FAIR) under EPSRC Grant EP/VO56883/1 and the Alan Turing Institute. Amir R. Asadi is supported by Leverhulme Trust grant ECF-2023-189 and Isaac Newton Trust grant 23.08(b).
References
Gholamali Aminian, Idan Shenfeld, Amir R Asadi, Ahmad Beirami, and Youssef Mrouch, Best-of-n through the smoothing lens: KI divergence and regret analysis. arXiv preprint urXiv22507.09113 2025.
Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, Ananda Theertha Suresh, and Ahmad Beirami. InfAlign: Inference-aware language model alignment. International Conference on Machine Learning (ICML), 2025.
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander A'mour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees the best-of-n alignment policy, arXiv preprint arXiv:2401.01879 2024.
J-F Bercher. simple probabilistic construction yielding generalized entropies and divergences, escort distributions and q-gaussians. Physica A: Statisfical Mechanics and its Applications, 391 (19):4460-4469, 2012.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford University Press, 2013.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4);324-35,, 1952.
Pierre Cardaliaguet, François Delarue, Jean-Michel Lasry, and Pierre-Louis Lions. The master equation and the convergence problem mean field gamss(() Princeton University Press, 2019.
Jonathan D Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for rlhf. arXiv preprint arXivv2404.08495 2024.
Atoosa Chegini, Hamid Kazemi, Iman Mirzadeh, Dong Yin, Maxwell Horton, Moin Nabi, Mehrdad Farajtabar, and Keivan Alizadeh. Salsa: Soup-based alignment learning for stronger adaptation in rlhf. arXiv preprint arXiv:2411.01799 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2rXiv:2110.144 14168. 2021.
Samuel N Cohen. Data-driven nonlinear expectations for statistical uncertainty in decisions. Elec- tronic Journal of Statistics, 11:1858-1888 1889, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafedbbcck: Boosting language models with high-quality feedback. 2023.
David A Freedman. On tail probabilities for martingales. the Annals of Probability, pages 100-118. 1975.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. 11462, 2020.
11

---

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.085: 08543, 2023.
Audrey Huang. Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dy- lan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. arXiv preprint arXiv:2407. 13399, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0666 2023.
Hung Le. Thommen Karimpanal George, Majid Abdolshah, Dung Nguyen, Kien Do, Sunil Gupta, and Svetha Venkatesh. Learning to constrain policy optimization with virtual trust region. Advances in Neural Information Processing Systems, 35:12775- 2022.
Hung Le, Quan Tran, Dung Nguyen, Kien Do, Saloni Mittal, Kelechi Ogueji, and Svetha Venkatesh. Multi-eeference preference optimization for large language models. arXiv preprint arXiv:2405. 16388, 2024.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou Skywork-reward: Bag of tricks for reward modeling llms. arXiv preprint arXiv:2410. 18451, 2024.
Youssef Mrouch. Information theoretic guarantees for policy alignment in large language models. arXiv preprint arXivv2406.05883, 2024.
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint 27X61:2006.01116 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint ar Arv: 10.005...820 18290, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0333 2024.
Arsalan Sharifnassab, Saber Salehkaleybar, Sina Ghiassian, Surya Kanoria, and Dale Schuurmans. Soft preference optimization: Aligning language models to expert distributions. arXiv preprint 1:Xiv:2405.007477 2024.
Yuda Song, Gokul Swamy, Aarti Singh, Drew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Hao Sun and Mihaela van der Schaar. Inverse-rlinnment: Inverse reinforcement learning from demonstrations for Ilm alignment. arXiv preprint arXiv:2405. 2024.
Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial 202.1.
coverage. arXiv preprint
12

---

Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generaliz- ing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, 2024a.
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Bo Mao, Sitaram Asur, et al. A comprehensive survey of Ilm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216. 2024b.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136): 2017.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 29995523999, PMLR, 2022.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning. 2024.
Erhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu, Francesco Quinzan, and Chengchun Shi. Doubly robust alignment for large language models. arXiv preprint arXiv:2506.00 183, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint arXiv:2402.07314 2024.
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability, In Conference Learning Theory, pages 2730-2775. PMLR, 2022.
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305. 14816, 2023.
Heyang Zhao, Chenlu Ye, Quanquan Gu, and Tong Zhang. Sharp analysis for kl-regularized contextual bandits and rlhf. arXiv preprint 1 1.04625, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 190.00999. 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Section and Abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Conclusion and future works section (Section 9).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
See Section 4.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All necessary details are listed in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
All code needed to reproduce the experiments will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Co//eeodbisseeeeeeeeee
/nips. wwtppccccc/caeeccccccccccccceaae Stti:/ggiideeeeeeeebimiiiiee cassoonooiccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Yes, the training algorithms are detailed in the paper and all the necessary information is discolosed.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
All experiments include 95% CI,
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We explicitly mentioned the type and amount of compute that was required for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / /ttp///eeruppeeeeeee/eeeeee peppubiic/tthcssuiiiieooo
Answer:
[Yes]
Justification:
We reviewed the Code Ethics and made sure we conform it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our main contribution is theoretical.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our main contribution is theoretical.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, all code that was used in this work was mentioned and cited appropriately.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We did introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:[NN
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https / 225p::/p//../oooeeeeee////////// for what should or should not be described.
20

---

A Algorithms
The RLHF algorithm with two reference models is shown in Algorithm 1. Furthermore, the forward KL-regularized RLHF algorithm with two reference models is shown in Algorithm 2.
Algorithm 1 Reverse KL-regularized RLHF with Two Reference Models
Require: 7, Ot Frof,1 1 Rrof,2 +
1: for 1= 17t do
2: Sample prompt 32 p and 2 responses with their preference of" 01 (-
Tof (-124)
3: end for
4: Compute the MLE estimator of the reward function based on Dn = {( V., DiE:
i arg max C(e, Dn);
5: Compute the RLHF output based on (16): 73 Ra ( exp(rra(:.))
Algorithm 2 Forward KL-regularized RLHF with Two Reference Models
Require: 7. (3 1r Feff, +
1: for = -....., do
Sample prompt é; p and 2 responses with their preference
0." o ~ [-xx)(-)2z) = B (+)) + (1 - (-
3: end for
4: Compute the MLE estimator of the reward function based on Dr. = (( " ;)))
t arg max ((),
5: Compute the RLHF output based on (20).
B
Technical Tools
In this section, we introduce the following technical tools and Lemmata. Lemma B.1 (Lemma C.2 from [Chang et al., 2024]). Under Assumptions 4.1 and 4.2, we have with probability at least 8 that
Fead Ey Y Rear
(rw (Ix Y²) rw- (2, Yu) - Y1) r2(2,, Y
(24)
128R max exp(4Rman) log( (R./8))
Lemma B.2 ([Boucheron et al., 2013]). Assume that function f(x) E [0, B) is bounded. Then, we have,
KL (L(P(X))lg(X)
Exx) - By Lemma B.3, Assume that (y(xx) x Trer(y|z) exp(1r(xx)) v)). Then, #.44(1/z) = (f(xx)), where A is constant.
(25)
2
Proof.
A))
7 (1+2(1/2))
Ey (xp((x(((zx)) + A))]
(26)
Tref (y(z) exp(1r(z,)))
Ey (1 exp(17(x, v))
21

---

C Assumption 4.1 and Assumption 4.2 Discussion
These Assumptions are common literature are common in the literature [Song et al., 2024, Zhan et al., 2023, Zhao et al., 2024, Chang et al., 2024, Xiong et al., 2024]. In particular, Assumption 4.1 primarily enable the use of concentration inequalities like Freedman's 's inequality [Boucheron al., 2013], which require bounded differences (as in Lemma B.1). However, this assumption can be relaxed under certain growth conditions, as discussed in [Freedman, 1975]. Moreover, even when original reward function is unbounded or sub-Gaussian- modeling- it is possible to apply a monotonic, bounded transformation to the rewards. For instance, the cumulative distribution function (CDF) of the reward under a reference model to normalize the rewards into bounded range, as proposed in [Balashankar et al., 2025]. This approach also retains the essential ordering of preferences and supports handling sub- Gaussinn behavior in the transformed space. Regarding finite class, we can apply covering number and relax this assumption utilized in [Zhao al., 2024]
is often the case in human preference
D Proofs and Details of Section 5
Lemma D.1. Let 0s 10, 1] for all i [k] and Σ i=1 a4 For any distributions Q3 for all E [b] and P over the space X, such that P x Qi we have
(LT(()))))) KT. (P((((()))))))))) log
Σ
t i=1 1
Of (0))
ZEA
Proof. We have
* Σ at
P(x)
Σ
a (T.(Q.))))))
Σ DEA
P(x) log
2i(x)
1=1
(ri(x)
P(x)
(x)
REA 1=1
P(x) k
Σ ((x) log
1T=11 Q7 (x)
KL (P][ - log
Σ 11 1.
(x)
FEA
Lemma D.2. Let A be an arbitrary set and function f A R be such that
[SL exp (-Sie)) x Qx (x)dde < oc.
Then for any Px defined A such that X Px, we have
(f
exp (-f(x) Qx(x)dz)
E[F(X))
AKL (Gibbe) -A
UEA
where
exp
_ f(x)
-
Qx(x)
pGibbs
I E A,
SEEA exp
f(x)
Qx (x)dx
the Tibbs-Boltzmans distribution.
22

---

Proof. We have
/ (()))x( X / Px (x log
Px(x))
E[((X) + AKI((xIIQxx
Qx(x))
A / Px(x))
Px (z)
exp
[(x]
Qx(x)
=AKL (Px P - Alog
( exp
(->
(f(xx)
@x(x)de)
Theorem 5.1. Consider the following objective function for RLHF with multiple reference models,
(x) }.
1
max
E Y~u Y~x(11x))
[ro- (ro- (x,
7
where E Or = and as E (0, for i E [K]. reference models objective function for RLHF is.
Then, the exact solution of the multiple
Ra. (01z))
T (vlx)
exp (Tro- (I, (Tra-(I.21)
(27)
2(x)
where
IT.
Fa (x)
K
Fa (z) = ΣI
ai
3EVi=1
and
2(x) Σ T net (y/22)
exp (nr(x, exp v))
The maximum objective value
1
log
ΣII
?
23

---

Proof. We can write
E
rre (2,
Σ
KL (
Y~u (1))
7
i=1
k
E
(ro- (x, Vi] -
0g <L (e)(((l)))))))
(28)
Ywa (11x)
ro (x, x)] - KL(T (L) + log Fa(x)
(29)
You (11))
1 (KKL( (ul=)) + log ((x) + log Fa(x))
(30)
7
1 -KL(T(114]IT) (ulz)) log EII Fer exp (1r(,, v))
(31)
7
where (29) follows from Lemma D.1 and (30) follows from Lemma D.2. Clearly, the right side of (31) is maximized when the KL divergence is set to zero. Thus, the max- imizing distribution x(-1x) is identical to Th. (ylz), and the maximum objective value is
log Ey I
as (vll)) exp (1r(2,8))).
Corollary D.3. Weighted multiple single reverse KL-regularized RLHF problem is an upper bound on multiple references reverse KL-regularized RLHF problem, i.e.,
rre (x, Y)] - 7 1=1
Σ
a KLKL (T())ll)))
max
Y ((x)
(32)
K
[rox (z. y)] 1 (
a,max
E You (1z)
Proof. It can be shown that the maximum of objective function in Theorem 5.1 is,
K
E (ro- (z. K)] - Σ
max 1
YNoT (-x((11))
log (Er lexp (yro- (z, x)))
1
log Fa(
7
(33)
log ΣI fer (vlz) (artre- (z. v))
<
a2
log
Tret, (ylx)) exp (Sror (I, (x.,)))
1=1
where the last inequality follows from Hölder's inequality. Note that,
E
[ra- (z, Y)]
KL (
Y You
(-(xr)
Σ (y)r)) exp (rrœ- (x, Y))).
log
(34)

---

Then, we have,
rro (x, Y)] as ( (xKK,((((a)))lmecs(())))
max
Y
(1)))
(35)
rre (x. Y)l
( (I)
0,max
You (2)
Therefore, multiple single RLHF problem is an upper bound multiple reference models RLHF problem.
Remark D.4 (Choosing a). The optimum for a given x, can be derived from the following optimization problem.
K T (1|z) exp (arnre- (z, v)).
max log
(III
(36)
y E1
Proposition D.5. For a given response, a E X, the following upper bound holds,
(T (1w). - / (ro- (z, v) (r(x, 2 (vlx) (z(()))))
Proof. Note that (-((((|<=]]] (-Ixa) is convex function with respect to (-Iaz). Therefore, J. ref (z),, (- (z)) is a concave function with respect to (-1xx) First, we compute the functional derivative of J. (ma ref (-1x), T-11z)) with respect to x (-11z),
8J, ret (-1x), n (-(x))
1 y
1 log(m()1o)//amrmdl + ?
ro- (z, v)
(37)
8T
Therefore, we have,
(T (1lx), (-1=))
Jy aet (I), T (-1x)) - Jn (1))
8J (Fa (11z3, m3 (vlx:) (1(x)) - (z(vl=))(dy)
<
8m
/ (x, v)
log 1og(=} Ta (-) + +) ( (u|x) (x((lr))
(38)
(z, y) (r((x,)) y) + log (ogg(Z(z)) (5 (0|=) - (3ll
S (To (2.(2z)) ((c.v))rae
- rg((,, (T (wl=) (vl)))))
It completes the proof.
Lemma D.6. Consider the softmax policy, n; (y(x)) x (e)eer(y)x) exp())(z, y)). Then, the sensitivity of the policy with respect reward function is,
θm7 (r) = 7 - 7, (v(x))
dr
Proof. We have (z(2)) = te exp(7r(z.0))
Using Chain rule, we have,
By mit (xx)(vr(a.)))
θmū (r) =7 net (1) exp(pr
T not exp(27r(x v))
Dr
(xx (xx Y)J
Ey on (1x) Y))
(39)
(1-
25

---

Theorem S.2. Under Assumption 4.1, 4.2 and 4.4, the following upper bound holds on the sub-optimality gap with probability at least (1 - 0) for å E (0,1/2).
( (-10), 77 (-1x))
log( (R)/0)
Sa 128e ARmax Rmmax
4
n
Proof. Using Proposition D.5, we have,
(7 =))x))
/ (ro (1. ro(a v)) (vl=) -x)
(40)
Note that, the integral in (40) is over y, therefore, have,
[T (112), m-
(41)
(ro- (2, rg(x.v) (Tz)) (2(())))
where l(()) is an arbitrary function over X. Note that тỘK (vlar) and TH (vlz) are function of re- (v, y) and r2(2, y), respectively. Furthermore, softmax policies are shift invariant, Lemma B.3, i.c. mg. (ular) x 6 aa.ret(-)az exp()(rr)) (x; y) -(z))) where h(x) is function dependent on æ. Therefore, we can apply the mean-value theorem to (Tg. (y/l²) - 73 (dl)) with respect to reward function
r(x, v). Therefore, we have for given h(x),
(17 - = 0π((122)
Or (rx) (T))(To (I, ra(0, Mx))
(42)
= - Ty (1x2) (x, rg(xx,) h(x)),
where rs A(ro- (x, h(x)) (1 \)rg(E,)) for some A E [0, and T. (11z)
Ta, exp(txxxzz (xxppp((( w)). Applying (42) in (41), we have.
( (12x), Tp (-1z))
(ro- (2, v) (2) 7- (xx(())))(d)))
$ / (x))()) u)) ) (+Ix)
(43)
s CasEui / (re- (z, - r5(2, v) (x))) not (-Iz)
Choosing h(x)= = Eyi (n)(ro (x, (1,,Y")rooaa (x, Y')) applying Jensen inequality and Lemma B.1, have,
JT (1x))
ro- re- (x,8)) + rg(2x, 2 & Ca.es / (I, - y²d)
aa we (-(l)))))) ( (dy")
(+)
< 5 1288 128 ARmax R2 log(
maax
n1
(44)
This completes the proof.
26

---

Theorem 5.3. Under Assumption 4.1, 4.2 and 4.4, there exists constant C' O such that the following upper bound holds on the optimality gap of the reverse KL-regularized RLHF with probability at least (1 5) for E (0, 1/2).
log( (
(17) s aaa 128e 1Rmax R Raax
+ C8 2 /
log( R|/5)
11
Proof. We have the following decomposition of the optimality gap.
KL (+1x))
(1x) (T)
?
(45)
Now, we provide an upper bound on the second term using Lemma D.6 and similar approach for choosing h(x) in the proof of Theorem 5.2, we have some X [0, 1].
(T. (-1x) [a)) (+2:)) (T. (
//- Or
dn (rx) (2 (log( Tra (1|2)) + (row (z, v) - rg(a,,) v) h(x))(dy))
(r,
= 7 / Tra (12()1=)(1 - Tra
- (1 (-120)
log
Tra (1(2) ) 1) (ro- (x, ra(a.v) hdx)))
aet
(46)
s S71 / (1 (1-TTr) Trx (1x)) log
Tra (-12)
+ +1) (dy)
Tet (-1z))
1/8 Tra (1xz) (x,y) ra v) h(x)))
where, in the last inequality, applied the Cauchy-Schwarz inequality. Using the fact that Tr. rof (-12) exp()rx) and Lemma B. we have,
(T rer (+1x)) - KL(TT
2Cass (8)
(47)
S 78 ( 27/ Rmax exp(2 V
The final result holds by applying the union bound.
In the following, we compare the RLHF objective function under the multiple reference model policy, net (-1x), with i-th reference model, Trof (+Jar). For this purpose, we bound the difference between these two RLHF objective functions in different scenarios.
Proposition D.7. Under Assumption 4.1, the following upper bound holds,
(Ta, - J. J.,(Trek (Tref, TZ-,)) 5 exp(> Rmmax) KL( (122) (-1x)).
7V2
Proof. Note that, for policy Mref we have,
[T TzZ)) 7
log Exne jexp(1re- (I, (r,2))]]
(48)
27

---

Therefore, using the functional derivative, we have,
J na, Tz-) (Tref, .g),;)
7 log Bn lexp(yre- (I, m/1l| 7
1
log Bexne, /exp(1ro (2c m))1]
1
exp(yro- y))
Taaref (dy))d)
Y yy E (exp(1ra (z, y))l
E
(49)
1
exp( (10()ooo00,))))) (r, Ta Tret,)
7 BT (xp w))l
exp() - (T (r) (ret,s(((1))
7
2
where Tref,A Tref,i A(Taa Trof,e) and the last inequality holds due to Lemma B.2.
28

---

E
Proofs and Details of Section 6
Lemma E.1. Let B, E [0, for all i [k] and Ei= 3r = For any distributions Qs for
all i E [k] and R such that Qi < P. we have
3,KL(Q.IP) H Σ B.Q.)
Σ
(Q.) KL Σ BQUPP)
Proof. We have,
(50)
B.Q, (o,)(.) BiQ, log(P)
& B ((H)) + Σ
k
B.Qi) log
B:Q.)
3.Q:) log Σ (QQ:) - -
3:Q;) log (P)
==1
1=1
1=1
1=1
(51)
k
k &H(Q.) + Σ
k
H
8,0
(22)) log
B,Qi) -
3:Q;) log(P)
(52)
1=1
1=1
1=1
1=1
H
Σ 3.Q.)
Σ
BH((Q)) KL
KL
B.Q.IPP.
(53)
Theorem 6.1. Consider the following objective function for RLHI with multiple reference models,
[ro- Y)]
3,KL (=To)
max Y
(1(r)
where 8 reference models objective function for RLHF is.
- =1 and Bi E (0, (),1) i [K]. Then, the implicit solution of the multiple
TB. y/
#7. (v/x)) =
7 (Z (x)
rg- (I, v))
where
K
78.ro((y(27) Σ B
and Zo. (x) is the solution Jyey #3. (ylæ) 1 for given x E X.
Proof. Using Lemma E. the objective function of forward KL-regularization under multiple refer- ence model can be represented as,
[ro Y)] (
max
Y~U (112)
where ( ( Be (y)la). As the function is concave function with respect to π ((-1x)). compute the derivative with respect to T-12x). Therefore, using the functional derivative
29

---

under the constraint that <-+[2) is a probability measure with Lagrange multiplier, (2), we have optimal solution that,
Te- (x. 8) + 1 1. Zo. (x) O.
(54)
Tg. (y)x)
Solving (54) results in the final solution, Tg.. (y ()]
Corollary E.2. Weighted multiple single forward KL-regularized RLHF problem is an upper bound on multiple references forward KL-regularized RLHF problem, i.e.,
E You
[ro- (x, Y)] - 3 (
Σ
(1)))
(55)
K
8amax
E YouT
ra- (2,, Y)]
KK ( (1)
7
Proof. It holds due maximum function property.
Assuming,
" Bell (1/z))
# (y/x) 7Z (x) - rg. (x. v))
we can provide the following property of Zg. (x), inspired by [Cohen, 2017].
Lemma E.3, The following property holds for Zo- (I),
For any X where p(x) O, we have supyey r (z, v) S Zo- (z).
Under Assumption 4. we have supzex Zo (x) S Pmmx 1
Proof.
Using the following representation,
TBr (y(r)
# (v/x)
7 77 (x) - ro- (x. y))
we can conclude that for a given a e X, yup Tg- (x, 3) s Zg. (x). Otherwise, R.. (y/(z) will be negative.
For the second part, let's proceed by contradiction. Suppose there exists some x E X such that: 28* (z) s Vro. (x.y)
Under this assumption, show that:
J. #7. #7. (ylo)(d))) < 1.
This contradicts the fundamental requirement that
Ty (ylæ),
must be a probability distribution. Therefore, our initial assumption must be false. Consequently, for all E X, must have;
(x) S sup ro- (a, y)
yey
Taking the supremum of both sides with respect to completes the proof.
30

---

Proposition E.4. For given response, X, the following upper bound holds.
J' (1)) ك / (2) y) (u/z) - (y)=))))
Proof. The proof is similar to Proposition D.5. Note that (L (et, (-1=)]] is a convex function
with respect T-|2z). Therefore, J,, (e (-12z), x--[x)) is a concave function with respect to x(-[ar).
First, we compute the functional derivative of J,, (R (-12) x(+(x)) with respect to ((-1x),
(o (r), (--1x))
ra- (I, + 7 xe
(56)
(-122))
Therefore, we have,
(1), = / (ro) Using the fact that (v(x)) -(2())-)g(2.9))
1 T (gg.et(v((z))
(#g (0l))))))(((110))) *g(olz))(d))
(57)
(u((x)
/ (ro- (zx, v) (z) v) + ((x)) (f/x)
(7g (g(z)))
(u/z) - #3(vlz)))dy)
(58)
/ (ro- (zz, ra( v) (5 (ulx) - (dll))))
where the last equality follows from the fact that (xx) is just dependent on æ.
Theorem 6.2. Under Assumption 4.1, 4.2 and 4.4, the following upper bound holds on the sub-optimality gap with probability at least 8) for ö (0,1),
log R|
( (1x) (+x²) S 16CB 22mar Remax
n
Proof. From Proposition E.4, have,
S /, (ro- (2r, (2, 1 w) (#7. (u|x) (zl=))))
= / ro- (I, v) - -rg(z, y) - h(z)) (#7- (2/z) #d(y)=))(dy)
= -/ʃ (z, r 9) - M(x)) (
(ulz) - - (1))))
-
(s/ax)
(dy)
TB. (r)
rg(2.v) J (1)(())))) m(t)) 2 ro(o, v) - b(a)) a. (21))))) /
(Fg, (#8))a((01))))((d)) ///
(#2 (ylz) - (y)x:)))
(dy)
(TT (1/ 2
/ (ro- (I, (IT
(T (u|z) - - (2l=)) (dy)
(TB. ())=))²
log(/R1/8)
16Cp.em 2Rmax Rmax
n
(59) inequality,
where the first, second, and last inequalities follow from the (F))))()1)))2 (z) and using Assumption 4.5 and Lemma B.I, respectively.
D
31

---

Theorem 6.3. Under Assumption 4.1, 4.2 and 4.4, there exists constant D 0 such that the following upper bound holds on optimality gap of the multiple reference forward KL- regularized RLHF algorithm with probability at least (1 5) for 8 (0,1),
[+Z (-1x), #g(()z)))
log( R|/6)
max (1 (loog(B,,eni)) log(? + 1))
S 16C e² Remax Remax
+
Proof. We have the following decomposition of the optimality gap,
J(HZ. (12) = (11x)
KL (-)1 (L(TA.ee(( (-1z))
(60)
For second term, using the fact that, [))((())) (gle)( and T.. (y|xx)
((2g())-7g((a))
-())-(0)- ra- (2,9))
we have,
KL
(TTAA (
7 (y> (log(1/ (Zz(z)) r2(2, v))i Ey - o (10g0 rg- b)))
Ey (1y ra(2,, w)Dill + |Ey ~ ind (1)) (x) rg- (z. y)
max (oo)(C logaaynnxxxxx Rimax 1))
(61)
where the last inequality follows from Lemma E.3. The final result holds by combining Theorem 6.2 with (61).
32

---

Extension to DPO
As discussed in [Song et al., 2024], DPO can not guarantee any performance under some conditions. In particular, The reverse KL-regularized case can fail under partial coverage conditions, necessitating the Global Coverage Assumption (Assumption 4.3). The forward KL-regularized case requires an stronger condition: the ratio of reference policy must be bounded from below away from zero. (p)(a) which a stronger assumption. For this Specifically, we should have () < (2.))))(())) (y)a)))) purpose, we consider the implicit bounded reward assumptions.
Our theoretical results for reverse KL-regularized RLHF and forward KL-regularized RLHF can be applicd DPO problems (22) and (23) under the following assumptions.
Assumption F.1 (Boonnded implicit RKL reward), For all y" y E y and x E X, there exists constant Buax such that,
To (z) rof (ynh (x)
1 log(
<o(3' Tot (y/ [x)
log(
S Bmax.
(62)
Assumption F.2 (Bounded implicit FKL reward). For all y" .y E y and X, there exists constant Dmax such that,
1 T. Tf (v)lz;) B. of (v!" (z4) Te (,,"" x.))
D max.
(63)
xx)))))
Lemma F.3 (Lemma E.5 from [Huang et al., 2024]). Under Assumptions 4.1, F.1 and 4.2, we have with probability al least 5 that
Byldwaadd [To- (1, v2) ro- (z, y ra(a, r.) + ra(o, v
(64)
128B max exp (mxx)) log( (
The same results also holds under Assumption F.2.
Theorem E.4. Under Assumptions F.I, 4.1, 4.2 and 4.4, there exists constant C > such that the following upper bound holds on the optimality gap of DPO based reverse KL- regularization with probability at least (1 - 8) for (0, 1/2).
log( ()
T [ (11z), (1))) & (Ca.em 128c 128 ARmax B2
max
C8 2 Rmmor 20 2Cam log([ /8)
Proof. The proof is similar to Theorem 5.3 using Lemma F.3.
Theorem F.5. Under Assumptions F22, 4,1, 4.2 and 4.4, the following upper bound holds on optimality gap of DPO based on forward Kl Kl-regularization with probability at least - 8) for 8 10,
log ())()))
J(7 (1z),,7((1z)) S 16Cp.en 2Rmax D Dinax
n
max ( (loog(C..na), logg log(> + 1))
Proof. The proof is similar to Theorem 6.3 by using Lemma F.3.
33

---

G Further Discussion
Coverage Assumption Discussion: The coverage assumptions for multiple references can differ from the single reference scenario. For the reverse KL-rrgularized case with reference policy (- we have:
(v][²) (-(la))
Fa (z) T T(Ul=) )
K
II i=1
(65)
where (I) is defined in (15). Therefore, we have II Cet. as the global coverage assumption, where Cref < oc is the global coverage with respect to the i-th reference. Note that, using Hölder's inequality,
can show that (x) S 1. A similar discussion applies the forward KI-reeularrzation scenario with reference policy Te Regarding the local reverse KI.-ball coverage assumptions (Assumption 4.4), Fo, (+xz) is defined common support among all reference models, then the set of policies with bounded
Egp [KL ( S &a,rkl
(66)
smaller than each reference model separately. Similarly global coverage, we can assume that Ca.e. = -II=1
Co "efiieaaaa
Comparison of RKL with FKL: The RKL. and FKL. exhibit fundamentally different characteristics in their optimization behavior. RKL between the reference model and target policy, defined as Egg. [log((Te) /ref)] demonstrates mode-secking behavior during optimization. When w represents the output policy of RLHF for language model alignment, it may assign zero probability to regions where Fee is positive. Conversely, FKL, expressed ET, (Tret Tg. )}, exhibits mass-covering properties. Its mathematical formulation requires тg. to maintain non-zero probability wherever Tref positive. This constraint naturally leads FKL produce distributions that cover the full support of the reference model, thereby promoting diverse outputs.
In the multiple reference model setting, the generalized escort distribution under reverse Kl-regulaiization covers the intersection of the supports of all reference models. Specifically, responses receive zero probability if they lack positive probability in any single reference model. This leads the generalized escort distribution assign non-zero probabilities only responses supported by all reference models simultaneously. In contrast, when using the average distribution as the reference model in the forward KL scenario, the resulting distribution covers the union of supports across all reference models, encompassing a broader range of possible responses.
Reference policy in multiple reference model scenario under FKI and RKL:
34

---

H
Experiment Details
Implementation code provided at http://gtttuuu.mmiiaaaaaaaaaeeee
To ensure fair comparison across algorithms, we began by conducting an independent hyperparameter search for each method. For the GRPO experiment, explored learning rates of (1e-3, 1c-4, 1c-5) and KL coefficients of {0.05, 0.1, 0.2). For the DPO experiments, we explored learning rates of {le-6, le-7, 1e-8). We also tried different y values but found that the default one works the best in all cases. After selecting the best configuration for each algorithm, we trained each setup three times with different random seeds estimate variability and compute confidence intervals.
In the case of GRPO, using the full FKL objective would require sampling from the reference model, which roughly doubles training time. To reduce this cost, we instead approximated the FKL term by sampling from the trained model and computing per-token objective- -ttiking a balance between efficiency and fidelity the theoretical objective.
Our data splits were chosen to reflect standard practice where possible. For GSM8K, we used the official train-test split. Since UltraFeedback does not provide official split, we randomly withheld 10% of dataset and used the corresponding prompts for evaluation.
All experiments were conducted on A 100 GPUs. Offline RLHF training used a single GPU, while online training required two. Although multi-reference RL introduces some additional computational cittrremenntsspccfffeeee
evaluating logits from another policy- the cost is modest. In offline settings such as DPO, reference model logits can be precomputed and stored, avoiding memory overhead during training. In online settings like GRPO, the reference policy must reside in memory. but placing on a separate GPU resulted in only 10% slowdown.
35


Reviews:
[{"review_id": "hWmLyeoBcp", "reviewer": "Reviewer_vZWY", "summary": "This paper provides a theoretical analysis of KL-regularized RLHF using multiple reference models. The authors derive the first exact closed-form solutions for both reverse KL (RKL) and forward KL (FKL) regularized objectives in this multi-reference setting. They further provide sample complexity guarantees for the optimality and sub-optimality gaps for both frameworks.", "questions": "1. The normalization constant $F_\\alpha(x)$ in Eq. 15 requires a sum over the entire vocabulary, which is often intractable. How was this handled in the offline DPO experiment? Does the corresponding DPO loss function for multiple references (presumably in App. F) circumvent this explicit calculation?\n2. The coverage constants $C_\\{ε_\\{rkl}}$ are central to the bounds but their dependency on the number (K) and diversity of reference models is unclear. Cud you provide some intuition on how these constants might scale with K? Does the bound become vacuous if the reference models are very different?\n3. The empirical gains from using multiple references ($\\alpha>0$) are positive but appear somewhat marginal in Fig 1. In what practical scenarios do you hypothesize the multi-reference approach would provide a much more substantial improvement over a strong single-reference baseline?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n*   **Novel Theoretical Contribution:** The core strength is the derivation of an exact closed-form solution for the multi-reference RKL-RLHF problem (Thm 5.1). This is a significant step beyond prior work (Le et al., 2024) which relied on a lower-bound approximation. The resulting optimal policy as a normalized geometric mean of reference policies (Eq. 15) is an elegant result.\n*   **Comprehensive Analysis:** The anlysis is thorough, covering both RKL and FKL regularization. For each, the paper derives optimality and sub-optimality gap bounds (Thms 5.2, 5.3, 6.2, 6.3), providing a complete theoretical picture. The comparison in Table 1 clearly situates the contributions within existing literature.\n*   **Strong Motivation:** The paper makes a compelling case for using multiple reference models to improve diversity, reduce bias, and leverage the growing ecosystem of open-source LLMs. Teh motivation is well-grounded and timely.\n\n**Weaknesses:**\n*   **Limited Empirical Validation:** This is the main weakness. While the experiments in Sec 8 support the theory, they are limited in scope. The use of relatively small models (0.5B) and only two datasets (GSM8K, UltraFeedback) makes it hard to gauge the practical significance for SOTA LLM alignment. The performance gains shown in Fig 1 over the single-reference baseline are modest.\n*   **Practicality of Assumptions and Implementation:** The paper is heavily theoretical. The practical implications of the Local KL-ball Coverage assumption (4.4, 4.5) are not fully explored. For instance, how does the constant $C_\\{ε_\\{rkl}}$ scale with the number of reference models? Also, the computation of the normalization term $F_\\alpha(x)$ in Eq. 15 seems computationally prohibitive (sum over all $y$ in $Y$); more discussion on how this is handled in the DPO-style experiment would be valuable in the main text.\n*   **Significance of FKL Results:** While the FKL analysis is complete, the experimental results (Fig 1a, 1b) show it consistently underperforming the RKL approach. This raises questions about teh practical utility of the multi-reference FKL methd, even if the theory is sound.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and generally positive assessment of the paper. We will address their concerns in detail below. \n\n----\n\n> **Limited Empirical Validation: This is the main weakness. While the experiments in Sec 8 support the theory, they are limited in scope. The use of relatively small models (0.5B) and only two datasets (GSM8K, UltraFeedback) makes it hard to gauge the practical significance for SOTA LLM alignment. The performance gains shown in Fig 1 over the single-reference baseline are modest. The empirical gains from using multiple references ( $\\\\alpha\\>0$) are positive but appear somewhat marginal in Fig 1\\. In what practical scenarios do you hypothesize the multi-reference approach would provide a much more substantial improvement over a strong single-reference baseline?**\n\n**R1:** Regarding the performance gains shown in Fig. 1, we politely disagree that a gain of 30% in success rate in GSM8K (from ~30% of the single-reference policy to ~60% of the multi-reference policy) is “marginal”. We recognize the reviewer’s worry that we only used small models and therefore we tried larger models.\n\n*Experiment with larger model:*  We provide an additional experiment, this time with a larger model - Qwen 2.5 7B. We train this model on the UltraFeedback [cite] dataset, and evaluate the trained model's win rate against the preferred answer in the test split of the dataset using GPT-4o as LLM-as-a-Judge. We follow the standard protocol of first performing SFT on the dataset before the DPO step.\nAs a second reference, we used Qwen 2.5 14B Instruct. Due to the time and resource limit, we only trained a single model at $\\alpha=0.5$.\n\n| Model / Setting                              | Score |\n|----------------------------------------------|-------|\n| Base model (Qwen 2.5 7B)           | 8.6 % |\n| SFT model                                    | 43.4 % |\n| DPO (single reference – SFT model)           | 56.4 % |\n| DPO (single reference – 14 B model)          | 59.8 % |\n| **Ours (DPO with both references Eq.60)**          | **66.1 %** |\n\nWe can observe that using the geometric mean of reference policies as the reference policy in Eq.60, we can achieve better performance with DPO based on both references (Eq.60). \n\n> **The normalization constant $F\\_{\\\\alpha}(x)$  in Eq. 15 requires a sum over the entire vocabulary, which is often intractable. How was this handled in the offline DPO experiment? Does the corresponding DPO loss function for multiple references (presumably in App. F) circumvent this explicit calculation?**\n\n**R2:**  We thank the reviewer for his question. The DPO loss does not need the normalization $F\\_{\\\\alpha}(x)$ factor to be explicitly calculated, and therefore, the nominator of equation 15 can be easily calculated using the per-token probability of the reference models. In particular, in DPO experiments, as shown in loss function of DPO for both RKL and FKL, the normalization constants are removed. Therefore, this constant is handled in DPO experiments.\n\n\n> **The coverage constants  $C\\_{\\\\alpha,\\\\epsilon\\_{rKL}}$  are central to the bounds but their dependency on the number (K) and diversity of reference models is unclear. Could you provide some intuition on how these  \nconstants might scale with K? Does the bound become vacuous if the reference models are very different?**\n\n**R3**: Thanks for your point. Regarding the constant \\\\( C\\_{\\\\alpha, \\\\epsilon\\_{\\\\text{RKL}}} \\\\), we have shown that  \n\\\\\\[  \nC\\_{\\\\epsilon\\_{\\\\text{RKL}}} \\= \\\\prod\\_{i=1}^K C\\_{\\\\text{ref}, i, \\\\epsilon\\_{\\\\text{RKL}}}^{\\\\alpha\\_i}.  \n\\\\\\]  \nTherefore, \\\\( C\\_{\\\\alpha, \\\\epsilon\\_{\\\\text{RKL}}} \\\\) corresponds to the geometric mean of the per-reference coverage constants \\\\( C\\_{\\\\text{ref}, i, \\\\epsilon\\_{\\\\text{RKL}}} \\\\), weighted by \\\\( \\\\{\\\\alpha\\_i\\\\}\\_{i=1}^K \\\\). To have a non-vacuous upper bound, we need to have \\\\( C\\_{\\\\text{ref}, i, \\\\epsilon\\_{\\\\text{RKL}}}^{\\\\alpha\\_i} > 0 \\\\) for all $i$. It means that we should have a coverage of each single model policy with respect to the reference model.\n\n\n> **Significance of FKL Results: While the FKL analysis is complete, the experimental results (Fig 1a, 1b) show it consistently underperforming the RKL approach. This raises questions about the practical utility of the multi-reference FKL method, even if the theory is sound.**\n\n**R4:** RLHF with FKL regularization was originally proposed by \\[1\\], but its regret analysis has been overlooked. In our work, we address this gap by providing a theoretical analysis of FKL. For the setting with multiple reference models, we have the final reference policy as the arithmetic mean of the individual reference policies for FKL regularization. Interestingly, we observe that using the geometric mean can lead to better performance than the arithmetic mean. Furthermore, as mentioned in Appendix H, in the case of GRPO, using the full FKL objective would require sampling from the reference model, which roughly doubles training time. To reduce this cost, we instead approximated the FKL term by sampling from the trained model and computing a per-token objective—striking a balance between efficiency and fidelity to the theoretical objective.\n\n---\n\n**References:**\n\n\\[1\\]: Wang, Chaoqi, et al. \"Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints.\"."}, {"review_id": "WvT0DhW7oD", "reviewer": "Reviewer_3gQj", "summary": "This paper derives closed-form solutions and sample-complexity bounds for reverse and forward KL–regularized RLHF with multiple reference policies. The optimal policy is a normalized geometric mean (reverse KL) or weighted arithmetic mean (forward KL) of references, exponentiated by scaled rewards. Additionally, the paper adds GSM8K  and UltraFeedback as experiments.", "questions": "* Can one use approximation for reducing cost in the given analysis framework?", "limitations": "Yes.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "---\n* Quality:  \n    * Strengths: Provides closed-form multi-reference policy (Eq. (14)) with tight $O(1/n)$ and $O(1/\\sqrt n)$ bounds under relaxed coverage.  \n    * Weaknesses: Relies on bounded rewards and finite reference class; coverage constants $C_{\\mathrm{rkl}},C_{\\mathrm{fkl}}$ are never instantiated.\n---\n\n* Clarity:  \n    * Strengths: Reverse and forward KL derivations are presented in parallel, and the core policy formula is highlighted.  \n    * Weaknesses: Heavy notation (escort distributions, functional derivatives) and no running example or notation cheat sheet hamper readability.\n\n---\n\n* Significance:  \n    * Strengths: Replaces global coverage with a local KL-ball condition, still achieving the best known rates; unifies reverse/forward KL analysis.  \n    * Weaknesses: Practical impact on large LLMs is unclear—combining $K$ models per update may not scale and experiments are limited to small models.\n\n--- \n* Originality:  \n    * Strengths: First to derive exact multi-reference RLHF solutions without surrogate bounds, highlighting geometric vs. arithmetic blends.  \n    * Weaknesses: Omits discussion of links to errors-in-variables or robust regression literatures that also perturb feature distributions.", "was_revised": true, "final_justification": "The authors have addressed the questions, and I would maintain my score.", "rebuttal": "We thank the reviewer for their comments, and generally positive assessment of the paper. We will address their concerns in detail below. \n\n---\n\n> **Relies on bounded rewards and finite reference class; coverage constants $C\\_{\\\\epsilon\\_{rkl}}$ and $C\\_{\\\\epsilon\\_{fkl}}$ are never instantiated.**\n\n**R1:** Regarding the bounded reward and finite class assumptions, Assumption 4.1 and 4.2,, in appendix C, we provided the following discussion:\n\n“These Assumptions are common literature are common in the literature \\[1,2,3\\]. In particular, Assumption 4.1  \nis primarily to enable the use of concentration inequalities like Freedman’s inequality, which require bounded differences (as in Lemma B.1). However, this assumption can be relaxed under certain growth conditions, as discussed in \\[5\\]. Moreover, even when the original reward function is unbounded or sub-Gaussian—as is often the case in human preference modeling—it is possible to apply a monotonic, bounded transformation to the rewards. For instance, one can use the cumulative distribution function (CDF) of the reward under a reference model to normalize the rewards into a bounded range, as proposed in \\[5\\]. This approach also retains the essential ordering of preferences and supports handling sub-Gaussian behavior in the transformed space. Regarding finite class, we can apply covering number and relax this assumption as utilized in \\[1\\].”\n\nRegarding coverage constants $C\\_{\\\\epsilon\\_{rkl}}$ and $C\\_{\\\\epsilon\\_{fkl}}$, we also utilized these constants in our regret analysis. These assumptions help us to provide more refined coverage assumptions due to regularization which keeps the final policy close to the reference policy.  \n \n\n> **Heavy notation (escort distributions, functional derivatives) and no running example or notation cheat sheet hamper readability.**\n\n**R2:** Thanks for your suggestion. We add a table of notations in the appendix for helping readers to follow the notations.\n\n>  **Practical impact on large LLMs is unclear—combining $K$ models per update may not scale and experiments are limited to small models.**\n\n**R3:**  Thanks for the question about scaling. Our method does not fuse full models at each update. We only ask the reference models to score the text; the main model does the learning. In offline algorithms (like DPO), those scores can be saved once and reused, so training time stays the same. In online runs with two references, the extra time we saw was small (about 10%), as generations and backpropagation are the steps that require most of the time during training. \n\n> **Omits discussion of links to errors-in-variables or robust regression literatures that also perturb feature distributions.**\n\n**R4:** Thank you for raising this point. We agree it is helpful to situate our analysis relative to errors‑in‑variables (EIV) and robust regression, both of which study **perturbations of the data‑generating process**. Our work perturbs a different object: the **policy over outputs** (responses) and constrains it to stay near an **ensemble of reference policies** via KL. In particular, our reverse‑KL objective with K references is equivalent to a single KL to the *normalized geometric mean* (“generalized escort”) of the references, (Thm. 5.1), whereas the forward‑KL objective reduces to a KL from the *arithmetic mean* of the references, (Thm. 6.1). This is closer to **distributionally robust optimization (DRO)** with f‑divergence ambiguity sets than to classical EIV: we regularize the *candidate policy distribution* relative to a baseline (reference ensemble), rather than correcting parameter estimates under noisy covariates. We will add a paragraph clarifying these links, citing classic EIV/measurement‑error references (e.g., Fuller \\[6\\]) and robust regression (Huber \\[7\\]), and noting the DRO connection (KL ambiguity sets). \n\n> **Can one use approximation for reducing cost in the given analysis framework?**\n\n**R5:** Thank you for your insightful comment.\n\n- **Inference cost.** The additional cost of using multiple reference policies can be mitigated through **speculative decoding**, which reuses partial computations to keep overhead low.  \n- **Alternative cost metrics.** If you have a different cost model in mind, please let us know and we will be happy to address it in detail.\n\n> **The manuscript feels space-constrained; consider relocating the detailed assumptions to the appendix to streamline the main text and improve readability.**\n\n**R6:** Thanks for your suggestion. We consider it for the final version of the manuscript.\n\n----\n\n**References:**\n\n\\[1\\]: Zhao, Heyang, et al. \"Sharp analysis for kl-regularized contextual bandits and rlhf.\"   \n\\[2\\]: Song, Y, et al. “The importance of online data: Understanding preference fine-tuning via coverage.”  \n\\[3\\]: Chang et al. “Dataset reset policy optimization for rlhf”  \n\\[4\\]: Freedman, David A. \"On tail probabilities for martingales.\" *the Annals of Probability* (1975): 100-118.  \n\\[5\\]: Balashankar, Ananth, et al. \"InfAlign: Inference-aware language model alignment.\"  \n\\[6\\]: Fuller, Wayne A. *Measurement error models*. John Wiley & Sons, 2009\\.  \n\\[7\\]: Huber, Peter J. \"Robust regression: asymptotics, conjectures and Monte Carlo.\" *The annals of statistics* (1973): 799-821."}, {"review_id": "XtFxHxVGLz", "reviewer": "Reviewer_S55a", "summary": "To address the significant theoretical challenges of integrating multiple reference models into reinforcement learning with human feedback (RLHF) frameworks, this paper provides the exact solution to the multiple reference model problem under reverse KL-regularized RLHF. A comprehensive theoretical framework is proposed, incorporating rigorous statistical analysis and offering theoretical guarantees on sample complexity. The analysis is further extended to forward KL-regularized RLHF, presenting new insights into sample complexity requirements in multiple reference scenarios. The main contributions are as follows:\n\n1. A comprehensive mathematical framework is proposed for reverse KL-regularized RLHF with multiple reference models, providing the exact solution to this problem and calculating the maximum objective value.\n\n2. Theoretical guarantees are established for the proposed multiple reference model setting under reverse KL-regularization.\n\n3. The multiple reference model scenario is also studied under forward KL-regularized RLHF, and its sample complexity is analyzed.", "questions": "For detailed comments, please refer to the Weakness section. If the authors could clarify the issues raised therein, I would be happy to consider raising the score.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The theoretical analysis is novel, particularly in the authors' innovative and comprehensive investigation of the sub-optimal gap and optimal gap for both forward KL and backward KL regularizations, effectively filling a gap in the existing theoretical literature.\n2. The theoretical results are well supported by experimental validation. The authors designed dual-scenario experiments to verify that their closed-form solution outperforms existing approximate methods and single-reference models.\n\nWeaknesses:\n\n1. The theoretical results are only validated on two relatively small models and on specific tasks, without involving larger-scale models. This limitation may affect the generality and rigour of the conclusions.\n2. Although the theoretical proofs are clear and rigorous, the paper lacks intuitive explanations of the theoretical results. It would be valuable if the authors could provide more intuitive interpretations or insights into their theoretical findings.", "was_revised": true, "final_justification": "I appreciate the authors’ thorough follow-up, which resolved my concerns. Accordingly, I have raised my score to 5 to show my support for accepting this paper.", "rebuttal": "We thank the reviewer for their comments, and generally positive assessment of the paper. We will address their concerns as detailed below. \n\n----\n\n>  **The theoretical results are only validated on two relatively small models and on specific tasks, without involving larger-scale models. This limitation may affect the generality and rigour of the conclusions.**\n\n**R1**: *Experiment with larger model:*  We provide an additional experiment, this time with a larger model - Qwen 2.5 7B. We train this model on the UltraFeedback [cite] dataset, and evaluate the trained model's win rate against the preferred answer in the test split of the dataset using GPT-4o as LLM-as-a-Judge. We follow the standard protocol of first performing SFT on the dataset before the DPO step.\nAs a second reference, we used Qwen 2.5 14B Instruct. Due to the time and resource limit, we only trained a single model at $\\alpha=0.5$.\n\n| Model / Setting                              | Score |\n|----------------------------------------------|-------|\n| Base model (Qwen 2.5 7B)           | 8.6 % |\n| SFT model                                    | 43.4 % |\n| DPO (single reference – SFT model)           | 56.4 % |\n| DPO (single reference – 14 B model)          | 59.8 % |\n| **Ours (DPO with both references Eq.60)**          | **66.1 %** |\n\nWe can observe that using the geometric mean of reference policies as the reference policy in Eq.60, we can achieve better performance with DPO based on both references (Eq.60). \n\n> **Although the theoretical proofs are clear and rigorous, the paper lacks intuitive explanations of the theoretical results. It would be valuable if the authors could provide more intuitive interpretations or insights into their theoretical findings.**\n\n**R2:** Thank you for the helpful comments. In Appendix G, we provide additional discussion of our theoretical results, we plan to move this appendix to main body in final manuscript, including: (i) a detailed examination of the coverage assumption, (ii) a comparison in terms of other relevant parameters, (iii) an analysis of the reference policy in the multiple-reference setting under both FKL and RKL, and (iv) a comparison between RKL and FKL. Furthermore, regarding the sample complexity analysis, we can observe the upper bound on the suboptimal gap of RKL has a sharper convergence rate $O(1/n)$ in comparison with FKL. Regarding coverage assumption, the global coverage requirement for the RKL-learned policy corresponds to the *geometric mean* of the individual coverage levels between each reference model and the learned policy. In contrast, for FKL, the global coverage requirement is characterized by the *arithmetic mean* of the coverage levels. This distinction highlights how the form of the divergence influences the overall support overlap needed for effective learning."}, {"review_id": "RKH2YmzOtW", "reviewer": "Reviewer_Gq4o", "summary": "This paper studies RLHF with multiple reference models. Traditionally, RLHF is trained with a single reference model. But that is over restrictive because the model can inherits biases of that single pre-trained source. On the other hand, it is also inefficient because there are a lot of pre-trained models available. So far, there is a gap in understanding this new training paradigm because unlike RLHF with single reference model, there’s no exact solution yet. This paper pushes in this direction and provide solutions for RLHF with multiple reference models for both reserver and forward KL constraint.", "questions": "- Line 113-114: It’s also possible that two outputs are sampled from a distribution different from the reference policy? An example of which would be the Antropic HH dataset which is collected offline. And how are outputs sampled with multiple reference polices?\n- How realistic is assumption 4.2? Typically reward function are trained based on existing LLMs. For example, one could train the last linear layer of a pre-trained LLM. In that case, the VC dimension is finite but the reward function class is not finite.\n- I am always a bit confused about the practicality of coverage assumption. It seems like because outputs are sampled from reference policy, I don’t need lower bound of all \\pi_ref(y|x) but just the ones that can be realistically sampled? Although I do acknowledge the technical difficulties involved.\n- If the assumptions in 4.2 is not needed for 5.1, maybe move the placement of assumptions?\n- In the proof of theorem 5.1, you mentioned that “Clearly, the right side of (29) is maximized when the KL divergence is set to zero.” Is it possible that there exists a policy with non-zero KL divergence but high exponential value? (I'll update my score based on answer to this question since I might have missed something obvious)\n- Does theorem 5.1 only works with optimal reward?\n- I am confused about what the actual algorithm is? How does it rely on the exact solution of RLHF with multiple reference models? The experiment section is written with a lot of details missing.", "limitations": "- I have questions about the limitations of assumptions which are discussed in the question section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- Strengthes\n    - Expect for the experimental section, I think the paper is written very clearly with solid motivation and the theory sections are easy to understand.\n    - The research direction on RLHF with multiple reference models is an interesting direction and this paper provides some theoretical justifications in that direction.\n- Weaknesses\n    - The experimental section can be expanded a bit. I am confused about what algorithm is introduced based on the theoretical results.", "was_revised": true, "final_justification": "I was confused about a technical detail and the confusion is cleared.", "rebuttal": "We thank the reviewer for their comments and generally positive assessment of the paper. We will address their concerns in detail below. \n\n----\n\n> **In the proof of theorem 5.1, you mentioned that “Clearly, the right side of (29) is maximized when the KL divergence is set to zero.” Is it possible that there exists a policy with non-zero KL divergence but high exponential value? (I'll update my score based on answer to this question since I might have missed something obvious)**\n\n**R1:** Thank you for this question. Note that the second term in equation (29), i.e., the logarithm term, is independent of $\\\\pi(\\\\cdot|x)$. In particular, the logarithm term is just dependent on reference policies. As the KL divergence is a non-negative term, the right side of equation (29) is maximized when the KL divergence is set to zero. Therefore, the final solution Eq.(25) holds. For more clarity, we will add this explanation in the proof of Theorem 5.1 in the final version.\n\n$$  \n \\\\begin{aligned}  \n&\\\\underset{Y\\\\sim\\\\pi\\\\big(\\\\cdot|x\\\\big)}{{\\\\mathbb{E}}}\\\\big\\[r\\_{\\\\theta^{\\\\star}}\\\\big(x,Y\\\\big)\\\\big\\]-\\\\frac{1}{\\\\gamma}\\\\Big(\\\\sum\\_{i=1}^{K}\\\\alpha\\_{i} \\\\mathrm{KL}\\\\big(\\\\pi (\\\\cdot | x)\\\\| \\\\pi\\_{\\\\mathrm{ref},i}(\\\\cdot | x)\\\\big)\\\\Big) \\\\\\\\& \\= \\\\frac{1}{\\\\gamma}\\\\left(-\\\\mathrm{KL}(\\\\pi(\\\\cdot|x)\\\\|\\\\pi\\_{\\\\theta^*}^\\\\gamma\\\\big(y|x\\\\big))+\\\\log \\\\left(\\\\sum\\_y \\\\prod\\_{i=1}^K \\\\pi\\_{\\\\mathrm{ref},i}^{\\\\alpha\\_i}(y|x)\\\\exp\\\\left(\\\\gamma r(x,y) \\\\right)\\\\right) \\\\right),   \\qquad\\qquad \\text{Eq.}(29)\n \\\\end{aligned}  \n$$\n\n\n> **Line 113-114: It’s also possible that two outputs are sampled from a distribution different from the reference policy? An example of which would be the Antropic HH dataset which is collected offline. And how are outputs sampled with multiple reference polices?**\n\n**R2:** Thanks for raising this point. Sampling preference datasets from the reference policy is a common assumption in theoretical analysis of RLHF and DPO \\[1,2,3,4\\]. Offline datasets (e.g., Anthropic HH) are valid so long as the behavior policy (the underlying policy for the preference datasets) covers the reference policy in KL divergence. We clarify this point in the final manuscript. \n\n> **How realistic is assumption 4.2? Typically reward function are trained based on existing LLMs. For example, one could train the last linear layer of a pre-trained LLM. In that case, the VC-dimension is finite but the reward function class is not finite.**\n\n**R3:** Assumption 4.2 (“finite reward class”) is a technical assumption for our regret analysis.** Our proofs only use finiteness to obtain $\\\\log(|\\\\mathcal{R}|)$ factor in Lemma B.1 and theorems; the results can be extended to **classes with finite capacity** (e.g., finite covering number or pseudo/VC dimension) as discussed in Appendix C. It can also be applied to **a learned linear head on fixed LLM features (pre-trained LLM).** The rates in our bounds are unchanged; only the $\\\\log(|\\\\mathcal{R}|)$ term is replaced by the corresponding capacity term. We already discussed to this relaxation in Appendix C and will make it explicit in the final manuscript.\n\n> **I am always a bit confused about the practicality of coverage assumption. It seems like because outputs are sampled from reference policy, I don’t need lower bound of all $\\\\pi\\_{ref}(y|x)$ but just  \nthe ones that can be realistically sampled? Although I do acknowledge the technical difficulties involved.**\n\n**R4:** The coverage assumption is a uniform assumption which is needed for regret analysis. It is also assumed in \\[2,3\\]. Note that under integral over all samples, we apply this assumption. Therefore, we need to assume a lower bound on $\\\\pi\\_{ref}(y|x)$ for all $y$.\n\n> **If the assumptions in 4.2 is not needed for 5.1, maybe move the placement of assumptions?**\n\n**R5:** Thank you for your suggestion. We also mentioned in Lines 208-209 that “Note that this result does not rely on the assumptions stated in Subsection 4.2 and in fact holds in greater generality.” We clarified this point in the final manuscript.\n\n\n> **Does theorem 5.1 only works with optimal reward?**\n\n**R6:** Thanks for your great suggestion. Our result in Theorem 5.1 is general and can work with any reward function. We clarify this point in the final manuscript.\n\n> **I am confused about what the actual algorithm is? How does it rely on the exact solution of RLHF with multiple reference models? The experiment section is written with a lot of details missing. I am confused about what algorithm is introduced based on the theoretical results.**\n\n**R7:**  Building on **Theorems 5.1** and **6.1**, we introduce two algorithms:\n\n- **Algorithm 1 — RKL**  \n  Replaces the single reference policy with the **geometric mean** of all reference policies for RLHF.  \n  *See Eq.&nbsp;60 in **Appendix F** for the explicit DPO loss.*\n\n- **Algorithm 2 — FKL**  \n  Uses the **arithmetic mean** of the reference policies, in line with Theorem 6.1.\n\n*The following table* (below) collects every objective used in our experiments. We acknowledge the inconvenience of having them spread all over the main paper due to page limits of the main submission, and we will clarify this discussion in the final version.\n\n| Experiment                                   | Equation                              |\n|----------------------------------------------|---------------------------------------|\n| DPO with multi-reference                     | Equation&nbsp;60 (Appendix&nbsp;F)    |\n| FKL DPO with multi-reference                 | Equation&nbsp;61 (Appendix&nbsp;F)    |\n| On‑policy RL with multi-reference            | Equation&nbsp;16                      |\n| On‑policy RL with FKL and multi-reference    | Equation&nbsp;20                      |\n\n\n   \n----\n\n**References:**\n\n\\[1\\]: Zhao, Heyang, et al. \"Sharp analysis for kl-regularized contextual bandits and rlhf.\"   \n\\[2\\]: Song, Y, et al. “The importance of online data: Understanding preference fine-tuning via coverage.”  \n\\[3\\]: Chang et al. “Dataset reset policy optimization for rlhf”  \n\\[4\\]: Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\"  \n\\[5\\]: Freedman, David A. \"On tail probabilities for martingales.\" *the Annals of Probability* (1975): 100-118."}]

Author rebuttal:

Meta-review: The paper presents a theoretical study on RLHF with multiple reference models. The main contribution of the paper is the novel theoretical analysis that provides the exact solution in the reverse KL-regularized RLHF with multiple reference models. After an effective rebuttal and discussion period, the reviewers are satisfied with the authors feedback and all appreciate the main contribution of the paper. Meanwhile the reviewers also pointed out some issues of the paper, mainly related to insufficient empirical validation of the results. Overall the paper meets the bar of NeurIPS, and thus I would like to recommend acceptance of the paper. The authors should provide a detailed revision addressing all issues raised by the reviewers.

AC
Decision: Accept (poster)