All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
Authors: Gokul Swamy
Sanjiban Choudhury
Wen Sun
Steven Wu
Drew Bagnell
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
RLHF
fine-tuning
Primary area: reinforcement learning
TL;DR: We provide theoretical and experimental support for the hypothesis that the value of RL in fine-tuning is fundamentally derived from generation-verification gaps.

Abstract: From a first-principles perspective, it may seem odd that the strongest results in foundation model fine-tuning (FT) are achieved via a relatively complex, two-stage training procedure. Specifically, one first trains a reward model (RM) on some dataset (e.g., human preferences) before using it to provide *online* feedback as part of a downstream reinforcement learning (RL) procedure, rather than directly optimizing the policy parameters on said dataset via *offline* maximum likelihood estimation. In fact, from an information-theoretic perspective, we can only *lose* information via passing through a reward model and cannot create any new information via on-policy sampling. To explain this discrepancy, we scrutinize several hypotheses on the value of RL in FT through both theoretical and empirical lenses. Of the hypotheses considered, we find the most support for the explanation that on problems with a *generation-verification gap*, *(1)* it is relatively easy to learn the relatively simple RM (*verifier*) from the preference data. Then, *(2)* the downstream RL procedure only returns policies (*generators*) that are optimal for such relatively simple verifiers. Thus, end-to-end, two-stage online FT only has to search over a reduced subset of the full space of policies, requiring less data than offline FT.

Published as a conference paper at ICLR 2026
ALL ROADS LEAD TO LIKELIHOOD: THE VALUE OF REINFORCEMENT LEARNING IN FINE-TUNING
Gokul Swamy Cameeiie Mellon University
Sanjiban Choudhury & Wen Sun Comell University
Zhiwei Steven Wu & J. Andrew Bagnell Carnegie Mellon University
ABSTRACT
From a first-principles perspective, it may seem odd that the strongest results foundation model fine-tuning (FT) are achieved via a relatively complex, two-stage training procedure. Specifically, one first trains reward model (RM) on some dataset (e.g., human preferences) before using it to provide online feedback part of a downstream reinforcement learning (RL) procedure, rather than directly optimizing the policy parameters said dataset via offtine maximum likelihood estimation. In fact, from an information thhooretihh perspective, we can only lose information via passing through a reward model and cannot create any infor- mation via on-policy sampling. To explain this discrepancy, we scrutinize several hypotheses the value of RL in FT through both theoretical and empirical lenses. Of the hypotheses considered, we find the most support for the explanation that on problems with gencraatonn-errifcatton gap, (I) is relatively easy to learn the relatively simple RM (verifier) from the preference data. Then, (2) the downstream RL procedure only returns policies (generators) that are optimal for such relatively simple verifiers. Thus, end-to-end, two-stage online FT only has to search over a reduced subset of the full space of policies, requiring less data than offline
INTRODUCTION
Whether refers to it as reinforcement learning from human feedback (RLHF, Christiano et al. (2017)), preference fine-tuning (PFT), or even "alignment," the last step in the training pipeline of wide variety of foundation models (FMs) is fundamentally concerned with raising the generation likelihood of preferred completions of a prompt relative to those of dis-preferred completions.
From this perspective, natural question may be why anything other than maximum likelihood estimation (MLE) - i.e., standard supervised learning - is needed for the PFT problem. Indeed, plethora of offiine approaches to PFT that directly optimize policy parameters via solving (regularized) classification problem on preference data have been proposed in the literature (c.g.. DPO (Rafailov et al., 2023), IPO (Azar et al., 2023). SLiC-HF (Zhao et al., 2023)).
However, when one looks at the training procedure of today's 's most capable models (Achiam et al., 2023; Team al., 2024; Dubey al., 2024), one almost always sees relatively complex two- stage procedure adopted instead. First, one learns reward model (RM) - i.e., a classifier on the preference data, before using to provide labels for downstream online reinforcement learning (RL) procedure that ultimately optimizes the policy's parameters (Bai et al., 2022; Ouyang et al., 2022; Stiennon et al., 2020; Gao et al., 2024a;b; Calandriello et al., 2024; Guo et al., 2024).
Across academic (Tajwar et al., 2024; Xu et al., 2024; Song et al., 2024a), industry (Tang et al., 2024), and open-source (Hugging Facc, 2024) comparisons, we robustly see the relatively complex, two-stage online techniques out-perform simpler purely offline approaches. More generally, online approaches to supervised fine-tum tine-tuning (SFT) have also been shown to out-perform vanilla next-token prediction (Sun& & van der Schaar, 2024; Chen et al., 2024; Wulfmeier al., 2024; Choudhury, 2025). Furthermore, recent models capable of complex reasoning (c.g., OpenAl's ol (Jacch et al., 2024) or DeepSeek's 's r1 (Guo et al., 2025)) still trained via on-policy RL rather than by using offline

---

Published as a conference paper at ICLR 2026
Figure 1: While offline preference fine-tuning (PFT) pro- cedures (e.g.. DPO (Rafailov et al., 2023)) directly opti- mize policy parameters via (regularized) maximum likeli- hood estimation (MLE, i.e., Tmle E II), online PFT proce- dures (e.g., standard RLHF (Christiano et al., 2017)) first fit reward model (RM) via MLE before using the RM to provide online feedback for a downstream reinforeement learning (RL) procedure. Empirically, this latter stosstgee procedure often works better, despite the loss of informa- tion caused by passing through RM. We find evidence that for problems with a generation-verificatiin gap, this result may be best explained by viewing the first stage as finding a relatively simple verifier / RM fsim E Rsim with the end-to search over generators policies now simplified to just the subset of policies that are optimal for relatively simple verifiers (i.e., Arhh E (T())sm)) C IT).
D
Aille
Film
RL
Rsim
[[(
II
R
Policies
Rewards (Verifiers)
(Generators)
MLE procedure, with academic investigations concurring (Chu et al., 2025; Setlur et al., 2025). Thus, we ask the following question:
Q: What the value of twwosttae, online FT if we just want to maximize data likelihood?
Part of the challenge of providing a satisfactory answer to this question lies in the difficulty of applying traditional arguments about the value of online RL FM post-training. First, the standard justification for interaction the learner being able to observe and therefore learn to recover from their mistakes (Ross et al., 2011; Swamy et al., 2021) - does not appear to be true prima facie unless we somehow give a language model the ability to delete tokens (Cundy & Ermon, 2023; Wulfmeier et al., 2024). Second, the fact that it is common practice to use reward models that are at least as large as the policy makes it difficult to naively apply classical arguments predicated simplicity of rewards when compared to policies. This is because such arguments often implicitly assume one is learning a reward model over relatively simple function class (e.g., Ng et al. (2000)).
the relative
Perhaps even more fundamentally, the data processing inequality (MacKay, 2003) tells us that we can only lose information when passing from the raw source of data to the reward model, and cannot create any information during on-policy sampling. Taken together, these points make it seem as though the "yellow-brick road" of online PFT may have been paved with pyrite rather than gold.
In response, we scrutinize a variety of hypotheses about the value of RL in PFT through theoretical and experimental lenses. We mostly ground our study in preference FT, but note that analogous arguments can be made for SFT and verifier-based RL settings. Our contributions are three-fold:
1. We prove that under idealized assumptions, online and offline PFT techniques should return policies of equivalent quality. Using tools from information geometry, we show that regardless of the coverage of preference dataset samples, offline and online PFT techniques have the same set of optima when the same function class is used for both policies and reward models.
We provide evidence against several prior/ novel hypotheses for the value of RL in PFT. In particular, we provide evidence against explanations that rest solely on online techniques performing better regularization to the reference policy, computational benefit provided by on-policy sampling, or the ability to use wider distribution of data to train a reward model than to train policy. While it is hard to conclusively rule out these factors, we provide evidence that they are not the whole story.
3. We provide theoretical and empirical evidence for alternative hypothesis gap. Many problems in computer science are widely conjectured to
problems
with a ceueratioonverifiiaan have simpler verifiers/ reward models than generators/ / optimal policies (Godel, 1956; Cook, 2023). We hypothesize that for such problems, it is easier to learn the relatively simple reward model than the relatively complex optimal policy from the preference data. RL then merely serves as way to compute a (soft) optimal policy for this simple verifier. However, end-to-end, online PFT only has to search over just those policies that are optimal for relatively simple verifiers. In short, we argue that:
h: The value of two-stage interactive FT is derived from an end-to-end reduction the space of policies to search over just those that are optimal for relatively simple verifiers.
2

---

Published as a conference paper at ICLR 2026
In the language of statistical learning, this hypothesis states that the real benefit of RL in fine-tuning is that it is the most convenient method that we know of for performing proper learning (Valiant, 1984), compared to improper offline FT. We find the least evidence against this hypothesis compared to all others we consider in this paper, which in some sense is the best we can hope for (Popper, 2014).
2
ON THE INFORMATION GEOMETRY OF FT
We begin with notation before deriving our core results. All proof' are located in Appendix B.
2.1 PRELIMINARIES
We consider finite-horizon, reward-free Markov Decision Process (MDP, Puterman (2014)). Let X denote the set of initial states (i.e., prompts) and po their distribution. We use A to denote the action space (i.e., set of tokens) and S to denote the state space (i.e., set of partial generations). The dynamics of our MDP are deterministic, known, and tree-structured: T(s'ls, a) = 1 if s' = soa and 0 otherwise (i.e., we can only append tokens). We use H to denote the horizon of our MDP (i.e., maximum generation length). A policy S - A(AA maps from a prefix to a distribution over next tokens. trajectory (i.e., prompt and its completion) & is generated by sampling an initial state so ~ Po and then sampling from the policy H times. We use Prr (5) to denote the probability of sampling trajectory& & under policy æ and use Fn to denote the set of all such distributions over trajectories generations induced by any II C {S + A(A)]. We use from such distribution (i.e., sampling generation from the policy). We use D = ={( 6,- ) to denote a dataset of trajectory-level preferences over pairs with the same so and PD > {2 to denote the empirical probability that & is preferred to &2. The full space of trajectories is denoted by E, and Suck is used to denote the set of trajectories with some prefix 8y:h- We use ref E II to denote the reference policy to stay close to. Both arg maxs and arg mins return the full set of optima.
π shorthand for sampling
2.2 GLOBAL AND LOCAL REWARD MODELS
Central to our study will the relationship between policies and trajectory-level reward models. We will use II denote the set of policies and R to denote the set of reward models, with r + R for all r E R. We note that it is standard practice to use the same architecture (and often the same initial checkpoint and dataset) train both policies and reward models. Specifically, reward models are often constructed by removing the final softmax at the end of the transformer (Vaswani et al., 2017) that gives us distribution over tokens and replacing it with a single layer or a shallow MLP that eventually outputs a single scalar value (Rafailov et al., 2023). These models are then evaluated with the concatenated prompt and entire completion HH being passed into the transformer. Throughout this paper, we will use the term global to refer to such trajectory-level (i.e., non-Markovian) RMs.
Beyond merely having a shared backbone, a more precise isomorphism holds for reward models that take the form of the sum of policy log probabilities over the tokens in generation (Degrave et al., 2019; Rafailov et al., 2023). More formally, we denote the set of local RMs Ta :E + R as
{ rT (5) = Σ log x x (anlla))
R(II)
A=0
It directly follows that RIII) is isomorphic to II.
2.3 A UNIFIED OBJECTIVE FOR TIN-TUNING
We now formulate a general loss function that both online and offline (P)FT methods optimize, albeit with different procedures. Loosely speaking, a variety of fine-tuning tasks (e.g.. SFT, PFT) roughly fit within the template of the following reverse KL-regularized policy optimization problem:
n arg min EgED
[DRr. (PD()) + ,3 DRI.. (P+(E)
(E) [PT
Fiat (E)),
(2)
Tlll
Data Likelihood
Peior Reg.
where 2 E Z denotes an ordered pair of trajectories for PFT. The first forward KL term measures how likely samples from D are under the learned policy and the second reverse KL term keeps the
3

---

Published as a conference paper at ICLR 2026
Daa (Piii)
Du. (DIR))
Figure 2: We can view both offline and offline PFT as solv- ing Eq. Offline methods directly project the preference dataset D onto policy class II under forward KL ($2.4), while online methods first project from D onto reward class R. under forward KI. ($2.4), before projecting onto IT under reverse KI. ($2.5). A similar characterization of online PFT is made in the concurrent work of Xiao et al. (2025).
D
mie
File
22 mla)
II
R
completion probabilities of to be close to those of the reference policy ree on-policy. Intuitively, had full coverage over all possible 2, we would have no need for the second term, but due to sample limitations, we add in regularizer to prevent us from being lead too far astray (Gao et al., 2023; Song et al., 2024a). That said, for simplicity of presentation, we will consider the case where B = 1 and temporarily replace the second KL. regularization term with entropy regularization:
arg min ESDD #ell
[Dx (PDD
H[))),
(3)
where (T) EEEUT Σx H
log x (a (hh) is the (causal) entropy of the policy (Ziebart, 2010).
2.4
MAXIMUM LIKELIHOOD IN PFT
We will first focus minimizing the forward KL (FKL) term of (3), which is equivalent to maximum likelihood estimation (MLE). Given this MLE objective, both online and offline methods aim to find model that best explains the data, but search over different model classes. The first stage of online PFT methods is MILE over reward models in R. common parametric model that relates rewards to preference probabilities is the Bradley-Terry (BT) Model (Bradley & Terry, 1952):
PPT > So) = a (r(E) - -r(5))
(4)
where > means preferred to and a is the sigmoid function. We can then maximize likelihood over global reward models, each of which parametrizes probabilistic model via BT:
mmle = arg min (E (ID(() > E2lso)ll > E2(e)))]
(5)
PER
N
arg max
Σ
(
(6)
FR
Observe that fitting a global RM essentially solving standard logistic regression or classification problem. In contrast, offline PFT methods instead perform MLE directly over the policy class II.
Recall that any policy π E IT is also a local RM rx E (ITT) (defined in (1)). Thus, we can fit a policy via MLE by substituting in the sum of log probabilities for rs in the Bradley-Terry likelihood:
mlle arg min E (ELEE))~ [KL > {2]so)][ P (51 > E2(so))
#Ell
arg max TEll
Σ log -rr
(8)
arg max TEN
Σ log o
Σ log (,
arg max 4 (TT). REl
(9)
Ignoring the reference policy for a moment, this is what offline PFT approaches like DPO are at heart: a trajectory-level classification problem over local (rather than global) reward models.
2.5 MAXIMUM ENTROPY IN PFT
The second stage of online PFT corresponds to optimizing a learned reward and the second entropy term in (3), Given learned global reward model one computes its soft-optimal policy गØ. i.e.
π = arg max Eg [r(E)] + H(T).
(10)
xell
4

---

Published as a conference paper at ICLR 2026
If we solved the above equation in closed-fomm over all policies (i.e., not just those in II), would compute a policy with prompt-conditioned trajectory (i.e., completion) distribution
exp(r(E)) Egrea exp(r(e))
exp(r(E))
(C] =
(11)
Z(r, su)
as first proved by Ziebart (2010). Observe that for trajectories &1, E2 with shared prompt so.
( > =2 =
- (E2()))))))))
(12)
Interestingly, turns out that solving soft RL problem like Eq. 10 can be thought of as a reverse KL projection (Nielsen, 2020) from P onto the set of policy-induced trajectory distributions.
Lemma 2.1 (Soft RL is an RKL Projection). Let & R be global or local reward model. Define π arg ein Dxr.PP (F as the trajectory-level reverse KL projection of Fr onto II. Next, define n as the soft-optimal policy computed by solving Eq. 10 over II. Then, x = [Proof]
prove version with a reverse KL. regularization to a reference policy the appendix. summary, view the online two-stage RLHF procedure as (1) minimizing the forward KL from the data over reward models in R to find fmle. before (2) minimizing the trajectory-level reverse KL over policies in II to the soft-optimal policy under mmee- We summarize this argument visually in Fig. 2.
2.6 EQUIVALENCES WITH ISOMORPHIC CLASSES
The preceding subsections beg the question: what does taking detour through a reward model buy us if we immediately project back to policy space? As we prove below, under certain assumptions, all we have done is taken more circuitous route to likelihood maximization. We now state our first equivalence result: assuming exact optimization, no reference policy, and most critically, isomorphic reward and policy classes, online and offline PFT will produce the same solution:
Theorem 2.2 (RLHF is MLE when R = R(II)). Assume that R = R(11) (i.e. they both cover the same set of reward functions regardless of representation). Define i mle in Eq. 6, mie as in Eq. and mm arg ma Ex-= mie ({) + H( mmle). Then, we have Ramt fi [Proof]
In more traditional terminology, MLE is invariant to re- re-parameterization. Under a realizability assumption,
prove an analogous result with regularization reference policy:
Theorem 2.3 (RLHF is DPO when R = =I(II). Assume that R R(II) (i.e. they both cover the same set of reward functions regardless of how they are represented). Define filee as Eq. and
N
H
TT77,1155.1)
(xpo(=)) = Σ logo a
Σ log
log
Trer (0,, 15t.))
T (raraa 17,,)
Next, define àop = arg maxxenl (T)) Tapo = arg max max 7 (dpo(TT), Timle
arg (( (T)e(T), and Rrtnt {arg max (Fme(() Dxx (P. -PP.PP1PPP..... ( E ('le Then, if π mio doo C 11, we have Rit म dpp [Proof]
In summary, the preceding results tell us that under certain assumptions, all roads lead to likelihood i... expending computation for on-policy sampling provides no discernible benefit over offline MLE.
We now attempt square these idealized theoretical results with the reality of empirical practice.
3 ON THE VALUE OF REINFORCEMENT LEARNING IN FINE-TUNING
To better understand where the preceding theory falls short, now proceed by performing series of controlled experiments. We focus on the task of learning to summarize from preference feedback (Stiennon al., 2020) (tl; dr) the pythia series of models (Biderman et al., 2023). We report the winrate of our trained models against human-gent -ennrrtte references, as evaluated by gpt-4o.
Controlling for Confounders. To make apple-to-apples comparison between online and offline PFT methods, we make a concerted effort to hold factors other than on-policy feedback constant. First, rule out the confounder of different loss function for offline (e.g., DPO (Rafailov et al.,
5

---

Published as a conference paper at ICLR 2026
2023)) and online (c.g., PPO (Schulman et al., 2017)) PFT, we use the same DPO loss (Rafailov et al., 2023) for all training, both because of its practical ubiquity and because there is no clear way to use an RM in offline PFT. Thus, our experiments focus on (offline) DPO (as originally proposed) vs. online DPO. Second, we follow standard practice and train global RMs using the same preference data and starting from the same SFT checkpoint as we use for offline DPO, with single additional linear layer bolted on for global RMs (Rafailov et al., 2023). Thus, our experiments can be seen as a relatively faithful implementation of the "isomorphic classes" setting we explored theoretically in $2.
For the second stage of online PFT, we perform online DPO (Guo et al., 2024). Concretely, starting from the base policy, we sample 25 completions for each prompt the preference dataset, rank these completions according to the RM, and use the top and bottom of the list as the preferred and dis- preferred completions for DPO loss minimization. We regularize to whatever policy generated the data. We emphasize that the only change between offiin and online DPO is the training data we use all of the same hyperparameters for all training runs. Below, we use Onl inn DPO (Model) to refer to online DPO using on-policy samples from the policy in parentheses.
Online DPO Offline DPO. In Figure 3, we see that despite our best efforts to control for confound- ing variables, we see significant gap between online and offline DPO, contradicting $2's theory. While any experiment will have imperfect optimization unlike we assumed in $2, we note it is equally imperfect across both online and offline DPO, and therefore unlikely to be the differentiating factor.
Vinrate
pythis-1 4B 11:d2
4e Vinrstw(
pythia-2.ce 88 s1/dr
Figure 3: We see online PFT robustly out- perform offline PFT on t1; dr as judged by gpt 4c, despite using the same SFT checkpoint and preference data for train- ing all policies reward models. We report winrates for all methods across 3 seeds.
00.0
50.3
52.4
96.9
32.5
3FT
D0C
BFD
Do. 160
0s
DP0
(260
(9P7)
(DP0)
Perhaps the most immediate question one might have is whether we are simply running more gradient steps for online DPO than for offline DPO. However, both DPO and Online DPO (SFT) start from the same SFT model and take the same number of gradient steps, yet we see a difference in performance. The same is true for DPO (2x) (in which we perform two epochs of offline DPO) and Onl nin DPO (DPO) (in which we perform online DPO starting from the output of offline DPO). Also noteworthy is the improved performance of Online DPO (DPO) over the base DPO policy: without any extra human feedback/ / data, a global RM trained on the same preference data as was used for DPO is able further improve the DPO policy, In some sense, there appears be more "juice to squeeze" from RMs than from policies, even when both are trained on the same human data.
In short, the results of our preceding carefully controlled experiments echo those of prior work: online PFT robustly outperforms offline PFT (Tajwar et al., 2024; Xu et al., 2024; Song et al., 2024a; Tang et al., 2024; HuggingFace, 2024). We note analogous results have been found in the SFT (Sun & van der Schaar, 2024; Chen et al., 2024; Wulfmeier et al., 2024) and reasoning (Chu et al., 2025; Setlur et al., 2025; Xiang et al., 2025) domains, suggesting a general phenomena beyond just PFT.
Appendix A: A Quintet of Hypotheses for the Online-Offtine Gap.
To explain this performance gap, we explore 6 hypotheses in total. Due to limited space, we highlight only the hypothesis that we failed falsify, while relegating all the other 5 to App. A. There, we present controlled experiments that rule out alternative explanations for the gap, including Hj: Intrinsic Value of Online Samples ($A.1), 1), Hy: Failure of Offline PFT Regularization to Tret (SA.2), Hgg: Relative Ease of Online PFT Optimization ($A.3), H4: Global RMs Can Be Trained on More Data ($A.4), and H5: Global RMs Generalize Better OOD (SA.5). To separation, online PFT out-performs offline PFT across different sampling distributions, labelers, and model sizes. Furthermore, it appears to be "casier" to learn a global RM than it is to learn a local RM, evidenced by better ID and OOD generalization. Together, our theoretical and empirical results beg the question of whether there is some feature of the problem we not accounting for in our theory.
summarize App. A, we observe that despite the lack of information-theoree
'Noe that the choice of the DPO loss rules out explanations that rest reward model variance (Razin al., 2025), as no algorithm (either offline or online) gets see raw reward model outputs, only hinarized labels.

---

Published as a conference paper at ICLR 2026
4
GAPS IN FINE-TUNING
To set up our final hypothesis Hils, we first note that for many problems of practical interest, the reward function a simpler object (i.e., can be represented by circuit of lower depth) than the corresponding (soft) optimal policy. For example, this is the heart of the classical argument for the inverse RL approach imitation (i.e., learning reward model from demonstrations and decoding via RL) over behavioral cloning (i.e., directly learning policy via MLE) given by Ng et al. (2000).
By framing policies as generators and reward models as verifiers, this argument can be viewed as part of widespread phenomenon in computer science, where generation is often harder than verification (Godel, 1956). Indeed, if the widely-held belief that P A NP (Cook, 2023) true, there must then exist such problems. For these problems in NP (loosely, polynomial-time verification) but not (loosely, polynomial-time generation), we know there must exist a polynomial-depth verifier circuit but not necessarily a polynomial then tells us that we would need fewer samples to accurately fit a verifier than a generator, as a less expressive function class could be used to approximate the former (Vapnik & Chervonenkis, 2015).
generator circuit (Cook, 2000). Uniform convergence
While initially promising, the story becomes more complex when we account for the facts that (a) uniform convergence often does not accurately predict the behavior of deep learning (Nagarajan & Kolter, 2019) and (b) even if it did, in the isomorphic classes setting (i.e., R = ).),, we have the same number of verifiers / generators to search over. This means that we should have the same sample complexity for both MLE problems. Indeed, Ng et al. (2000) implicitly assume that |R| << |I|].
However, wide variety of works have observed that overparameterized models like deep neural networks, when optimized with stochastic gradient descent, learn lower-depth circuits with fewer samples than higher-depth circuits. One can attribute this to either an implicit regularization effect of the training algorithm (e.g., the bias of SGD towards minimum norm solutions (Soudry et al., 2024; Gunasekar et al., 2017: Li et al., 2018)), architectural inductive biases towards simpler functions (e.g., the double descent phenomenon (Belkin et al., 2019; Nakkiran et al., 2021), the spectral bias of deep networks (Rahaman et al., 2019), or those mentioned in some explanations of grokking (Varma et al., 2023) and length generalization (Zhou et al., 2023)), deep learning performing approximate Solomonoff Induction (Schulman, 2023), or certain data- dettaddeenndntt complexity measures (Arora et al., 2019). Regardless of one's preferred reasoning to why, using a larger network than necessary often doesn't hurt sample complexity in practice (Krizhevsky et al., 2012 2; Radford et al., 2019).
Building on these observations, now propose a novel hypothesis Hg for the online-offline perfor- mance gap on problems with (/) a inmeration-verificann gap and (2) a reward model class R where simpler functions can be learned with fewer samples (e.g., deep neural networks like transformers):
FR Online PFT is Proper Policy Learning.
For fine-tuning problems with simpler underlying reward function than (soft) optimal policy and a reward model class R that enables learning simpler functions with fewer samples, the first step of online fine-tuning is finding relatively simple reward model sim E Reim R. with the second step then finding a (soft) optinal policy for felm. fim E (Rm) Thus, end to end, online fine-tuming only has to choose between policies in (Rm) II, rather than across all of II.
In essence, the above hypothesis stating that offline FT solves the harder, improper (Valiant, 1984) learning problem over all of II, while online FT only has to solve the easier proper learning problem over (Rim C II, thereby reducing sample complexity and leading to better policy performance. Put differently, Hig states that there is statistical separation between online and offline (P)FT.
Under the above hypothesis, if we could somehow constrain offline FT to only pick policies that are (soft) optimal for relatively simple reward models (i.e., mlle E [l(RRim)), we would achieve the same results as online FT. We can prove this under (slightly) weaker assumptions than isomorphic classes:
In various theoretical settings, one can show there is a computational speedup from improper learning (e.g.. Shalev-Shwards et al. (2012)). However, such relaxation can incur significant sample complexity increase if the encompassing class we now have to search over is large enough. Another perspective on Hg is that the camputatiinallstatisitii trade-off may not be in favor of improper learning on practical PFT problems.
7

---

Published as a conference paper at ICLR 2026
Theorem 4.1 (RLHF is MLE Over (I(mm)))). Let Raim R be arbitrary subset of the space of reward models and let
(R
arg min Dxy (x=(P=.|PP)) Raim"
sell
be the corresponding set of soft-optimal policies. Also, let [Kx (PPp(E > E2)s0)11== (EI > E2(s0))
Ssim
arg
min B(g,
yiht
arg max EgNT (F (m) + HT) isim E fsim
wEll
it sim
arg min (E (Px falss)][[? Ealsu))) =cll(Ram)
Then, if { क. r Rsi
II. Triht = ftgm [Proof]
In words, this theorem is saying that if the secondary, RL-based reverse KL. projection is without loss, RLHF will recover the MLE over the constrained policy space (imms Critically, it is unclear how else one could actually enforce this constraint other than by the two-stage procedure of first learning relatively simple RM before optimizing it via RL. Intuitively, we can view this hypothesis as stating that while all roads lead to likelihood, RL on simple RM lets us take shortcut through IT.
4.1 FAILING TO FALSIFY HG
First, one may naturally ask what evidence have for verification being easier than generation for summarization problems. In response, we provide evidence that the underlying reward function is well-approximated by a smaller network than one needs for the policy. In Fig. we observe that using a global RM that is significantly smaller than the generation policy leads to nearly identical BoN performance as using an RM that is the same size as the policy. The converse claim is also true: using global RM that is significantly larger than the generation policy leads to no discernible improvement in terms of BoN performance. Taken together, these experiments suggest that the verifier is qualitatively simpler (i.e., well-approximated by a shallower depth circuit) than the generator.
80t-1o Ninrste(
50
pythis-6. 6.98 Boll: SFT
g0nt 440 1ovoooato
pythia-1.48 Bol: SFT
Figure 4: Scaling up re- ward model size doesn't 'tt improve BoN perfor- mance, but scaling up policy size does. Re- sults across seeds.
0lobol 1.45
43
38-
01obal 98
80
Baoc
28
12
next observe that H., correctly predicts the experimental results in App. A, which falsified
the other hypotheses. In short, none of these experiments altered the relative complexity of generation versus verification of the underlying problem, which explains why we consistently saw a performance gap. First, the effective reduction in search space explains the improved performance Next, the generation-verifee verifiicaiin gap is unchanged by prompt augmentation (Fig. 8), different sample and label distributions (Fig. 9), and provides concrete explanation for the observed difference in in-distribution validation likelihoods between local and global RMs (Fig. 10): global RMs only need learn simpler reward function, while local RMs must learn a fundamentally more complex object (equivalent to a generator) to effectively fit the training set. Such better in in-distribution margins would likely correlate with the better OOD generalization of global RMs relative to local RMs in Fig. 11.
in Fig. 3.
To further attempt to falsify H.s we make a testable prediction: on tasks where there is limited if any generation- veriiiinatiiiiiiiinnnn gap, online PFT would be unlikely out-perform offline PFT. For example, if we were to use a reward function for grading the quality of summarization that is as complicated
An interesting direction of future work is off-policy RI. algorithms that take advantage of learned RMs.
8

---

Published as a conference paper at ICLR 2026
as the corresponding (soft) optimal policy, we would expect the previously consistent gap between online and offline PFT to vanish. We explore 2 ways of eliminating the innerration-vrificaati gap:
pythia-1.48 Twa-Word
pythia-1. ROUGE-I.
gpt-4o Minrate
Val. ROOCE-LA L(())
21.9
23.2
0,359
0.354
0.352
6.3
SPT
DPQ
DP. (DPO)
D20
Da. DP.
DPQ
(DPG)
(2x)
Figure 5; On the bandit-like task of two-word summarization (== 1 of the usual horizon H), 10 we see online PFT provide a minimal at best improvement of = 1% winrate over offline PFT, unlike all prior results. Results across 3 seeds.
Figure 6: When we label data and evaluate com- pletions with the ROUGE-L metric against unob- served reference summaries, we see online PFT provided no improvement over offline PFT, un- like all prior results. Results across 3 seeds.
Closing the compared to that of verification is to reduce the horizon H of the problem. In the extreme of H = 1 (i.e.. a contextual bandit (Li et al., 2010)), the complexity of learning the reward function and of learning the (soft) optimal policy should be the same the former with a simple action-level/ / "token-wise" softmax and vice-versa (i.e., no real multi-step planning / reinforcement learning required). Intuitively, setting H = 1 is simplifying the policy.
-errificinnn Gap One way to reduce the relative complexity of generation
directly compute the latter from
We instantiate this idea with the task of generating hwo-word summaries (i.e., less than TO preceding maximum generation length). Given don't have ground-truth human preferences for this task, we use gpt 4o to generate pairs of summaries, rank these summaries, provide reference summaries on the test set of prompts, and compute the final winrate numbers. before, we use this data for offline DPO as well as for learning the global RM. Continuing to parallel the above, we then sample from the offline DPO policy, rank these completions with the global RM, and use this dataset for an iteration of online DPO. In Figure 5, we see that in contrast to all prior experiments that kept the complexity of generation higher than that of verification, we see that online DPO does not meaningfully improve the performance of the offline DPO policy, as predicted by lil.
L of the
Closing the Generation- -vriffiaieti Gap II. Another way to eliminate the gap is by choosing a reward function from which we can easily "read off" the (soft) optimal policy without explicit planning reinforcement learning procedure. Specifically, we conduct an experiment using the ROUGE-L metric (Lin, 2004), which essentially counts how many words of the generated summary appear (in order) in an unobserved human- generated reference summary, as a reward function. For such problem, the minimal-complexity verifier would need to include lookup table that maps from prompts to the exact text of the unobserved reference summaries, a tall order. Furthermore, given this lookup table, would be trivial generate optimally, implying that generation and verification should be of similar complexities. Intuitively, this setup is complicating the reward reward, function.
To create our preference dataset, we sample from the policy and use the ROUGE-L metric rank samples, as well as for evaluation. In Figure 6, we see that doing an iteration of online DPO with feedback from a learned global RM does not improve the performance of the base offline DPO policy, unlike all but the previous result, predicted by Hg. Doing an extra epoch of offline DPO does (slightly) improve the ROUGE-L score, implying offline DPO has not reached optimal performance.
We conclude this section by noting that similar results on the benefits of learning a verifier rather than directly learning a generator for fine-tuning have been observed both the SFT (Sun & van der Schaar, 2024; Choudhury, 2025)- where this flavor of approach is more commonly known as inverse RL (Zicbart, 2010; Swamy et al., 2021) - and reasoning domains (Chu et al., 2025; Setlur et al., 2025; Xiang et al., 2025). These results can be seen further evidence for Hgg, outside of the PFT domain.
4.2 ISOMORPHISMS AREN'T ALWAYS A TWO-WAY STREET
A lingering question in the reader's s mind might be "if policies and rewards are isomorphic soft RI, why does it take more data to learn the former?" In particular, we know from Eq. 11 that

---

Published as a conference paper at ICLR 2026
can map from rewards to trajectory distributions, from which can extract policy via soft value iteration (Ziebart, 2010). Perhaps the most impressive insight of Rafailov et al. (2023) is that we can invert this isomorphism (up to the partition function Z(r, so) which cancels out in the Bradley-Terry likelihood) and instead express a local reward model in terms of its implied soft-optimal policy, as in Eq. 9. However, as later observed by Rafailov al. (2024), it is more accurate think of a local RM as Q-function rather than as a reward function: fundamentally different and often more complex object that needs to perform token-level credit assignment and encode multi-step reasoning.
In greater detail, we know from Ziebart (2010) that ग.. has a (soft) Q-function given by
exp(Q; an)) Σ.EA exp(QF a))
(sh: an) = log
Σ
exp(r(E))
-., (anls))
(sh)
Observe that the next-token logits of softmax policy are its implied soft Q-function (up state- wise shift). Also, recall that these were the optimization variables in our offline MLE problem (Eq, 9). Thus, MLE over softmax policies (i.e., fitting local RMs) corresponds directly fitting Q-functions.
Defining OR 1Q E R} as the set of soft-Q functions for some r E R, we know that R. = Or by construction. However, just because such an isomorphism exists, it does not need be true that both endpoints are equally easy to represent: one endpoint (c.g.. 2 could require far deeper circuits to be well-approximated than the other (c.g.. R). When learning algorithms optimize over circuit classes that contain these function classes, uniform convergence would then tell us we need more samples to learn the former than the latter, as we have to search over eircuits of greater depth. Thus, even though R S Qi. the fact that we are actually fitting these functions from data means that we have to pay statistically for their relative representational complexities during offline MLE.
To build some intuition for the complexity of vs. Q let us consider navigating through a maze with a single goal with +1 reward. Then, a circuit that represents r only needs to encode the position of the goal, while circuit that represents Q, needs to encode the path to the goal from any position in the maze. As we scale up the horizon H of the planning problem, the optimal policy x, can reach the goal from wider set of states. Thus, Q: becomes more complex, as nonzero value is achieved on a broader swathe of the state space. As π., is a simple softmax over Q", it also grows in complexity.
SS max Qi(s,
Figure 7: Consider maze navigation. Reward r just needs encode the goal, while O7 needs to encode the distance the goal from anywhere. Q; grows max (7 a, 7) more complex with h.
r(s)
max a, 1) oCA
As we discussed in $4, such a difference in circuit complexity between r and ग. or Q; is one way of formalizing the notion of a generation- verifiiaivve gap. This means that beyond the examples explore in $4, App. A, and Fig. 7's maze example, for problems in NP but not P, we should expect that nे and Q; require deeper circuit to represent than r. Then, learning algorithm that doesn't 't have the ability to search over just Qi (c.g.. via RL on all E R) must instead search over class of deeper circuits containing O Thus, learning Q or मे requires more data than to learn r.
Intuitively, these RL problems in NP but not P require meaningful, multi-step reasoning to solve. However, not all RL problems require multi-step reasoning have a generation-verification gap. In fact, in 1, we provide multiple examples of such problems for which the greedy (H = 1) policy is optimal and don't 't see a difference in terms of the number of samples required learn vs. π. or Q;- These problems have an effective horizon (Laidlaw et al., 2023) of H ~ 1. Thus, r, T and Q; have similar circuit complexities and require similar amounts of data to learn, matching our results.
To close, one might counter that whether we first fit reward function and then apply RL to it or directly fit a Q function, we eventually want to end up with a policy. While true, recall that the second stage of online PFT requires no more preference data samples: we only pay in terms of computation. Thus, a different perspective on Hig is that online PFT trades data for computation: rather than paying statistically for directly fitting a potentially more complex Q function via MLE like offline PFT, online PFT instead pays computationally during the secondary RL procedure. One can view this as a computational-statsiie trade-off' (Chandrasekaran & Jordan, 2013) for preference fine-tuning.
10

---

Published as a conference paper at ICLR 2026
In summary, while isomorphisms go both ways, they aren't always two-way street: the two endpoints of an isomorphism can have dramatically different circuit complexities, which can manifest statistical cost when fitting the more complex endpoint (c.g., QR. Instead, one can fit the simpler endpoint (e.g.. g.. R) and compute the isomorphism (e.g., via RI.) to save data at the cost of compute.
5 DISCUSSION
Practically, Hg tells us that for problems where believe verification is simpler than generation, it a better use of limited human preference data to learn verifiers rather than generators. Hypothesis Hig also suggests that for increasingly complex problems that require longer-range planning (e.g. multi-turn RLHF, agentic tasks, or even real-world robotics), we should see an even greater separation between online and offline PFT than research directions. For example, one might ensure that current reward model architectures are able accurately represent human preferences. As argued by Swamy et al. (2024), the diversity rater views often results in intransitive preferences, which can't 't be rationalized by any reward model. More broadly, given our overarching framing in terms of likelihood maximization, analogous arguments to ours for H3 could be made beyond the PFT setting (e.g., for the SFT / imitation learning setting).
in our experiments. HH, also suggests variety of future
ACKNOWLEDGMENTS
We thank Geoff Gordon and Kiante Brantley for helpful discussions, and Zhaolin Gao and Yuda Song for experimental advice. GKS and 7SW were supported in part by STTR grant. WS acknowledges funding from NSF IIS-2154711, NSF CAREER 2339395, and DARPA LANCER: LeArning Network CybERagents. SC is supported in part by Google Faculty Research Award, OpenAl SuperAlignment Grant, ONR Young Investigator Award, NSF RI #2312956, and NSF FRR#2327973.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint 3XXx:2333308874 2023.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- aieeggraned analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332. PMIR, 2019.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello. Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. 0...//0.20666 /2036, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint
2022.
Afonso Bandeira, Amelia Perry, and Alexander Wein. Notes on computational-to -ow.atttiiooo---tttttiiittittt statisssti gaps: predictions using statistical physics. Portugaliae mathematica, 75(2): 1866, 2018.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-1555 15854, 2019.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle OäÄžBrien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.

---

Published as a conference paper at ICLR 2026
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. arXiv preprint a4X3v:2403.08635, 2024.
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Valle--ncettivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2445.119222 2024.
Venkat Chandrasekaran and Michael Jordan. Computational and statistical tradeoffs via convex relaxation. Proceedings of the National Academy of Sciences, 110(13):E1 181-E1 190, 2013.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01333 2024.
Sanjiban Choudhury. Process reward models for Ilm agents: Practical framework and directions, 2025. URL https:/ http:s///aavvv.vee ort. /abps////////ee /2002/225222222 10325.
Sanjiban Choudhury and Paloma Sodhi. Better than your teacher: Llm agents that learn from privileged ai feedback. arXiv preprint arXiv:2410.05434, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, r1 generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025.
Stephen Cook. The p versus np problem. Clay Mathematics Institute, 2(6):3, 2000.
Stephen Cook. The complexity of theorem-proving procedures. In Logic, automata, and computa- tional complexity: The works of Stephen A. Cook, pp. 143-152. 2023.
Chris Cundy and Stefano Ermon. Sequencematch: Imitation learning for autoregressive sequence modelling with backtracking. arXiv preprint arXiv:2300.055226, 2023.
Amit Daniely, Nati Linial, and Shai Shalev Shwartz. More data speeds up training time in learning halfspaces over sparse vectors, 2013. URL hwt.thtts///aiiiiieeeeeee
Jonas Degrave, Abbas Abdolmaleki, Jost Tobias Springenberg, Nicolas Heess, and Martin Riedmiller. Quinoa: a fuunctio you infer normalized over actions, 2019. URL htwps: http::/:///aax httpt::ss///xxx...oog./o abs/1911. 01831.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Jacob Eisenstein, Chirag Nagpal, Alckh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, al. Helping herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint 231::2331.00224,, 2023.
Adam Fisch, Jacob Eisenstein. Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405. 201124055.116.... /9316, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pp. 10835-10866. PMLR, 2023.
Zhaolin Gao, Jonathan D Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, Andrew Bagnell, Jason D Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. arXiv preprint 4rXiv:240..16767 2024a.
12

---

Published as a conference paper at ICLR 2026
Zhaolin Gao, Wenhao Zhan, Jonathan D Chang. Gokul Swamy, Kianté Brantley, Jason D Lee, and Wen Sun. Regressing the relative future: Efficient policy optimization for multi-turn rlhf. arXiv preprint arXiv:2410.04612. 2024b.
Kurt Godel. Letter to john von neumann, 1956. URL https:://cccooooaaeeeeea// Utt.ttpss//eeeooooso.oooe..eeeee www.:eoo....oo............... server/apil content.
6aef 994 abbe---57aaabbe--bbbbllllaaaa---
Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization, 2017. URL 09280.
/abs//////// /175.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprins arXiv:2501. 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi I Liu, Misha Khalman, Felipe Llinnrrss, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024.
Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. arXiv preprint arXiv:2412.0195/ 2024a.
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dy- lan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. arXiv preprint arXiv:2407. 2024b.
HuggingFace. Online dpo trainer, 2024. URL MRRLhttts///gggingface.com http://g.a..ccoooooo/tt////////
11. 4/ //eee//nnine__pp_rr____e spc_ee__p________e trainerr..
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 1rX1v:2412.16720, 2024.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Cassidy Laidlaw, Stuart Russell, and Anca Dragan. Bridging rl theory and practice with the effective horizon. Advances in Neural Information Processing Systems, 36:58953-59007, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.1 13777737..77..... 2024.
Lihong Li. Wei Chu, John Langford, and Robert Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pp. 661-69 661-670, 2010.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over- warrparameterizzed matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory, pp. 2-47, PMLR, 2018
Chin-Yer Lin. Rouge: A package for automatic evaluation of summarics. Text summarization branches out, pp. 74-81, 2004.
Yong Lin, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, and Tong Zhang. On the limited generalization capability of the implicit reward model induced by direct preference optimization. arXiv preprint arXiv:2409.036550 2024.
13

---

Published as a conference paper at ICLR 2026
Zhihan Liu, Miao Lu, Shenao Zhang. Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436, 2024.
Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:/711.05101 5:5, 2017.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward, 2024. URL https: /ttp/aaxvvvv.......... www.riwwworg/a/////////
14734.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of out-of-distribution and in-distribution generalization. In International conference on machine learning. pp. 7721-7735. PMLR, 2021.
Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment,
2021.
Andrew Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2, 2000.
Frank Nielsen. An elementary introduction to information geometry. Entropy, 22(10)11100 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Karl Popper. Conjectures and refutations: The growth of scientific knowledge. routledge, 2014.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to q: Your language model is secretly q- q-function. arXiv preprint arXiv:2404.1233 12355, 2024.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks, 2019. URL
https::/arxiv... http:///ss//....... 08734.
Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective. arXiv preprint arXiv:2503.15477. 2025.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings 2011.
14

---

Published as a conference paper at ICLR 2026
John Schulman. A compelling intuition is that deep learning does approximate solomonoff induction, December 2023.
URL httpp/////.ccoo/o/ooonauluuuull
7741178475946600
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal, 2025. URL
org abs 2202. 12118.
Shai Shalev-Shwarzz, Ohad Shamir, and Eran Tromer. Using more data to speed-up training time, In Artificial Intelligence and Statistics, pp. 1019-1027. PMLR, 2012.
Yuda Song, Gokul Swamy, Aarti Singh, Drew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade. Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvement capabilities of large language models. arXiv preprint vrxii:2412.02674 2024b.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data, 2024. URL 10345.
/1710.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
Hao Sun and Mihaela van der Schaar. Inverse-rlinmment: Inverse reinforcement learning from demonstrations for Ilm alignment. arXiv preprint urXiv:2405.15624, 2024.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pp. 10022-10032. PMLR, 2021.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaxi- malist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056 2024.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.1438
2024.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the perfor- mance gap between online and offline alignment algorithms. arXiv preprint 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11 1134)111111144 1142, 1984.
Vladimir N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. In Measures of complexity: festschrift for alexey chervonenkis, pp. 11-30. Springer, 2015.
Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02333 02390, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
15

---

Published as a conference paper at ICLR 2026
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objectivee reward modeling and mixture- mixture-of-experts expertss arXiv preprint arXiv:2406.12844 12845, 2024.
Markus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja, Jorg Bornschein, Sandy Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, et al. Imitating language via scalable inverse reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system reasoning in Ilms: Learning how to think with meta chain-of-though! arXiv preprint arXiv:2501.04668 2025.
Teng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen, and Vasant G Honavar. On connection between imitation learning and rlhf, 2025. URL https: / /tt///aaxic /ttp//awwwgg/aaaaaaaa///a abs///bs//// 2500. 05079.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? comprehensive study. arXiv preprint 1.Xiv:2404.107/99 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.1044 2023.
Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? study in length generalization, 2023. URL https: / //tp///aivvvv//////////../.awa org/ ababbb///aa 32b00 16028.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b 7b: Improving llm helpfulness harmlessness with rlaif, 2023.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Camegie Mellon University, 2010.
16

---

Published as a conference paper at ICLR 2026
A A QUINTET OF HYPOTHESES FOR THE ONLINE-OFFLINE GAP
CONTENTS
A.1
Hlg: Intrinsic Value of Online Samples.
17
A.2
Hlg: Failure of Offline PFT Regularization to rrl
17
A.3
Hiy: Relative Ease of Online PFT Optimization.
17
A.4
HIg: Global RMs Can Be Trained on More Data.
18
A.5
Hly: Global RMs Generalize Better OOD,
19
We now consider (and in some cases, falsify) a series of hypotheses, both from prior work and of our own that attempt to explain the discrepancy between $2's theory and the reality of PFT in $3.
A.1 Hy: INTRINSIC VALUE OF ONLINE SAMPLES.
Intuitively, it may seem as though getting feedback on samples from the policy is providing qualita- tively new information from the fixed offline dataset. Tang et al. (2024) come to this conclusion.
However, it is not clear as to what, precisely, is the mechanism by which this on on-policy data actually helps with policy optimization when we account for the fact that the labels for this data are merely imputed by a RM trained on the same off-policy dataset, rather than truly new information from some ground-truth reward function. Put differently, there no information the RM that wasn't either in preference dataset or the (shared) base model, both of which the offline methods have access to.
Recall that from an information-theerrticc perspective, we know on-policy data is redundant via the data-processing inequality (MacKay, 2003) we cannot create any new information (i.e., bona fide human preferences) to learn from via sampling from the policy. auto-regressive generation also doesn't 't get new information from a real environment. Furthermore, given the RM, one could in theory run a (soft) value iteration procedure to compute the optimal policy without ever sampling on-policy, meaning that on-policy samples are technically not necessary solve the (soft) RI. problem.
A.2
Hy: FAILURE OF OFFLINE PFT REGULARIZATION TO TREF
Via an elegant argument, Song et al. (2024a) prove that offline PFT algorithms like DPO (Rafailov al., 2023) and IPO (Azar et al., 2023) require global (i.e., stronger) preference dataset coverage conditions than online PFT methods, as they are unable to effectively regularize to the reference policy otherwise. Intuitively, this is because reverse KL regularization involves an expectation over samples drawn from the learner's policy. Thus, all generable trajectories need to be within the support of the preference dataset for a purely offline algorithm to effectively control the reverse KL. A variety of approaches, from sampling on-policy to directly calculate the reverse KL and adding it in as an auxiliary loss term (Song et al., 2024a), to several forms of offline pessimism (Huang et al., 2024b; Cen al., 2024; Fisch al., 2024; Liu et al., 2024) have been proposed to mitigate this issue.
While certainly a contributing factor, there are several pieces of evidence that imply this distinction does not explain all of the gap in performance between online and offline approaches to PFT. First, adding in reverse KL penalty to DPO doesn't 't completely close the gap to bona fide online PFT methods (Song al., 2024a; Gao al., 2024a). Second, PFT methods that don't 't explicitly regularize to the prior like SimPO (Meng et al., 2024) have shown strong performance on multiple benchmarks. Third, there are fine-tuning problems where staying close to the reference policy not particularly important for performance, and we still see online methods out- nut-perform offline methods (Choudhury & Sodhi, 2024). Lastly, all the experiments in Figure 3 use the same regularizer for both online and offline algorithms, and we still observe gap in performance. Thus, H2 fails to predict these results.
A.3 Ha: RELATIVE EASE OF ONLINE PFT OPTIMIZATION.
One might ask if offline PFT is somehow faced with a harder optimization problem than online PFT because the former is forced to escape extra local minima, argued by Xu et al. (2024).
17

---

Published as a conference paper at ICLR 2026
However, remarked above, because the same loss function (DPO's) for both offline and online PFT, the only difference between the procedures we' re comparing is the data we re passing in. It is unclear how the same number of online samples could make it easier to optimize the same loss function - Xu et al. (2024) instead compare offline DPO to online PPO (Schulman et al., 2017).
pythia prompt aug.
823-46 Winrate
56.1
56.6
60.8
61.2
Figure 8: Augmenting the set of prompts we use for online DPO with those from the SFT dataset (= 3x much data) barely improves winrate. This is contrary to what a refinement of IHg predicts. Results across 3 training seeds.
DPD
Da. DPO
0a. DPD
Dn. 2.88 (aug.)
1.48
1.48
2.88
(aug.)
A refinement of the above hypothesis is grounded in the phenomenon of cimpuuaatioal-statiiitii gaps: for certain problems, additional samples, even if cnpommttion-thhortticeeee the amount of computation required to find solution (Daniely et al., 2013; Bandeira et al., 201 8). For example, filling in more of a Sudoku puzzle makes it easier to solve, even if we've already observed the minimum set of numbers to uniquely specify the solution. One might therefore view the (redundant) on- policy samples as providing these extra "constraints" on the policy search space.
redundant, reduce
To attempt to falsify this refined hypothesis, we perform prompt augmentation to increase the size of the preference dataset we use for training the online DPO policy. Specifically, we generate on-policy and compute RM labels on prompts from the SFT dataset, nearly tripling the training set size. Intuitively, the refined hypothesis were true, we' expect that we would see an increase in policy performance with these "redundant" samples. However, in Figure we instead observe almost no improvement in downstream winrate, which the refined version of H3 fails to predict.
A.4
Hy: GLOBAL RMS CAN BE TRAINED ON MORE DATA.
When look at the most performant global reward models available (Zhu et al., 2023; Wang et al., 2024; Lambert et al., 2024), we often observe that they are trained on relatively wide distribution of data compared the preference datasets used for offline PFT. whether there is something intrinsic about global RMs that make them more amenable to training on a wider distribution of data than local RMs / policies.' In fact, the human preference data we use to train our global RMs and policies is generated by diverse assortment of comparisons from wide variety of policies (Stiennon et al., 2020), rather than just samples from our particular base policy.
therefore natural question as to
To attempt to falsify this hypothesis, generate more concentrated dataset by using samples only from the SFT policy, with winners picked by got-4o query. As we are reducing the diversity the preference dataset, H, predicts that the gap between online and offline PFT would shrink.
pythia-1.48 gpt-labels
Ept-40 Minrate(
65.2
Figure 9: Performing an iteration of online DPO (third bar) on top of offline DPO (first bar) significantly improves winrate, even when all samples in D are generated by the SFT policy and labeled by gpt-4o, a narrower preference dataset. This is contrary to what H4 predicts. All results reported across seeds.
58.8
DPO
0n. DPD
DPO
(SPT)
(DPO)
In Figure 9, we see online DPO (third bar) on top of offline DPO (first bar) still significantly improving performance, even though we train all models with a relatively narrow, on-policy dataset, which H, fails to predict. We note that it is unsurprising that online DPO on top of the SFT policy (second bar) roughly matches the performance of offline DPO (first bar) - the former procedure amounts to merely relabeling the same SFT policy samples with the RM instead of the ground-truth gpt-44 4o labels.
'There is a O(H) computational speedup provided by not needing to do per-token forward backward pass for global RMs as in local RM training, which could allow more data to be trained on with the same computation budget. Howevever, we the same amount of data for local/glohal RM training all experiments.
18

---

Published as a conference paper at ICLR 2026
Thus, while it is hard to argue that more (and more diverse) data wouldn't lead to a better RM, don't see evidence that policies and RMs would have different levels of efficacy taking advantage of this wider distribution of data, at least for the sorts of problems consider in work.
A.5 Hg: GLOBAL RMS GENERALIZE BETTER OOD.
In their comprehensive empirical study, Tajwar et al. (2024) argue that when the peak of the learned reward model falls outside of the support of the preference data, online PFT techniques are better able to maximize this reward than offline approaches. Implicitly, such explanations are assuming that reward models generalize better OOD than policies. Lin et al. (2024) observe that, empirically, the reward model implied by DPO generalizes less well OOD than a standard, global reward model, a perspective echoed by the top of the RewardBench leaderboard (Lambert al., 2024). The work of Chu et al. (2025) finds evidence of this phenomenon in vision-based reasoning tasks.
However, there are several open questions in the above story. First, given persistent concerns about reward model over-optimization (Gao et al., 2023; Eisenstein et al., 2023), it seems that neither policies nor RMs generalize all that well OOD in general. Second, it is not immediately clear why policies and reward models that are trained from the same initial checkpoint on the same dataset (as the experiments Fig. 3) should generalize differently OOD. Third, the comparison between DPO RMs and standard global RMs changes two factors at once: DPO regularizes to the reference policy (unlike unregularized global reward model training) and optimizes a local (rather than a global) RM.
We conduct several experiments to provide answers the preceding questions. To remove the confounder of the reference, we also train local RMs without regularization (i.e.. reference-less DPO), which we refer to as Local. We note that this is precisely the mile (Eq. 9) analyzed above.
RM Validation Likelihood (f) on tl:dr
0.526
0.567
0.593
0.545
0.6e9
0.610
Figure 10: Going from global RM local RM or from a local RM to a DPO RM hampers (in-distribution) validation accuracy. We report results for all RMs across seeds.
DPC
Local 1.48
Global 1.43
DPO 2.88
Local 2.88
Global 2.60
1.48
Storstel
Pythier:.44 42 Boll: BFT
pythie-1. Ball: DP3
pythia-2 88 Bal: SET
Rell
k0
Stoket
Wit-0 Vicca
8
We compare global, local, and DPO reward models in terms of their BoN winrate as measure of OOD generalization. We use the same size for the policy and the reward model and generate samples either from a base SFT or offline DPO policy. As we increase N, we see a perfect correlation emerge between the in-distribution validation likelihood reported in Fig. 10 and BoN performance. All standard errors are reported across three training seeds.
Figure 11:
We first compare DPO, local, and global RMs of the same backbone in terms of their validation likelihood (i.e.. in terms of their in-distribution generalization when evaluated as classifiers). In Figure 10, we consistently see that enforcing a token-wise dccomposition leads to worse performance in distribution. Furthermore, adding in regularization during training further hampers the hold-out validation likelihood of the local RM. These results persist across multiple model sizes.
We next evaluate the OOD generalization (as in, not samples from the same distribution the human preference dataset) of each of these families of models by evaluating their Best-Of-N (BoN) performance samples from both the SFT and offline DPO policies. Higher values of N correspond
19

---

Published as a conference paper at ICLR 2026
to testing reward model wider swathe of generations. In Figure 11, see higher in-distribution validation likelihood perfectly correlate with BoN performance for high enough N.. 5
Better in in-distribution margins are known to correlate with some kinds of OOD generalization (Miller et al., 2021). Intuitively, by having a larger margin in-distribution, one hopes a classifier will be able to better separate OOD samples, similar to classical arguments for SVMs. While this observation provides a potential explanation for the better OOD performance reported in prior work, it merely kicks the can down the road: we still haven' I explained why there should be difference in-distribution in the first place. Thus, while we don't falsify H5, the root cause of the observed phenomena of global reward models generalizing better than local reward models is still unclear, leaving conceptual gap.
'The results where we sample from DPO policy and further filter the samples with the internal DPO RM and improve winrate are separately interesting as they provide evidence of iefroveepeeettt echoing the findings of Song et al. (2024h); Huang et al. (2024a). However, note that an extemal global RM is still significantly hetter at accurately ranking learner samples across the board.
20

---

Published as a conference paper at ICLR 2026
B PROOFS.
CONTENTS
B.1 Proof of Lemma 2.1
21
B.1.1 Proof of Lemma 2.1 with Reference Policy
21
B.2 Proof of Theorem 2.2
22
B.3 Proof of Theorem 2.3
22
B.4 Proof of Theorem 4.1
23
B.1 PROOF LEMMA 2.1
Proof.
arg min (P,| arg min Eooopa
P=(((ss)((og((<(((())) - log(PP
#E11
sell
CE
arg min Rooooo
P=(((ss)((og((<(((1)))) - r(E) log (rr (())))))
Tell
EE
arg min Esoepo xell
(=(E))))(log(P+((E) - r(E))
EE
arg max ERgrpo
Σ (o(()s))) +({))
Tell
SEE
arg max Be (r(E)) TEll
(E)]+++ ((T))
B.1.1 PROOF OF LEMMA 2.1 WITH REFERENCE POLICY
Given two distributions P,Q E A()), define their trajectory-level mixture P. Q as
P(E)((E) (P. QE()) Eess P(E))(()) E
(13)
Lemma B.1 (MinRelEnt RL is an RKL Projection). Let r E R be a global reward model. Define n² arg mingen (Fr(I=I I a as the trajectory-level reverse KL projection of the trajectory- level mixture of PP and P r onto II. Nexi, define nē as the soft-optimal policy computed by solving
arg max Be- [r(E) DRI. (P, | Pra))
(14)
fEIl
Then, n. = x²
21

---

Published as a conference paper at ICLR 2026
Proof.
x²= arg min DKL (P> m sell
arg min E Barupo well
Σ (=(((1so)(og((((((ss)))) (E)( log m. (5 sa))
SEE|50
arg min E Paprpo TEN
Σ (=(((so)((og((((((1ss))) r(E) + log ((r, so) log( Andr (E]so)))
arg min E Bppopo mEll
(L()) r(5) - log(( am (E (E|s))))
arg max Eao-Pa
Σ P=(E)so)(-log((P((HH)) + +(() + log(P and (Elso)))
TEll
EE
arg max Re [r(()] - xI. >
|
WEIl
B.2 PROOF OF THEOREM 2.2
Proof. Below, we use mmle E mil and imll E mle to refer to specific minima of Eqs. 9 and 6.
First, we observe that because R R(II) and the fact that minimizing the same functional over the same function class must produce the same set of minima, émle R (fmee). This further implies that for each mll mle.
mill mle such that Ag E E, Fire (E) = mie (g) and vice-versa.
From Lemma 1, we know that each Ayn E nntn can also be written as the minimizer of reverse KL projection:
Triht = arg min Dzr. (P=I||?;m))) Pime) rell
arg min Dxi. (IP (PPI||P Fa
*Ell
arg min Dx (I ((P>...)) xEll
Tmle
where the last step uses the fact that 7mle E II. We can repeat the above steps for each Tmil E A'le to prove that Rrin Amle.
B.3 PROOF OF THEOREM 2.3
Proof. Below,
mile mil and i'mle m refer specific minima of Eqs. 9 and 6.
First, we observe that because R = R(TI) and the fact that minimizing the same functional over the same function class must produce the same set of minima, fmll R(mmle). This further implies that for each mle E mll.. 35 fmle such that Vε E E, T=rm (E) = Fml (E) and vice-versa.
Next, observe that for all datasets and V Aapo Topo Fi i such that VG E.
H
H Σ
Σ
log #ipo (an(sh) (ss)== =
log / (a (s))))) + log (rer(aalss))
(15)
h
22

---

Published as a conference paper at ICLR 2026
Next, because of our realizability assumptions (i.e. Tile dopo II), also know that Amle mil and fippo
dpo Put together, this tell that y Topo Edpo- mle mlle such that A4 E.
H
H
Σ
log Tpp (an (s)))
Σ
log Tile (an + log Trat (a))))
(16)
Then, substituting Mie for the equivalent T'Ema
arrive at
Σ log (an Fimt (5) +
Σ log Trat (an)())))))
(17)
This tells us that ra = Til + Tam and therefore 1 rape
P
Fine P Next, from Lemma B.I, we
know that each Whht Wht can also be written as the minimizer of reverse KL projection:
Trht =
arg min xx [P=I|[F
(18)
well
arg min LLL (1xx ||PT 7
(19)
well
arg min 1 (TP ||[ Rapo) xell
(20)
aapo:
(21)
where the last step uses the fact that Adpo prove that firnt Adpo-
II. We can repeat the above steps for each Í'le E fmle to
B.4 PROOF THEOREM 4.1
Proof. Below, we use 7sim E ssim and Trm E Ritht to refer to specific minima.
First, we observe that because of Lemma 2.1, we can write that
Ant =
arg min DxI. (IP (PP Fam Fsim Tell
E isim
(22)
We can then combine the above with the fact that Asim C Rgim to conclude that Tith C (I
Now, we assume for the sake of contradiction that Figtt Rhnn such that Waht & 7ssm (i.e., Rhht is not a BT likelihood maximizer over (T()s)im)). Then, by the fact that Ryth E Rehht C (T(()si)) and the definition of each ăsim fismm BT likelihood maximizer over 11(Reim), it must be true that
E
(PD(E > &a [a] Team (EI > E2(
< ((G DKD(P > &n ( re (E1 > Eglso))
Then, because assumed that Visim fsim Rgim Than 11 (i.e., the secondary RKL projection is exact), it must then be true that Fisim E fsim S.t. Triht = T which further implies that
EE (D) (PD(Ei > & {2|s0)[[TE)) |PBT (51 > E2|s0))
DKL > & [ 7 (& + E21s0))
BEELIED (T' > falso)|| (E1 & {2)))))]
Together, these imply that:
D (PD(E1 [2 Fram (&r E2|so))
E(G (P > {2 Fam (f1 > E2(s)))
However, by realizability assumption, also know that Vitsm Asim C II(Rsmm). Fmgm E Rsim- Thus, this expression contradicts the definition of islm as BT likelihood maximizer over Rsim- [
In words, the realizability assumption of the above theorem is that the policy class II includes all policies that are soft-optimal for the relatively simple verifiers R but not necessarily for all verifiers in R. Thus, fully isomorphic classes implies but is not implied by this (weaker) assumption.
23

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS.
CONTENTS
C.1 Dataset Details
24
C.1.1 Online DPO Dataset Details
25
C.2 Model Details
25
C,3 Training Details
25
C.4 Evaluation Details
26
Throughout all of our experiments, we use the tl; dr data from https: htt//ggtthbbccoo com/ openai/ apenai/summaree www.maiiaeeeeaaeeeeeeeeeeeee fref-oo-- coam-feeddi feedback, optimize models from the pythia family (Bider- man et al., 2023), and build upon the codebase for REBEL (Gao et al., 2024a), available at https: // //ggttttuubh. con/ ZhaolinGe caaolinGao//a inGao/REBEL. We use VLLM for fast inference (Kwon et al., 2023). We will open-source the code, models, and data for this project upon paper acceptance.
C.I DATASET DETAILS
For the standard summarization tasks (Figs. 8, 10, 11, 4), training data has the following characteristics:
Table 1: Dataset split, prompts, and maximum generation length for TL,DR summarization
DATASET
TRAIN/VAL/TEST
PROMPT
GENERATION LENGTH
SFT
117K/6.45K/6.55
"TL;DR:" "TL;DR:"
53
PFT
92.9K/83.8K/-
53
As is standard practice, we first fine-tune the base pythia models on the SFT training set before performing some variant of DPO on the PFT training set.
For the prompt augmentation results in Figure 8, also the prompts from the SFT training set for on-policy generation.
For the "GPT-label" results in Figure 9, sample twice from the SFT policy with temperature 0.1 and use the same prompting strategy we use for evaluation to pick a winner with gpt-4o. We perform this procedure all PFT training prompts generate new training set.
For the twodword summarization results in Figure 5, we reduce the maximum generation length to tokens well as replacing "TL:DR:" with "Two-Word TL;DR:". We use the following prompt generate a pair two-word summaries via gpt-4o query:
Given the fol following forum post, please write a summary of the post of length at most two words. The summaries should not include any unimportant or irrelevant details.
### Post: (post))
### Instructions:
Your response should use the format:
Summary: <two-word summary>
Do not include any other text or explanations in your response.
We use the same prompting strategy as for evaluation to pick a winner. We do this for all prompts in the PFT training set.
For the ROUGE-L results in Figure 6, we rank 25 prompt completions sampled from the SFT policy with temperature 0.1 each of the PFT training prompts via the ROUGE-L F1 score against the
24

---

Published as a conference paper at ICLR 2026
preferred prompt completion in the training set. We the highest and lowest scoring generations from the SFT policy as the preferred dis- tis-preferred completions.
C.I.1 ONLINE DPO DATASET DETAILS
The above discussion is on how we get the preference dataset we use for training reward models and offline DPO policies. For online DPO, we sample from either the SFT or Offline DPO policy 25 times with temperature 0.1, rank the samples according to some kind of reward model, and use the top and bottom of the list the preferred and dis- dis-preferred completions the prompt for new training set. Also, unlike some implementations of online DPO (e.g. those in Gao et al. (2024a)), we do not continuously sample from the policy and instead sample once from the initial policy across prompts in "batched" fashion reduce computational expense. This further aligns our online and offline PFT implementations.
C.2 MODEL DETAILS
Across all experiments in this paper, we use the pythia series of mod- els. In particular, ants available at https:/ /tops://bet httg:///hggiggacce..ccoooo cectaaaceccccceeoooo sythia-scaliin swwte-64ffsdaaaeeeeeeeeee atf.baffaec22eebbb3bll2d.2
use the 1.4B, 2.8B, and 6.9B parameter de-duped vari-
coo/oollecions/eeeeeeeee as our base models. We remove the final softmax and add in a single linear layer to transform a policy model into global reward model. We train all policies and reward models ourselves and do not use LoRA anywhere. We train SFT models via epoch over the SFT dataset. We the further fine-tune these models via one epoch of training on the PFT data create local and global reward models. We also only perform one epoch of training for DPO on the PFT data, except for the results marked as DPO (2x) in Figs. 3 and 6. We use a mixture of A100s, A800s, and H100s across experiments. No training run took more than 8 hours.
C.3 TRAINING DETAILS
We only three training algorithms in this paper: logistic regression for global reward model training. SFT and online/offline DPO. We use consistent hyperparameters for both of these procedures across all experiments in this paper. We use AdamW (Loshchilov al., 2017) for all optimization in this paper.
PARAMETER
VALUE
BATCH SIZE
64
LEARNING RATE
3E-6
E
lE-S
SCHEDULE
COSINE DECAY
Table 2: Hyperparameters for global RM training.
PARAMETER
VALUE
BATCH SIZE
128
LEARNING RATE
3E-7
1E-5
/ SCHEDULE
0.05
LINEAR DECAY
Table 3: Hyperparameters for DPO training. We found relatively low learning rate lead to significantly better performance for DPO across the board. For offline DPO, we set Tref nstt. For online DPO, we set Fret to whatever generated the training data (i.e. either Tiht or d depending on the experiment) and scale logits by the inverse of the sampling temperature 0.I = 10 before taking the softmax, as is standard practice.
1
25

---

Published as a conference paper at ICLR 2026
PARAMETER VALUE
53
Table Hyperparameters for local reward model training that differ from those for DPO. There is no ref for local RMs.
C.4 EVALUATION DETAILS
We sample with temperature 0.01 for all winrate ROUGGE- policy evaluations. We use temperature 0.1 for all BoN results.
For all winrate computations, we use gpt-4o. For standard summarization, we use the following prompt:
Which of the following summaries does a better job of summarizing the most important points in the given forum post, without including unimportant or irrelevant details? Judge based on accuracy, coverage, and coherence.
### Post: ([post))
### Summary A: [[sumaryya]]
4## Sunmary
[[summaryb]]
## Instrontios::
FIRST provide one-sentence comparison the two summaries, explaining which
you prefer and why. SECOND, on a new line, state only "A" or "B" to indicate your choice. Your response should the format: Comparison: <one-sentence comparison and explanation> Preferred: <"A" "B">
We use 600 randomly sampled prompts from the SFT test set for evaluation.
For two-word summarization (i.e. Fig 5) we use the following prompt:
Which of the following two-word summaries does a better job of summarizing the most important points in the given forum post, without including unimportant or irrelevant details? Judge based accuracy, coverage, and coherence.
### Post: ([post)}
### Summary A: [[summarya]]
### Summary B: ([summaryb)]
### Instructions:
FIRST provide one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, on a new line, state only "A" or "B" to indicate your ce.ice. Your response should the format:
26

---

Published as a conference paper at ICLR 2026
Comparison: <one-sentence comparison and explanation> Preferred:
<"A" or "B">
We use 6000 randomly sampled prompts from the SFT test set for evaluation.
For ROUGE-L evaluations (i.e. Fig. 6), we use the FI variant of the metric against preferred completions from the PFT validation set.
For likelihood evaluations, we use the appropriate validation loss for the model class. For global reward models, this is just logistic loss. For DPO reward models, this includes the scaling factor B and the reference policy probabilities. For local reward models, this includes just the scaling factor B.
27


Reviews:
[{"review_id": "fB5hSUv8Mh", "reviewer": "Reviewer_Caqs", "summary": "This paper aims to provide an intuitive explanation for the empirically observed performance gap between two-stage (e.g., RLHF, online DPO) and direct (e.g., DPO, offline DPO) preference fine-tuning methods. Through a combination of theoretical analysis and empirical studies, the authors evaluate multiple hypotheses concerning the value of reinforcement learning in two-stage approaches. Among the tested hypotheses, only the generation–verification gap (it is easier to learn a simple reward model from preference data and optimize policies for simple verifiers) remained supported by the evidence. The paper concludes that two-stage methods effectively operate over a reduced policy space, requiring less data than offline fine-tuning.", "questions": "Clarification question: \n\nThe experiments supporting the generation-verification gap hypothesis show:\n- cases where verification is easier than generation, and online methods outperform offline ones;\n- cases where complex reward functions diminish the benefit of online techniques.\n\nIn LLM preference fine-tuning, even when reward and policy classes have comparable complexity, do we still observe a gap favoring online methods? If that is the case, is the generation-verification gap alone sufficient to explain this phenomenon, or am I missing something?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written, and the results are rigorous.\n\nIt makes both theoretical and empirical contributions to understanding the origins of the performance gap between online and offline preference fine-tuning.\n- shows that when policy and reward classes match, online and offline methods share the same set of optima.\n- falsifies several existing hypotheses about the benefits of RL in preference fine-tuning and introduces the generation-verification gap as a plausible alternative explanation.\n\nProvides useful practical insights:\n- for problems where verification is simpler than generation, two-stage methods are preferable.\n- for tasks requiring long-horizon reasoning or complex planning, the gap between online and offline approaches is likely to widen.\n\n(Note: I have not carefully checked the additional hypotheses in the appendix.)", "weaknesses": "Potential related work: Nika et al. (2024) conducted a comparative theoretical analysis of RLHF and DPO, highlighting the generation-verification gap as a key factor explaining when RLHF statistically outperforms DPO. They show that when the reward class is of lower complexity than the policy class, RLHF tends to perform better, consistent with this paper's findings.\n\nNika et al., 2024. Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences. ICML, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IlGO5sm6A6", "reviewer": "Reviewer_Ypdj", "summary": "This paper tackles the question of why do complex, two-stage online methods (like RLHF) empirically outperform simpler, direct offline methods (like DPO), even when they both optimize the same likelihood-based objective?\n\nThe paper's first key finding is theoretical: when the policy and reward model function classes are isomorphic (i.e., $\\mathcal{R} = \\mathcal{R}(\\Pi)$), the optimal solutions for both online and offline methods are identical (Theorems 2.2 & 2.3). This theoretical equivalence contradicts robust empirical findings.\n\nThe paper then systematically conducts controlled experiments to rule out several common hypotheses and propose a more suitable hypothesis: the generation-verification gap.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The generation-verification gap is a nice conceptual contribution. It reframes the online vs. offline debate to a root cause a \"statistical efficiency\" problem.\n\n\nThe paper is a model of good scientific reasoning. It cleanly formalises the theory-practice gap (Theorems 2.2/2.3) and then treats various explanations (e.g., optimization, regularization, OOD) as falsifiable hypotheses.\n\nThe paper's best evidence comes from its two \"gap-closing\" experiments. Predicting that the online PFT advantage would disappear on a bandit-like task (Fig 5) and a ROUGE-L task (Fig 6) is not obvious. \n\nThe authors online DPO setup -- where an RM is used to re-label on-policy data, which is then fed into the same DPO loss is a clever way to isolate the core variable. It successfully controls for confounders like the optimization algorithm (PPO vs. DPO loss), ensuring the comparison is truly about the two-stage process versus the one-stage process.", "weaknesses": "1) The \"Simplicity\" of Verification is a Black Box: The entire argument of H-6 hinges on the assertion that a verifier is simpler than a generator. This central concept of simplicity is never formally defined.\n\n2) The paper frames H-6 as the sole surviving explanation. This seems unlikely to be true. It's more plausible that other factors are also true and compound the effect. For example, for H-5  (OOD Generalization), Fig 9, Fig 10 show that global RMs do generalize better (both in-distribution and OOD) than local RMs. The paper claims H-6 causes this. But it's equally plausible that better OOD generalization is a separate benefit of the global RM architecture, which then adds to the statistical efficiency benefit of H-6.\n\n3) How does the paper conceptually differ from [1] which seem to propose a similar argument?\n\n4) The initial theoretical question is built on the isomorphism between policies and reward functions. But in practice offline DPO uses a local RM (architecturally tied to the policy, $r_{\\pi} = \\sum \\log \\pi$) and online RLHF uses a global RM (using the final hidden state of the full sequence). Does this create a confounder?\n\n\n[1] Self-Improvement in Language Models: The Sharpening Mechanism", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hw73MhATTl", "reviewer": "Reviewer_sSEo", "summary": "This paper investigates why two-stage fine-tuning (FT) procedures—training a reward model (RM) followed by reinforcement learning (RL)—often outperform direct offline optimization, despite appearing information-theoretically inefficient.  They propose the hypothesis that such a phenomenon is due to the generation-verification gap.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper appears to be the first to investigate the underlying reason why the two-stage fine-tuning procedure outperforms purely offline approaches. The proposed hypothesis is well-motivated, and the authors provide a theoretical analysis to support it. Furthermore, they conduct extensive numerical experiments that not only reinforce their hypothesis but also help rule out alternative explanations.", "weaknesses": "Overall, I do not have major concerns about this article. The core idea is clearly articulated, and the presentation is well-structured and easy to follow. My only reservation is whether the contribution is substantial enough for publication at ICLR. Although the paper spans nine pages, a significant portion is devoted to setup and related work, with the main theoretical contribution centered on Theorem 3.1.  I would suggest that some proof sketches and key arguments be included in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "530cIcfz7i", "reviewer": "Reviewer_J9QF", "summary": "This paper discusses several hypotheses for the benefit of online fine-turning in preference-based RL, focussing on large language models.   Starting from a unified objectives for online and offline fine-tuning, it shows that both online and offline fine-tuning can be seen as maximizing data likelihood, although for the online setting the learned policies is within a restricted policy class induced by learned reward models. Then, this paper raises a new assumption for the benefit of online fine-tuning: simpler reward models induce smaller policy classes, which turns out to have better performance. Finally, the paper offers evidence for the benefit of using simpler reward models.", "questions": "1.  beside the summary task being considered, is there any other task that also has generation-verification gap? \n\n2. Is the assumption H6 valid for tasks where the generation content is longer than the prompt?\n\n3. Why do you use BoN in Fig.4?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the presentation of this paper is easy to follow.\n\n2. The discussion through the lens of maximum likelihood estimation is suitable for the LLM setting and offers new insight for understanding the benefit of online fine-tuning.\n\n3. In particular, the assumption of the connection between simper reward models and better performance is interesting and worth investigation. For example, verifiable rewards used for training reasoning models are also simple ones.", "weaknesses": "1. The presentation of H6, which is the core assumption made in this paper, is hard to understand in the first glimpse. I would say that $\\Pi(R_\\text{sim})\\subset \\Pi$ is a general statement, since $R_\\text{sim}$ is unlikely to cover **all possible reward functions**. A better way for stating the benefit of using small reward models would strengthen this paper. Alternatively, you can define $R$ to be all the reward function consisting with the preferences, so that $\\Pi$ will be all the policies that can generate the preferences.\n\n2. The empirical justification is very limited. Since the theoretical justifications in this paper are straight-forward and rely on idea conditions, it is better to include more empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)