Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
Authors: Bo Wang
Qinyuan Cheng
Runyu Peng
Rong Bao
Peiji Li
Qipeng Guo
Linyang Li
Zhiyuan Zeng
Yunhua Zhou
Xipeng Qiu
Venue: NeurIPS 2025 poster ()
Keywords: Natural Language Processing
Large Language Model
Reinforcement Learning
Markov Decision Processes
Post-training
Primary area: deep_learning

Abstract: Post-training processes are essential phases in grounding pre-trained language models to real-world tasks, with learning from demonstrations or preference signals playing a crucial role in this adaptation. We present a unified theoretical framework bridging Supervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training. Through rigorous mathematical derivation, we demonstrate that both SFT and preference learning methods like Direct Preference Optimization (DPO) operate within the same optimal policy-reward subspace, with SFT representing a special case of implicit reward learning. Our analysis reveals a critical limitation in conventional SFT: the KL divergence term in distribution matching becomes constant with respect to the policy during optimization, failing to constrain model updates. To address this, we propose a simple yet effective learning rate reduction approach that yields significant performance improvements (up to \textbf{25\%} relative gain and \textbf{6\%} absolute win rate increase in instruction following tasks. Additionally, we derive alternative SFT objectives from various f-divergence functions that preserve the KL term during optimization, further enhancing post-DPO model performance. Finally, we extend the theoretical relationship between LLM logits and Q-functions from preference learning to the SFT context, providing mathematical derivations and experimental validation.

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
Bo Wang
Qinyuan Chengl* Runyu Peng' Rong Bao'
Peiji Li'
Qipeng Guo Linyang Li Zhiyuan Zeng' Yunhua Zhou²
Xipeng Qiu
School of Computer Science, Fudan University 2 Shanghai Artificial Intelligence Laboratory (bwang22, chenggy21, rypeng22, rbao22) ffffaaan edu.cn (linyangli19, xpqiu)&fudan edu. cn (guoqipeng, zhouyunhua) orr............................. org. cn
Abstract
Post-training processes essential phases in grounding pre-trained language mod- els to real-world tasks, with learning from demonstrations or preference signals playing a crucial role in this adaptation. We present a unitied theoretical framework bridging Supervised Fine-Tuning (SFT) and preference learning in Large Lan- guage Model (LLM) post-training. Through rigorous mathematical derivation, we demonstrate that both SFT and preference learning methods like Direct Preference Optimization (DPO) operate within the same optimal policy-reward subspace, with SFT representing a special case of implicit reward learning. Our analysis reveals critical limitation in conventional SFT: the KI. divergence term distribution matching becomes constant with respect to the policy during optimization, failing to constrain model updates. To address this, we propose a simple yet effective learning rate reduction approach that yields significant performance improvements (up to 25% relative gain and 6% absolute win rate increase in instruction following tasks. Additionally, we derive alternative SFT objectives from various f-divergence functions that preserve the KL term during optimization, further enhancing post- DPO model performance. Finally, extend the theoretical relationship between LLM logits and O-functions from preference learning to the SET context, providing mathematical derivations and experimental validation.
Introduction
Post-training represents critical phase in grounding Large Language Models (LLMs) in real-world applications. After accumulating general prior knowledge from numerous pre-training corpora, post- training aims to leverage the potential of LLMs for different needs, such as following natural language instructions [1. 2, 3, 4, 5]. Two principal methodological approaches dominate the post-training landscape. The first approach learns from expert demonstrations [6, 7], commonly known as imitation learning, which in the context of L.LMs is typically referred to as Supervised Vine-Tuning (SFT). The second approach focuses on learning from environmental signals, primarily through Reinforcement Learning methods [8, 9, 10].
Within the post-training landscape, preference signals have emerged as particularly valuable forms of feedback, attracting substantial research attention[1 1, 1]. Preference learning typically follows a two-stage process (hereafter referred to as sequential training): an initial stage of SFT followed by preference optimization methods like Direct Preference Optimization (DPO) [12]. However, the relationship between these critical stages remains predominantly understood through empirical
"Equal Contribution.
t Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

SFT: Moving Along the Mean Gradient Optimal Policy Subspace
DPO: Moving Along Chaset--Reeccted Direction Optimal al Policy Subspace
Space
Optira Palicy Fath
c. Durmer Optima Polty
Policy Aftr 3FC
Optiai Pannss for Each Reward Funct on
Chosen Sample
bohe The Data Points Along Cptimal Pat
31 Excer ey (5-1 lo 1o00
c Rojecied Samale
5F1 Startng Pain: dos Optima Pathi
1: SFT Search Dorection dling Optimall Path
DPD Biect (Persiiie Chosen Sejecred
Averspe Dimecion from Dara fowiss Expert
Figure 1: Schematic diagram: SFT and DPO optimizing implicit rewards in the optimal policy subspace. The z-dimension refers to the space of all possible policy models. The y-dimension represents the space of all possible reward functions. Given specific reward function, each possible policy will have its own expected return, forming a curve in the policy-return plane. The optimal policies corresponding different reward functions constitute subspace. Left: The SFT process searches within this subspace and moves along the average direction indicated by the demonstrations. Right: Likewise, DPO operates within this same subspace, but navigates along the direction vector formed by the chosen and rejected examples.
observations rather than thcoretical foundations, and SFT is often treated merely as a preparatory warm-up step [13]. Despite the widespread adoption of this sequential paradigm, significant gap persists in theoretical perspectives regarding how these two approaches fundamentally relate to one another. While previous research [14] has extensively explored various aspects of LLM learning dynamics, the theoretical connections between SFT objectives and preference learning frameworks have received insufficient attention, limiting understanding of their combined effectiveness in the post-training process.
To mitigate this gap, we prove that implicit reward learning can be utilized unified view connecting SFT and preference-learning processes. Previous work [12] established that preference learning in the second stage can operate through implicit rewards. In our research, revisit the distribution matching objective and apply necessary adjustments for post-training. We provide a comprehensive mathematical proof following earlier works [15, 16]. Our proof demonstrates that the conventional SFT objective represents a special case of learning implicit rewards. Figure illustrates our theoretical conclusion. The optimal policy for each possible reward function forms policy-reward subspace, with both SET and DPO operating within this subspace.
This theoretical framework of implicit rewards yields several novel insights. By framing SFT as training target derived from distribution matching, we uncover a key insight: the crucial KI. term in the objective functions merely serves as zero-order component. Since this term remains constant with respect to π, it imposes no constraints on model updates following differentiation. We propose a simple yet efficient heuristic to mitigate this issue by reducing the learning rate. Furthermore, identify alternative training objectives by choosing different f-divergence derivative functions for distribution matching that preserve the KL term during optimization and show their effectiveness empirically. Finally, we demonstrate that LLM logits can function as a Q-function corresponding to implicit rewards during the SFT process. This extends the theoretical framework in [17], which primarily established this relationship the DPO setting, while our work reveals similar mathematical structures within the SFT process. Our empirical training results align strongly with these theoretical predictions in the instraction-follooinn tasks.
Our main contributions are follows:

---

1. We revisit the distribution matching objective and mathematically prove that SFT also learns an implicit reward function identical to that of DPO. This provides unified theoretical view that clarifies the relationship between SFT and preference learning.
2. Within this theoretical framework, we provide a simple yet effective approach by reducing the learning rate during the SFT phase mitigate the absence of the KL term. The KL. term typically ensures that the policy does not deviate excessively from the base model, promoting stable and efficient learning. This significantly improves results, with relative improvements up to 25% (absolute win rate increases of up to 6%, from 15.6% to 21.5%).
3. We also propose several alternative SFT objectives derived from other f-divergence functions for LLMs. We demonstrate further improvements in model performance after DPO training (hereafter referred to post-DPO), which yields up to 4% absolute win rate improvements.
4. We mathematically extend the relationship between LLM logits and Q-functions from the DPO context to the SFT process, supporting this extension with indirect experimental evidence. This formulation enables us to efficiently estimate state values under the model's s implicit reward and provides deeper insights into the role of SFT in the alignment process.
2 Related Work
2.1 Inverse Reinforcement Learning and f-Divergence
First formalized by [18], Inverse Reinforcement Learning assumes that the expert represents the optimal policy under certain reward function. It tries to recover this reward function and train policy model that maximizes This approach was significantly advanced by [19] with the principle of maximum entropy to ensure proper exploration. Some commonly used IRL methods include game-theoretic approaches, most notably GAIL [20]. It trains an adversarial manner using explicit reward recovery and learns policies from the recovered reward function. [21] mathematically provides a unified view using /-divergence. [22] also leverages different forms of f-divergence derived for imitation learning. It involves iterative optimization and primarily explores applications in classical RL scenarios.
Learning an explicit reward function often involves additional parameters and high-nariance adversar- ial optimization. [15] avoids explicit reward learning by learning a Q-function parameterized by both reward and policy. [16] followed this work and introduced non-adversarial imitation learning for large language models. [23] formulates imitation learning using reverse KL divergence with an entropy regularizer and avoids explicitly training a reward function, which helps reduce overfitting, [24] involves different formulations of f-divergence in solving the di-tiiiiibtiion (OOD) problem in the generation process and enables backtracking in sequence generation.
Instead of using the Bellman equation like [15, 16), we apply closed-form solutions and avoid inner-loop optimization. In comparison with [23], our work utilizes a more general f-divergence formulation [21] and does not introduce negative samples. Unlike [24], we primarily examine the relationship between SFT and DPO in the sequential training context of large language models. [25] also formulates Ke-regularized SFT as IRL. Our work goes further by using the implicit-reward view to connect it with DPO.
2.2 Implicit Reward in LLMs
The typical Reinforcement
Learning from Human Feedback (RLHF) methods generally involve substantial computational budgets (e.g., PPO [26] uses four models in the training process), and DPO [12] was proposed to reduce the computational overhead of the RL process. It proved that the maximization problem has closed-form solution, allowing to perform reward modeling directly
on the implicit reward. Many related works like IPO [27], KTT [28], SimPO [29], and R-DPÓ [30] have continued development along this path. [17] has also established the relationship between
f-functions and LLM-logits during the DPO process. [31] treats logits as Q-values and trains Q-head on robotic tasks. We demonstrate that analogous structures exist in the SFT process.
3

---

2.3 Post- Teat-Trainng Theory Analysis for LLMs
There are also several analyses examining relationships in post-training phases. [32] analyzes the isomorphic relationship between implicit rewards and reward models. It connects implicit rewards with the generation-verificatie
gap. [14] pays more attention to the learning dynamics during different training phases. In work, foeus more the relationship between training objectives during the post-training process and use implicit rewards as medium to understand them.
Unified View between SFT and DPO
Token-Level MDP in LLM In language models, the Markov Decision Process (MDP) applies to token-level decisions. The state S represents the context, actions A are possible next tokens, and the policy P(als) gives token probabilities. State transitions T(s'|s, a) are deterministic, with s' continuing until reaching a terminal state. Y is the discount factor. This framework formalizes how language models make sequential decisions during text generation.
(token appended context). The model receives reward r(s, a) for each choice,
Definition (Occupancy Measure [15]). The occupancy measure of policy x defined as:
p(s) = (1 - Σ P(ss = =)
k=0
where P(s; = | x)) denotes the probability of visiting state at time step under policy x.
Definition (State-Action Distribution [15]). The state-action distribution of a policy π is given by:
p(s, a) p(s)s)
where u(s,o) a) represents the stationary distribution over state-action pairs induced by π.
3.1 Distribution Matching in Post-training
A well-established training objective in imitation learning is distribution matching [19, 15]. It focuses on minimizing the f-divergence between expert's state-action distribution μæ and that of the policy model Mr,, while incorporating entropy regularization term to promote exploration. However, entropy not entirely suitable in the post-training scenario for LLMs. The use of proba- bilistic averaging over the whole vocabulary could potentially damage the natural language priors established during the pretraining of the base model. Therefore, we modify the regularization term from entropy the -eiibac (KL) divergence between the base model Tref and the policy model π.
min (y(Ma))HE)
+BDKL (II(TTTTTe)))
(I)
- -()((x) traditional setting
where (-1-))- denotes the f-divergence, and H(T)) is the entropy of the policy. , is the coefficient of the regularization term and often serves as a hyperparameter.
A similar object was also introduced in [32]. We approach this concept from a different theoretical perspective and provide additional clarification here:
1) From the Cross-Entrows Term Perspective: The KL divergence can be split into the entropy term and the cross-entropy term: H(T, Te)) H(TT). As the base model has converged during the pretraining phase and has been exposed to extensive natural language data, the cross-entropy term should not large. This implies that the policy model should not deviate from the domain of natural language when maximizing exploration.
2) From a KL Divergence Perspective: The base model obtained from the pretraining phase already possesses sufficient quality and contains additional knowledge. Therefore, when minimizing the divergence for distribution matching, we need to preserve the intrinsic properties of the base model.

---

3.2 Imitation Learning as Implicit Reward Discovery
Following the derivation process of non-adversarial imitation learning [15], the training objective can be expressed as an equivalent min-max problem. We have the following key result:
Theorem 1 (Equivalent Objective for Distribution Matching). Learning a policy that minimizes the f f-divergence between expert and policy state-action distributions is equivalent to first learning
an optimal policy under an arbitrary reward function, then optimizing a function of that reward function:
-min LF + max -r,, 3DT.. ( (T|I
-
(2)
Has closed- form solution
where f is the convex conjugate finction corresponding the chosen f f-divergence, pzz is the state-action distribution of the policy being learned, and μ&g is the expert's 's state-action distribution. is the independent variable of f and is commonty interpreted as the reward function.
We provide a detailed proof in Appendix A.1. Here, the reward function is not yet related to implicit rewards but rather represents an arbitrary function. The commonly used SFT loss still differs from distribution matching approaches in fundamental ways. However, this formulation establishes connection between finding policy model and identifying a suitable reward function. Although the equivalent objective formulated as a bi-level optimization problem, the latter part, which we denote as [=x = Ex, [r] Brr.. ( (et),, has a closed-form solution as demonstrated by [17]. We leverage this established result in our approach.
arg J(T), J(x)) = (so)
x
where ग 7'(a 8) is the optimal policy. V* is the value function of the optimal policy.
Lemma 1 (Relationship between Reward and Policy[17], Implicit Reward). The relationship between reward and corresponding optimal policy is:
n (y x)
r(x, y) = 3 log
+ (su)s - (st),,
(4)
Ty x))
where r(z, y) represents the reward for the LLM's imput-output pair (x.
As the reward in the training objective eq. (2) can be initialized arbitrarily, can have the following assumption.
Assumption 1 (Initial Reward Simplification). Without loss of generality, the initial reward can be treated as (Su) with (st) = O for all t O. Under this assumption, the initial policy π. is the optimal policy with respect the initial reward.
Now we can substitute the latter part of eq. (2) with its closed-form solution. The correspondence between divergence measures and their conjugate functions has been established in previous work [21]. We also list them in Appendix B. We select the total variation distance as our divergence measure, for which the corresponding conjugate function is simply the identity function. Additionally, the relationship between the reward and policy æ satisfies eq. (4). This allows us to directly obtain the final objective, which takes the familiar form of SFT:
max E [8 log x (y)()) - log (y)() - (s:),,
(5)
MLEE
Serve Constants
where s, represents the terminal state. The expected return after this state becomes constant.
Conclusion 1: Commonly used SFT a special case of finding implicit reward, same as DPO. We now derive that the commonly used SFT loss constitutes special case of reward discovery through imitation learning when total variation is selected as the f-divergence measure. Appendix B presents alternative training targets derived from different f-divergence functions. Since the derivation process above is reversible, we conclude that the SFT process searches along the optimal peeicy-reward subspace, attempting to model the reward implicitly embedded in expert demonstrations. At the start point of the SFT process, the policy is the optimal policy under Assumption 1. During the optimization process, the relationship between the model and reward continues satisfy eq. (4), resulting in searching the optimal subspace of pewicy-reward. This implicit reward structure aligns perfectly with that in DPO, offering a harmonious theoretical view that unifies both approaches.
5

---

KL term absent in commonly used SFT. The difference in eq. (5) between the reference model and policy model takes zero-order form, which acts as a constant when performing stochastic gradient descent. However, it constrains the update step size of the policy model and plays an important role most RI. algorithms. The absence of this term leads to a substantial distance between the post-DPO model's training starting point and the base model. We propose simple but effective method to mitigate this limitation. With smaller learning rates to reduce the optimization step size, show significant performance improvements in the instruction following domain in Section 4.2. Furthermore, by selecting different f-divergences, we can derive objectives similar to SFT while preserving the KL term. Most of these involve logarithmic and exponential operations that may lead to numerical instability. We select three representative divergences and present their comparative results in Section 4.3.
Conclusion 2:
3.3 Model Maintains Intrinsic Expected Return Estimation During SFT
In the DPO process, [17] noted that the logits of LLMs can be interpreted Q-function under mild assumptions. We extend this conclusion based on similar structures in eq (2).
Theorem 2 (Intrinsic Expected Return). During the SFT process, the logits la of a language model correspond to the Q-function Q(s, a) of the learned implicit reward:
la = a)(s,)))= C(s) a(s,a) + - (V+((11))
(6)
where i is the model's implicit reward function satisfying eq. (4), Y is the discount factor, and V1($++1) is the value function of the next state. C(s) is a function conditioned only on the state.
We provide detailed proof in Appendix A.3. Our findings indicate that not only in the DPO process but also in SFT, the model's logits can be interpreted as Q-function characterizing the model's estimation of expected returns. These returns are calculated based on the implicit reward Iearned by the model itself. The function C(s) represents the gap between the true Q value and the logits, but it does not affect the relative ranking among different actions since depends only on the current state and acts as a constant across actions. The value function can be calculated using log-sum-exp according to Appendix A, and we hypothesize that:
Assumption 2 C(s1) and (#2) is smaller than the difference between V(s1) and V V(22)-
Assumption). For most two states and s2, the difference between
This conclusion allows us to use the log-sum-exp of logits from the LLM as a Value function, instead of performing Monte Carlo sampling when utilizing other divergence formats.
4 Empirical Study
In this section, we provide empirical analysis on the cestuctiionfollowingg task. Our detailed discus- sion is presented as follows.
Small learning rate during SFT process can yield significant benefits for post-DPO models. We prove that reducing the learning rate to decrease the single-step optimization stride for SFT during sequential training improves results, as demonstrated Section 4.2.
Alternative f-divergences that preserve KL terms also lead to better results. Training targets derived from other f-divergence do not suffer from losing the KL term. We select Pearson x" and Squared Hellinger, which avoid numerical stability issues associated with logarithmic and exponential functions, to demonstrate these improvements in Section 4.3.
LLM logits exhibit value function properties, evaluating state quality similarly. By leveraging the characteristic of value functions reflect expected state quality, demon- strate that different models maintain similar judgments across states in Section 4.4.
SFT mitigates initial reward randonness and quickly aligns implicit rewards to rea- sonable values. We explain the role of SFT in post-training as correcting the initial reward Assumption 1. We demonstrate that (s0) converges rapidly during the SFT process and present the corresponding empirical results in Section 4.5.

---

4.1 Basic Experiment Setting
Model and Dataset Selection General instruction following is a fundamental capability of large language models required for most downstream tasks. Following the setting SimPO [29]. we select Llama3-8B [2] and Mistral-7B [5] as our base models. To demonstrate that our method yields consistent improvements across model scales, we additionally evaluate 5wen2.5-3B(33], UltraChat- 200K [34] is a commonly used SFT dataset. For general instruction-following tasks, models typically complete SFT training on UltraChat-2000 before performing DPO on Ultra-feedback [35] to obtain the final model. We use these two datasets in our experiments.
Hyperparameters, Device, Baselines, and Evaluation Benchmarks The most commonly used learning rates during post-training are 2e-5 for SFT and 5e-7 for DPO. For all of our experiments, we train with a batch size of 128 on 8xH100 GPUs using the OpenRLHF [36] framework. For the DPO training process, 8 is important hyperparameter, and we set 3 = 0.01. We evaluate our models using AlpacaEval2 [37], Arena Hard [38], and MT-bench [39]. Since these benchmarks can be influenced by many implementation details, such the vLLM [40] version, we maintain consistent implementation versions with SimPO. We use the same decoding parameters as SimPO during downstream evaluation. As evaluating all three benchmarks simultaneously would incur significant API costs, in some experiments, we used AlpacaEval2 as the representative benchmark.
4.2 Small Learning Rate SFT Leads to Better post-DPO Results
As mentioned in Section 3.2, the KL term, i.e., log Ter. that constrains the SFT learning process is zero-order term and provides no gradient contribution to policy optimization after differentiation. Considering the importance step size constraints traditional RL, we infer that the learning rate for SFT should be reduced to decrease the effective update magnitude. Compared with the commonly used 2 x 10 5, we implement smaller learning rates of 5 x 10 6 for Llama3-Bast Base and Qwen2.5-3B. 1 x 10 for Mistral during the SFT process. For the RL process following SFT, we select DPO and SimPO algorithms while maintaining the same hyperparameters as the 2 x 10 configuration. We utilize the publicly released checkpoints from the original SimPO implementation and evaluate them in the same testing environment to establish our baseline results.
Main Results Results presented in Table and Table 2. Our reproduced baseline results outperform the results reported in the original SimPO paper. We maintain identical settings ensure fair comparison. It can be observed that reducing the learning rate leads moderate improvements for SFT check- points and signíficant enhancements af- ter applying alignment algorithms. The SimPO results show relative improve- ments of 20% (absolute improvement of 5%) for Llama3-8B and 25% (absolute improvement of 6%) for Mistral after applying DPO. As we maintain identi- cal hyperparameters in the DPO train- ing process, the performance improve- ments primarily derive from adjusting
Table Results on Qwen2.5-3B. The other forms of SFT compared against the result of smaller learning rate.
Qwen2.5 (wen2.5-Base(3BB
Method
AlpacaEval
Arena-Hard
MT-Bench
LC
WR (%)
WR (%)
GPT-4 Turbo
GPT-4
Baxeline
3.1
6.1
DPO
9.9
Smaller Learning Rate SFT
SFT Irj
5.7+0.6
3.3+0
3.7
5.9
5.8
+ DPO
10.8
8.741.
11.6-11
7.0
6.9
Other Forms of SFT
Pearson-SPT DPO SH-SFT + DPO
6.540.8
3.7+0
3.2
6.240.3
6.140.3
11.3,0.
940.3
14.3
740.1
6.8
4.7
3,2
4.540.8
6.3
6.1
11.1
9.1
13.5
6.9
6.6
the learning rate during the SFT phase, which confirms our hypothesis.
4.3 Other Forms of Imitation Loss Behave Better in Sequential Training
Beyond the Total Variance divergence, other f-divergence functions shown Appendix B yield training targets where the KI. term is not limited to zero-order approximations. However, many alternatives involve logarithmic or exponential calculations, or even composite log-exp operations (as in divergence), which can lead to numerical instability. We select two additional
7

---

Table 2: Downstream results for the smaller learning rate setting. The reference-only result is reported by SimPO[29]. We reproduce the SFT, DPO, and SimPO results using publicly available checkpoints. The models trained with a smaller learning rate for SFT and subsequent models fine-tuned from this SFT checkpoint are marked in blue
Llama-3-Base (8B)
Mistral-Base (7B)
Method
AlpacaEval Arena-Hard
MI-Bench
AlpacaEval
grena-Hard
MT-Bench
LC
(R(%)
(R(%)
GPT4 Turbo
GPT4
(0
(R)
(R)%)
GPT-4 Turbo
GPT4
Reference Only. Not Compared Directly
RRHF[41]
11.6
10.2
5.8
5.4
6.7
12.1
10.1
5.8
7.0
SLiC-IIF(42]
10.9
8.9
7.3
5,8
7.4
12.3
13.7
6.3
7.6
CPO[43]
9.8
8.9
6.9
5.4
6.8
10.8
8.1
5.8
6.0
7.4
IPO[27]
11.8
9.4
7.5
5.5
7.2
14.4
14.2
17.8
6.5
7.4
KTO[28]
13.1
9.1
5.6
5.4
7.0
14.2
12.4
12.5
6.3
7.8
ORPO[44]
14.7
12.2
7.0
5.8
7.3
12.2
10.6
10.8
6.1
7.6
R-DPO[30]
17.4
12.8
8.0
5.9
7.4
17.6
14.4
17.2
7.5
Learning Rate Optimization For Supervised Fine-tuming
SFT
5.8
3.7
2.7
5.9
6.8
5.7
3.6
1.6
5.4
6.1
[P][[2]
17.3
14.2
19.7
6.8
7.3
15.6
12.5
11.7
6.3
6.4
[2mPO[[9]]
23.5
21.3
30.3
7.0
7.3
24.1
22.9
22.6
6.5
6.8
SFT (smaller Ir)
6.3.0.5
4.340.6
3.3_..6
5.9
6.3
6.7.1.0
3.7.0.1
3.1+1.6
5.740.8
6.4.0.3
19.4,2.1
16.2
21.1+1.4
7.0,0.2
7.4
21.5,5.9
16.7,42
21.6,9.9
6.5...22
7.0,..5
+ SimPO[[9]] 28.5,6.0 25.0
34.3
6.6
7.3
27.3
24.0,11
14.9
6.8
/-divergence formulations, which are Pearson x² and Squared Hellinger. Their derived training targets are presented in Table 3.
For Pearson x2, there exists a squared probability difference term that acts as KL constraint. For Squared Hellinger, coefficient term related to probability differences is multiplied before the classic gradient term after applying the chain rule, which modulates the update step size. We compare these results with our previously obtained Total Variance results.
Table 3: We list three optional f-divergences and their corresponding training targets. We use AV as an abbreviation for (=0) - Vst):) for simplicity of notation. See Appendix B for the complete table.
Name
Q)(PliQ)
Conjugate
(1) Training Target max, E
Total variation
$ (p()) q(2)) de
8 log (y)(:) log werl xr) (.( x(v(s) 10 (log Feel
Pearson x²
f [u(a)-pla)] dz p(m)
+ AV)2 log Feer(oln) + AV
s(vle)
Squared Hellinger f (Vrlr) {(7) dr
1+ Fraytole
winl +AV
Table 4: Downstream results for different training targets and their corresponding DPO check- points. SFT refers to the commonly used training target derived from total variation. Pearson-SS refers to the imitation objective derived from Pearson-x divergence. SH-SFT refers to the objective derived from Squared Hellinger divergence.
Llama-3-Base (8B)
Mistral-Base (7B)
Method
AlpacaEval
Arena-Han Hard
MT-Bench
AlpacaEval
Arena-Hard
MTT-ench
LC
WR (5))
(R())
GPT-4 Turbo GPT4
LC
(K)
WR(%)
GPT-4 Turbo GPT-4
Turho
Traditional SFT
SFT
6.3
4.3
3.3
5.9
6.3
6.7
3.7
3.1
5.7
6.4
DPO
19.4
16.2
21.1
7.4
16.7
21.6
6.5
Other Format of SFT
Pearson-SFF
5.1
3.7
3.3
5.8
6.1
6.2
3.4
3.5
6
6.5
DPO
20.1
17.741.5
24.7.3..
7.1
7.2
23.1
19.0
21.8
6.8
6.7
SH-SFT
4.8
3.7
17.3
5.7
6
6.5
3.5
3.1
5.7
6.1
+ DPO
19.6.0.2
17.3.
19.9
6,9
7.2
23.6+
20.9
22.
6.8
Main result and Squared Hellinger lead to weaker SET but better results after DPO, regardless of whether
The results are presented in Table can observed that both the Pearson x2

---

NoSeeA An
Paker Moze Wen Pudes
ERO ESS Siiue
A311
8.14
1000
0.0
0377
2.30
5-00
(a) KLCC between different (b) Alpaca-Eval 2 results and cor- (c) Training loss for different objec- checkpoints
responding DPO Loss.
tives.
Figure 2: Left: We obtain rankings of different models identical step-wist instruction- response pairs and calculate KLCC to measure agreement between these rankings. Middle: Lower values of DPO Loss indicate better alignment between implicit reward and environment reward. And the corresponding A AlpacaEval2 result. Right: Training loss for objectives derived from: total variation, Pearson x², and squared Hellinger.
use Mistral or Llama. We can reach an interesting conclusion that better SFT checkpoint doesn't necessarily lead to better DPO results. These improvements from KL-regularized SFT validate our theory, showing the importance of the KL term during post-training. The training loss curves for these three SFT approaches are shown in Figure 2c.
4.4 Value and Reward in SFT
With the theoretical conclusions presented in Section 3.3, we estimate the value function using the logits of L.LMs. Traditional value estimation typically involves Monte Carlo sampling. Moreover, values are conventionally calculated using ground truth rewards, which in our context are implicit and not directly accessible. We aim to provide empirical evidence that LLM logits exhibit properties of value functions: their scores can be used to evaluate state quality.
More precisely, we demonstrate that for LLMS trained on the same domain, the evaluations for different states maintain similar rankings across models. We divide the UltraChat-200k dataset into 4 splits and perform SFT on Llama-3-base to obtain different checkpoints. We also select Llama-3-instruce as representative model that shares the same prior but was trained on different datasets, and Zephyr [45] as model with a different prior but trained on similar datasets. For clearly defined steps, we choose MATH-500 [46] as the validation set, sample one trajectory for each question, and split the reasoning path into steps. We extract the logits from the model's s output after inputting the final token of each step, calculate the log-sum-exp, and rank these values within each individual model. If the logits of an LLM possess the property of evaluating state quality, they should share similar ranks across models. We calculate the Kendall rank correlation coefficient (KLCC).
The results are shown in Fig. 2a. We find that the value rankings of LLMS are positively correlated across all experimental settings. The correlation approaches 1 for the four dataset shards and remains positive even when the post-training dataset or model prior changes. An interesting observation is that the ranking correlation between Zephyr and Alama3--nstrus is significantly higher than that between Zephyr and our sharded models, despite Zephyr being trained on UltraChat rather than the same dataset. The positive correlation indirectly validates that the logits function as a value, thus confirming our Assumption 2.
Main Results
Another observation for implicit reward We find that stronger alignment between the model's 's implicit reward and downstream reward correlates with better model performance. During the SFT training process, we calculate the DPO loss on previously annotated pairwise data of AlpacaEval used as an evaluation set. As the benchmark serves as an environment, lower DPO loss indicates greater consistency between implicit rewards and the environment. The results are shown in Figure 2b. We observe that alignment between rewards and the environment positively correlates with model performance, which aligns with intuitive expectations.

---

4.5 Reward Stabilization through SFT
SFT in our theory attempting to bring the implicit reward into an appropriate range for fine-graine grained modifications. Assump- tion assumes that at the starting point of the implicit reward search process, the model represents the optimal policy under some unknown reward function. This reward function may significantly differ from the reward function in real downstream tasks. We plot the training curve of the log-sum-exp of the first logits after the prompt and create some early-exit SFT check- points to perform DPO on them, as shown in Figure 3. It can be observed that the V (so) log Σ() increases rapidly and converges quickly. The downstream task performance exhibits highly consistent trends. We conclude that SFT has already completed its task of bringing the implicit reward to a reason- able range by 150 steps, and subsequent steps focus more refined modeling.
2.5
B012 1800 3000
1000
Figure 3: The trend of ((so) using the logits of the first response token. Performance on AlpacaEval 2 for post-DPO on SFT checkpoints at corresponding training steps.
5 Broader Impacts, Limitations and Future Work
Broader Impacts: Philosophical Dimension As we mentioned, models learn implicit rewards dur- ing SFT, which may lead to further discussions about whether LLMs can be considered entities with preset environmental awareness. This opens philosophical inquiries into the nature of consciousness and the extent which artificial systems might exhibit consciousness-like properties.
Limitation: We did not experimentally explore additional divergence functions The commonly used KL divergence and IS divergence involve logarithmic and exponential calculations, which can lead to numerical instability. We have attempted various implementations and performed small-value clipping on the data. Although this prevented NaN errors, the loss would still reach extremely large values, such as 6e7. We did not design specialized operators to implement these methods, resulting in unknown effectiveness for these KI. divergence approaches.
Future Work: SFT and DPO multi-object learning Since both the SFT process and DPO process model the implicit reward, a natural idea is to formulate them into multi-objective learning rather than sequential training. Appendix C details some failed attempts at implementing this multi-objective approach, hoping these findings can contribute to future research in this direction.
6 Conclusion
In conclusion, we establish implicit reward learning as a unifying view connecting SFT and preference learning in LLM post-training. We demonstrate that conventional SFT is special case of implicit reward learning using total variation divergence, limited by an absent KL term. Our approach of reducing learning rates significantly improves model performance, while alternative f --divergence objectives preserving the KI. term show additional gains. We extend DPO's 's logits-to-Q Seiis-to-Q-functinnn mapping to SFT and confirm SFT's erucial role in stabilizing random implicit rewards, advancing both theoretical understanding and practical strategies for more effective post-training.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. U24B20181) and Shanghai Pilot Program for Basic Research Fudan University 21TQ1400100 (22TQ018).
References
[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano,
10

---

Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh. editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, IA,, USA, November 28 December 9, 2022, 2022. 1
[2] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jeimer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca. Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li. Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.211 21783, 2024. 7,
[3] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang. Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407.106711 2024. 1
[4] DeepSeck-AI, Aixin Liu, Bei Feng. Bing Xue. Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Yc, Shirong Ma. Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. Deepseek-v3 v3 technical report. CoRR, abs/2412. 19437, 2024. 1
[5] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample. Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06822 2023. 1.7
[6] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. LIMO: less is more for reasoning. CORR, abs/2502.03387, 2025.
[7] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and
11

---

Omer Levy. LIMA: less is more for alignment. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[8] DeepSeek-Al, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li. Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingehang Chen, Jingyang Yuan, Junjic Qiu, Junlong Li, L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang. Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan. Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan. and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501.12948, 2025.
[9] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. CoRR, (bbs/2402.03300) 2024. 1
[10] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao. Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self- self-improvement. CoRR, abs/2409.12122, 2024.
[11] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. CoRR, 706.0716,,,741, 2017.
[12] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. CoRR, abs/2305. 18290, 2023. 1, 3,
[13] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.7
[14] Yi Ren and Danica Sutherland. Learning dynamics of LLM finetuning. CoRR, abs/2407.100990
2024. 2, 4, 18
[15] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. CORR, 1et/2100..11442, 2021. 2, 4,5
[16] Markus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja, Jorg Bomschein, Sandy H. Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, Sarah Bechtle, Jost Tobias Springenberg, Nikola Momchev, Olivier Bachem, Matthieu Geist, and Martin A. Riedmiller. Imitating language via scalable inverse reinforcement learning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Uirich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024. NeurIPS 2024, Vancouver. BC, Canada, December 10- 15, 2024, 2024. 2, 3, 15
[17] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q Your language model secretly a fuunctio... CoRR, abs/2404.12355 2024. 2, 6, 15
[18] Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Pat Langley, editor, Proceedings of the Seventeenth International Conference Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 July 2, 2000, pages 663-670. Morgan Kaufmann, 2000. 3
12

---

[19] Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Dieter Fox and Carla P. Gomes, editors, Proceedings of the Twenty-Thing AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008, pages 1433-1438. AAAI Press, 2008. 3,
[20] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4565
4565-4573, 4573, 2016. 3
[21] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. CoRR, (ass/1606.00700 2016. 3,5
[22] Seyed Kamyar Seyed Ghasemipour, Richard S. Zemel, and Shixiang Gu. A divergence mini- mization perspective on imitation learning methods. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1259-127 1277. PMLR, 2019. 3
[23] Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Ruoyu Sun, and Zhi-Quan Luo. Entropic distribution matching in supervised fine-tuning of llms: Less overfitting and better diversity. CoRR, abs/2400.116677 2024. 3
[24] Chris Cundy and Stefano Ermon. Sequencematch: Imitation learning for autoregressive sequence modelling with backtracking. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. 3
[25] Hao Sun. Supervised fine-tuning as inverse reinforcement learning. CoRR, abs/2403. 12017, 2024.
[26] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 0121177770634777 2017.
[27] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo. Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Danicle Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238 of Proceedings of Machine Learning Research, pages 4447-4455. PMLR, 2024. 3,8
[28] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning, JCML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. 3,8 8
[29] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhangg editors, Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. 3, 7, 8
[30] Ryan Park, Rafacl Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages Linguistics, 2024. 3, 8
Association for Computational
[31] Joey Hong, Anca D. Dragan, and Sergey Levine. Q-SFT: q-learning for language models via supervised fine-tuning. CoRR, abs/241 0tss/2111.05 1.05 2024. 3
[32] Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J. Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. CoRR, abs/2503.01067. 2025. 4
[33] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan,
13

---

Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024.7
[34] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 3029-3051. Association for Computational Linguistics, 2023.
[35] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. CoRR, abs/2310.0137// 2023. 7
[36] Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143, 2024. 7
[37] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlle alpacaeval: A simple way to debias automatic evaluators. CoRR, 0ix/2404.044755 2024. 7
[38] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controllee alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404.04475 2024. 7
[39] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. 7
[40] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[41] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. RRHF: rank responses to align language models with human feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[42] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, abs/2305. 10425, 2023. 8
[43] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the bound- aries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. 8
[44] Jiwoo Hong, Noah Lee, and James Thome. ORPO: monolithic preference optimization without reference model. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 11 11170-11189, Association for Computational Linguistics, 2024. 8
of the 2024 Conference
[45] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
[46] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman. llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 2rXiv:2305.20051 2023.
14

---

A Theoretical Proofs
In this section, we provide detailed proofs of our theoretical results. Lemma 2 (Fixed-Poim) Point Solution for Maximum Maximum-Entropp RL [17]). The optimal policy x' (a, st) and the corresponding optimal value funetion (st) in the maximum-entirn framework satisfy the following fxed-poin equations:
a (at (1) = exp
(7)
B
(st): = Blog /
(Q"(H).
exp
dai,
(8)
A
J(x') = (sn),
(9)
where (a () is the optimal policy. Q" and V* are the quality function and value function of the optimal policy.
A.1 Non-adversarial Imitation: Finding Reward
Theorem 3 (Equivalent Objective for Distribution Matching). Learning a policy that minimizes the f-divergence between expert and policy state-action distributions is equivalent to first learning an optimal policy under an arbitrary reward function, then optimizing a ffuuctiinn of that reward function:
min Dy(ux|luE) + (T(FTT)
(10)
min Eng (-r) + max E,, [rl (DK((TTTT
(11)
closed-form solution
where f is the convex conjugate function corresponding the chosen f f-divergence, μzz is the state-action distribution of the policy being learned, and μ&g is the expert's 's state-action distribution.
Proof. Following the derivation process of non-adversarial distribution matching [16], the training objective 1 can be rewritten into min-miaeee -mx problem with a conjugate function:
Given that f (g)
max {ag f(x)},
f(x
max
{rg -f (g)),
(12)
xxdoomi
gEdom(f*)
(13)
min Dy + 3DKI (T((TTTTTT
(14)
/
Hx
min
uEf
dsda + (TT
(15)
FF
/ uE
H g - fz + DKL
min
max
(16)
9:Sx x:SAA A-dom((")
HE
min
max
Epp [f'( + Eyrr igl 3DDIL (T)).
(17)
x
x Adomm,
By substituting 9 with -r, rewrite the min-max formulation: Ef + E,, [r] - (
-mxx a
min
(18)
r: r:SxA- A-
The target is also saddle point problem, the conjugate function exhibits convexity and the KL term can be split into an entropy term that is concave and a cross-entropy term that is linear with respect n. Therefore, the order of min-max operations be exchanged:
min max r:S A
[f* E,, [r] 3 ( (()))
(19)
The maximization is only performed on the latter part of the expression and has a closed-form solution according to previous work [17]:
min E (-r)] + max E,, [r] 3DKL (e)
(20)
closed-form solution
15

---

A.2
SFT is a Special Case
The final objective is
min Giin (-r)] + max Ep [r] 3t.. (I (rer]
(21)
Has closed form solution
= min Bag ( + (sn).
(22)
Throughout the training process, the reward maintains the equation (4) and the policy æ is always the optimal solution of the latter parts. When choosing the Total Variation distance, the conjugate function is (*) = t, and the training objective is just the MLE term shown below:
min E [f (-r)] V (50),
(23)
max B (- (-r)]] (sn),
(24)
max ys [r] V (so),
(25)
* (v]
max p Dlg
+ Vi V=($s]] - V=(50),
(26)
max Eu [B log T (tlar))
log Ter (ylz)) -V (s)))
(27)
MLE
constant
A.3 SFT Maintains Intrinsic Expected Return Estimation
Theorem 4 (Intrinsic Expected Return). During the SFT process, the logits la of a language model correspond to the Q-function (s,, a) of the learned implicit reward:
La Qr C(s) f(s,a) + PeBess (F
(28)
where i is the model's implicit reward function satisfying eq. (4), is the discount factor, and Vi($++1) is the value function of the next state. C(s) is a function conditioned only on the state.
Proof. The language model represents token probabilities through a softmax operation over logits l:
e///
plaa(s)
(29)
Ejelit
where is the temperature parameter, typically set during training.
As previously discussed, the model represents the optimal policy under some implicit reward function. The probability distribution for this optimal policy satisfies:
el(.,.))18
p(a.(s) E, c(s,0)))
(30)
Equating these expressions, we have:
ch/r
Qisai)/1
E, Ejelii = E, e(oraa))13
(31)
Ejebl
clil1 = olean)10
(32)
Σ E, e((a,))11
ct/l = -
(33)
I a.) 4 -=--0
(34)
where is the KL divergence coefficient. The relationship shows that logits have linear mapping to the --vluus, with C(s) depending only on the state since it incorporates terms summed over all possible actions. We hypothesize that C(s) remains numerically similar across different states, and we provide supporting evidence for this in the experimental section.
16

---

B Different f-Divergence Leads to Different Loss Format
In this section, we present the detailed formulation of loss functions derived from different f- divergences.
Name
()()))
Conjugate (1)
Training Target
max, Ep [-]
Total variation
J [p(x)) 2(x)) dzr
=log((ylx)) log Nerlyle) - (s.(s))
Kullback-Leibler
--Fro(()) [
(KL)
I p(a)) pla) da ((x))
exp(t - 1)
exp( AV -
log(log x(yl=) ))e((g|))
AV)
Reverse KL
J g(x)) log p6o dæ
-1 log(-t)
-V((s))
s(yla) - (log (00))
+ AV)?
Pearson x²
(ulx)-p(a)>? dæz
1 +
+ log (xa)(
r(yla)
+ AV
Squared Hellinger
J (Vplz)-Va(an) Va(z)) dr
14 Tie Spafieln
2p(ar) f(r)) log ((()) +
log(2 Trer(ula) [(yle))
Jensen-Shannon
log(2 exp(t))
((x) log port
20la) dz
exp(-AV))
3p(ar) p(it) log p(x)+
log (ulr( x(yle)
GAN
log((1 exp(t))
((x) log p(r)+a(() de log(4)
24(a)
exp(-AV))
Table 5: different f-divergences, we their corresponding conjugate functions f the derived training targets. We use the symbol AV as an abbreviation for V= (so) V=(+)) for simplicity of notation.
C Multi-Object
Since both the SFT process and DPO process model the implicit reward, we investigate whether they can be formulated as multi-objective learning rather than sequential training. We approach this using Lagrangian methods. Specifically, we formulate the multi-objective learning process using the Lagrangian multiplier method. We assume that during the SFT process, the accuracy of the DPO loss should also be kept as low as possible. We formulate this with the following objective:
min SFT loss, s.t., (aceuracy of (chosen, reject) target ace) < 6,
(35)
min SFT loss + X DPO loss.
(36)
Where is the hyperparameter. Following the process PPO, the parameter adjusted to 3 when acc target acc < 6 and is set to 2 otherwise. We train the model directly on the UltraFeedback dataset, and some training metrics during one training shown below.
17

---

Training Metrics Overview
trainspret
1.4e-00
7.5e-01
7.0e-33
1,4+000
5.5e.22
12e+00
5.0-22
1.0e100
3.5--1
5.0-03
6.0c-11
1.0
Jpochesen Set, spo, Jargeii.
1.0e000
3.00-21
R.0+01
6. 6.0e-53
4.00-22 01
2.0e-22
0.0e-50
1.0
5.00.1
4.0e-3
2 3.
0.
unt. spechesen Set. spo IN10002 sargett
Trinighebal
Figure Training Metric Visualization for Multi-object Learning
We also experiment with various hyperparameters, including different growth and decay coefficients for X, but all of these configurations lead to poor downstream results. We think that our failure appears consistent with the theory presented in [14]. As the SFT loss coefficient increases, it tends to increase probability mass in negative regions, thus diminishing the effectiveness of the squeezing effect. We validate in other experiments through interleaved experiments, where we split the SFT and DPO data into four segments and train them in an interleaved manner. The results are shown below.
Table 6: Results for interleaved SFT and DPO training. The table shows the performance after each training stage in terms of Win Rate and LC Win Rate percentages.
AlpacaEval
Training Stage
Win (%)
LC-Win (%)
Interleaved Training
SET
5.28
4.86
DPO
7.03
8.11
2
5.06
4.63
DPO 2
11.27
11.94
SFT
8.94
4.83
DPO 3
12.94
13.30
SFT
8.44
5.13
DPO 4
8.24
5.02
can be observed that after each DPO stage, subsequent SFT training experiences a significant per- formance drop, indicating that SFT and DPO objectives are inherently confficting in our experimental setting.
However, failure does necessarily mean this approach is fundamentally infeasible. We have not yet explored its effectiveness in other domains, and we leave this investigation to future work. We hypothesize that in tasks where expert trajectories more closely align with SFT's assumption that the training data represents optimal behavior, the results may be more promising.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We finish this in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We finished this part in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
We provide all of our assumptions in Section and our corresponding proof in shown in Section A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provided our training detail and other implementation detail in Section 4
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
The dataset we used are public accessed and we also give out the detail in
Section 4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
Seeeii/guuides/oeeuuiieeee soiobiiisin for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSubmiis
httppsscccpubiic//eecc / ssi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All of them are listed in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Most of the benchmarks we use already take statistical significance into account. We also illustrate statistical significance in figures such as 2a.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mentioned that in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttps:///nuris.... wttppppp.cccpppeeateeeee httppp/lccethiccccueeeeeee..eeee www.iihisseeellliccceeccc ines?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed in Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve this.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve this.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We mentioned them in Section 4
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: thttps://neet /ttpass:////u...//eeee 220 for what should or should not be described.
25


Reviews:
[{"review_id": "a8nezY8uvu", "reviewer": "Reviewer_4L4H", "summary": "This paper reframes the relationship between SFT and DPO, arguing they both exist in an \"optimal policy-reward subspace\" where an implicit reward is being learned. The core of the work is deriving the SFT objective from an f-divergence minimization problem, showing that SFT corresponds to using the total variation distance. This viewpoint highlights that SFT lacks the explicit KL-regularization present in DPO and other RL algorithms. The authors propose that this missing regularization can be compensated for by using a small learning rate or by adopting alternative f-divergence objectives for SFT. Experiments show these modifications to the SFT stage improve the final performance of a model after it undergoes DPO.", "questions": "1. The paper frames DPO as navigating along a \"chosen-rejected\" direction and SFT as moving along an \"average direction\" (Figure 1). Could you formalize what this \"average direction\" means in the context of your implicit reward framework? Is it the gradient of the expected reward over the expert demonstration distribution?\n2. How does your framework relate to other methods that modify the SFT objective, such as ORPO, which combines the SFT and preference-learning objectives into a single loss function? Does your theory offer any insight into why such approaches might work?\n3. You show that better regularization during SFT leads to a better starting point for DPO. Does this imply that the reference model used in DPO should ideally be the final, regularized SFT model rather than the original pre-trained base model?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper does a good job of building a conceptual and mathematical bridge between SFT and DPO, which are often treated as separate, sequential steps. The \"implicit reward\" framing is interesting.\n- The analysis pinpoints the theoretical weakness in the standard SFT objective of lacking a KL constraint, which explains some of the known drawbacks of SFT.\n- The improvements from reducing the learning rate are clearly demonstrated. The finding that alternative f-divergences can also improve post-DPO performance (Table 3) is a compelling piece of evidence for the overall theory.\n- The paper is well-written and well-structured. Figure 1 provides an excellent schematic for the core idea, and the mathematical arguments are sound.\n\nWeaknesses:\n- The extension of the logits-as-Q-function theory to SFT (Theorem 2) relies on Assumption 2. This assumption is quite strong and is not directly validated. A more thorough discussion of when this assumption might hold or fail would strengthen the claim.\n- A fascinating result in Table 3 is that the Pearson and SH-SFT models, which perform worse than the standard SFT model on their own, produce better post-DPO models. This strongly supports the paper's thesis about regularization but is a non-obvious finding. The paper could benefit from a slightly more detailed discussion of this implication: that the best SFT model (by imitation metrics) may not be the best foundation for preference tuning.\n- The main practical takeaway is to use a smaller learning rate for SFT. While the paper provides a new theoretical justification for this, using smaller learning rates for more stable fine-tuning is a common heuristic. The novelty is therefore more in the \"why\" than the \"what,\" and the practical impact could be seen as an incremental improvement on existing best practices.\n- The framework provides a unified perspective but does not result in a unified algorithm. The SFT->DPO pipeline remains sequential. The paper shows that doing SFT \"better\" helps DPO, but the two objectives are still optimized separately. The failed multi-objective experiments underscore that a true unification is still an open problem.", "was_revised": true, "final_justification": "While the paper provides fresh theoretical perspectives on SFT and DPO, the practical validation is not thorough enough (e.g., on old and small base models), and the improvement is relatively marginal. Therefore, I am not confident in its significance.", "rebuttal": "Thank you for your patient reading and your very insightful questions.\n## About logits-to-q \n\nYour question about logits-to-q is very perceptive. There are three points we would like to discuss with you:\n1. As we mentioned in L187, the relationship between logits and the q function in the DPO process has also been discussed in previous works(https://arxiv.org/abs/2404.18922, https://arxiv.org/pdf/2404.12358). We derived a consistent mathematical structure based on the SFT optimization objective and extended the logits-to-q conclusion from the DPO process to the SFT process. Assumption 2 is similar to that made in previous work.\n2. Beyond theoretical proof, we provide indirect experimental evidence in Section 4.4. Since Q-V functions represent expected returns and their numerical values indicate the quality of state-action pairs, we showed that different models share some common judgments in their quality assessments. This indirectly proves the correctness of logits-to-q.\n3. After thoroughly investigating the validity of logits-to-q, we find that the magnitude of logits changes is more pronounced (Avg std. 5.81) when two models share similar judgments, compared to when they have opposite assessments of state quality (Avg std. 2.76). This also supports the validity of Assumption 2.\nWe hope these explanations can make the logits-to-q conclusion and corresponding assumption 2 more convincing.\n\n## A better SFT starting point for DPO may not show better performance on metrics\n\nThank you for your interest in the phenomenon we proposed. To the best of our knowledge, the commonly used methods for selecting starting points for DPO are greedy. The conclusion we proposed in L268 may serve as a contribution to correcting misconceptions when selecting the starting point for DPO. Our explanation for this is that these SFT checkpoints may not be the best ones for performing a specific task, but they maintain better diversity in responses, i.e., higher entropy. This makes these SFT checkpoints better starting points\n\n## More new insights beyond the small learning rate provided by theoretical analysis.\n\nYour insight about empirically using small learning rates to stabilize fine-tuning during post-training is very perceptive. Our theoretical analysis obtains the same methods from a KL perspective, which can serve as a theoretical explanation. Beyond this, we also proposed a series of imitation learning objectives that maintain KL properties in Section 4.3, analyzed the implicit reward properties during SFT in Section 4.4, and attempted to answer the role of SFT in post-training in Section 4.5. We hope these new insights from theoretical analysis can also excite you!\n\n## About the unified algorithm combining SFT and DPO\nThe failure to combine the SFT and DPO processes into one process is regrettable. However, we have made some progress toward this goal in this work, as we have obtained some theoretical guarantees. There are many possible reasons for the failure of multi-objective optimization, and we believe the most likely cause is data quality. We leave this to future work and hope our progress can inspire subsequent insights.\n\n## About the average direction in Figure 1\nYou are absolutely correct. Here, \"average direction\" means that, assuming there are four data points in a batch, as shown in Figure 1 of our paper, these four data points contribute equally to the estimation of the optimization direction, that is, to the estimation of the reward gradient.\n\n## Some more theoretical proof of ORPO\nYour questions about other methods are interesting! We provide some extended theoretical analysis here about ORPO. The original optimization objective of ORPO is written as:\n$$L = L_\\text{SFT} + \\lambda L_{OR}$$\nThe first part is the classic SFT loss, which can be transformed into a reward learning process through the same approach. The second part can be expanded and written as:\n$$L_\\text{OR} = \\log \\sigma(\\log \\frac{\\frac{P(y_w|x)}{1-P(y_w|x)}}{\\frac{P(y_l|x)}{1-P(y_l|x)}} )$$\nConsider the term inside the $\\log \\sigma$\n$$\\log \\frac{\\frac{P(y_w|x)}{1-P(y_w|x)}}{\\frac{P(y_l|x)}{1-P(y_l|x)}} $$\n$$= \\log \\frac{P(y_w|x)}{1-P(y_w|x)} - \\log \\frac{P(y_l|x)}{1-P(y_l|x)} $$\n$$= \\log \\frac{P(y_w|x)}{P_\\text{ref}(y_w|x)} - \\log \\frac{P(y_w|x)}{P_\\text{ref}(y_l|x)} + \\log \\frac{1-P(y_l|x)}{1-P(y_w|x)} + \\log \\frac{P_\\text{ref}(y_w|x)}{P_\\text{ref}(y_l|x)} $$\n$$= \\log \\frac{P(y_w|x)}{P_\\text{ref}(y_w|x)} - \\log \\frac{P(y_w|x)}{P_\\text{ref}(y_l|x)} + \\log \\frac{\\Sigma_yP(y|x) - P(y_l|x)}{\\Sigma_y P(y|x) - P(y_w|x)}+ \\log \\frac{P_\\text{ref}(y_w|x)}{P_\\text{ref}(y_l|x)} $$\n$$= \\log \\frac{P(y_w|x)}{P_\\text{ref}(y_w|x)} - \\log \\frac{P(y_w|x)}{P_\\text{ref}(y_l|x)} + \\log \\frac{\\Sigma_y\\frac{P_\\text{ref}(y|x)}{P_\\text{ref}(y_l|x)}\\frac{P(y|x)}{P_\\text{ref}(y|x)} - \\frac{P(y_l|x)}{P_\\text{ref}(y_l|x)}}{\\Sigma_y \\frac{P_\\text{ref}(y|x)}{P_\\text{ref}(y_w|x)}\\frac{P(y|x)}{P_\\text{ref}(y|x)} - \\frac{P(y_w|x)}{P_\\text{ref}(y_w|x)}} $$\n$$=r_w - r_l + \\log \\frac{\\Sigma_y\\frac{P_\\text{ref}(y|x)}{P_\\text{ref}(y_l|x)}\\exp(r(x, y)) - \\exp(r(x, y_l))}{\\Sigma_y \\frac{P_\\text{ref}(y|x)}{P_\\text{ref}(y_w|x)}\\exp(r(x, y)) - \\exp(r(x, y_w))} $$\n\nThe $r_w - r_l$ is the classic BT model, aiming to widen the difference in reward between the chosen sample and the rejected sample. The latter part takes the total amount of reward into account. After dividing both the numerator and denominator by $\\Sigma_y P_\\text{ref}(y|x) \\exp(r(x, y))$, we obtain:\n$$ r_w - r_l + \\log \\frac{\\frac{1}{P_\\text{ref}(y_l|x)} - \\frac{\\exp(r(x, y_l))}{\\Sigma_y P_\\text{ref}(y|x) \\exp(r(x, y))}}{\\frac{1}{P_\\text{ref}(y_w|x)} - \\frac{\\exp(r(x, y_l))}{\\Sigma_y P_\\text{ref}(y|x) \\exp(r(x, y))}} $$\n$$ =r_w -r_l + lop \\frac{c_1 - \\text{ratio of } r_l}{c_2 - \\text{ratio of } r_w} $$\nThis can be viewed as a correction term that represents, beyond increasing the reward difference between win and loss samples, further consideration of the total reward allocation for other possible responses. It considers the relationship between the current sample's reward allocation and the total reward allocation, requiring that the win portion accounts for a higher proportion of the total reward allocation than the loss portion, thereby further constraining the reward learning process.\n\n## Choosing KL-SFT checkpoint as the reference model in DPO is better\n\nYou reach an absolutely correct conclusion about the reference model selection during the DPO process. To the best of our knowledge, current DPO implementations have already begun using the model after the SFT process as the reference model instead of the base model. In our work, we further showed that the KL-constrained SFT checkpoint is a better choice for the starting point of DPO. This phenomenon maintains high consistency with our theoretical analysis.\n\nWe hope the above response finds you well. Thank you again for your time for our work."}, {"review_id": "xDUskQrnjJ", "reviewer": "Reviewer_GAjJ", "summary": "Can we understand supervised fine-tuning (SFT) and direct preference optimization (DPO) as part of a unified reward-based framework? The paper present argues that SFT is in fact optimizing an implicit reward function, just like DPO does explicitly. This reframes both SFT and DPO as reward-based methods under a shared mathematical umbrella. The authors show that SFT corresponds to inverse reinforcement learning (IRL) by matching expert distributions. They point out a limitation of SFT, where the KL term in distribution matching becomes constant during training—making SFT unable to constrain model updates effectively. To address this, they advocate using alternative divergences (e.g., Pearson χ²) to recover stable reward shaping and show that this leads to better downstream DPO performance. They also generalize prior work connecting LLM logits and Q-functions in preference learning, extending it to SFT as well. Experiments with LLaMA-3 and Mistral on UltraChat/UltraFeedback validate their theory.", "questions": "# Questions\n\n1. Could the unified framework potentially support more complex multi-turn feedback protocols beyond binary preferences (e.g., fine-grained ratings, trajectory-level reward models)?\n2. Is there a formal characterization of when the logits-to-Q-function approximation in SFT breaks down? Under what training regimes or tasks does this interpretation become unreliable?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strengths\n\n* The paper theoretically unifies two central components of LLM alignment — SFT and DPO — under a shared framework based on implicit rewards and distribution matching.\n* It clearly identifies a flaw in conventional SFT training: the constant KL term fails to regulate model updates and can harm downstream performance. This is a non-trivial insight with actionable implications. \n* It goes beyond theory and proposes two concrete techniques backed by experiments: (1) reducing the SFT learning rate, and (2) switching to divergences (χ², Squared Hellinger) that preserve gradient signal.\n* Empirical results validate theory across multiple models (Mistral, LLaMA-3) and evaluation sets (AlpacaEval, Arena-Hard, MT-Bench), showing large relative win rate improvement post-DPO.\n* The extension of the Q-value/logit interpretation to SFT broadens its usefulness for reward estimation and could inspire further work on early-stage reward shaping diagnostics.\n\n# Weaknesses\n\n* The experimental validation is focused on a narrow domain: instruction tuning with UltraChat and UltraFeedback. It's unclear if the same effects hold in tasks with fundamentally different reward structures (e.g., code generation, tool use, multi-turn dialog).\n* The Q-function interpretation of logits in SFT is supported mostly by correlation experiments rather than rigorous causal or ablation-based testing. No direct test of how useful this proxy is in practice.", "was_revised": true, "final_justification": "After carefully reviewing the papers rebuttal and other conversations on this thread I maintain my score.", "rebuttal": "We are very grateful that you are willing to spend time carefully reading our work and for your high recognition of this research.\n\n## About Different Reward Structure\n\nOur work mainly provides theoretical relevance for understanding the SFT process from the perspective of implicit reward. Thus, alignment algorithms that use implicit reward can easily establish connections with our theoretical framework. For different reward structures like complex multi-turn feedback, there are already some works based on implicit reward, such as https://arxiv.org/pdf/2406.14868, that can establish connections with our work via implicit reward. At the same time, because in our imitation learning process, we target the finest-grained reward function, i.e., token-level reward, trajectory reward, or step-level reward can all be obtained through additive aggregation from token-level reward, allowing for very natural theoretical extensions.\n\n## About Logits-to-Q\n\nYour interest in the logits-to-Q extension is very insightful. As we mentioned in L187, the logits-to-Q properties in the DPO process have already been pointed out in previous work. Our work derived a consistent mathematical structure in the SFT process and extended the theory. There are works(https://arxiv.org/abs/2502.01456, https://arxiv.org/abs/2404.18922) that have already used the logits-to-Q properties in the DPO process to guide the learning process and achieve good results.\nPractically, we found that in the experiments conducted in Section 4.4, the logits of LLMs show greater differences when they have consistent judgments(Avg std. 5.81) about quality compared to when they have opposite judgments (Avg std. 2.76). Since the statistical data shows that the quality judgments of the two models are positively correlated in the experiments, we suggest treating the conclusion of non-validity as random fluctuations caused by tokens having insignificant contributions to the final reward. We hope this can make the logits-to-Q question more convincing.\n\nThank you again for taking the time! We are very grateful for your recognition of our work!"}, {"review_id": "CKPIg8AGDS", "reviewer": "Reviewer_dTok", "summary": "The paper attempts to connect SFT and preference learning through the idea of the implicit reward and describes SFT as distribution matching using the TV distance without a KL term. Through this, they describe SFT as searching the optimal policy-reward subspace and suggest using smaller learning rates for SFT. They also provide alternatives to standard SFT derived from different f-divergences. They claim that the logits after SFT correspond to state quality. Experiments are performed to verify these claims.", "questions": "- Can you provide some justification for Assumption 1? as well as for why you expect the cross-entropy between the policy and the reference to be small? Output probabilities do change significantly over the course of SFT especially if introducing a chat format. \n- Can you provide some clarification on eq. 4? What is $s_t$? and what is meant by $V^*$ is the value function of the optimal policy?\n- Could you provide experimental results for SFT with the KL term? \n\nJustification on the assumptions and steps taken are the current main concern and better explanations or experiments demonstrating that these steps are reasonable would increase the score.", "limitations": "The assumptions necessary may be a limitation that needs to be addressed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Background on distribution matching and the problem setting was clearly presented with definitions and explanations. \n- The paper provides a novel analysis that provides an understanding of the connection between SFT and DPO which is a common part of the language model training pipeline. \n- The paper overall clearly states results and assumptions with empirical verification of results. \n\nWeaknesses:\n- There are some steps that need further justification. Mainly Assumption 1 which makes a rather strong assumption about the value function and the claim about the KL term in the derived SFT being treated as a constant with little effect. It seems unclear how true or practical these claims are and seem to be an important part of being able to draw the connection using the implicit reward. This is a large weakness especially since the claim about SFT searching the optimal policy-reward completely depends on this assumption and following results build on this. Justification for replacing entropy with KL and why the cross-entropy term is small is currently weak. \n- Using the identity function as the conjugate function for TV only holds over a small range so if pi starts to deviate more from the initial model, this connection is no longer valid and also further suggests that the KL term in SFT could have a non-trivial effect. \n- It seems in the comparisons with other objectives from f-divergences, SFT is used without the KL term which seems like an unfair comparison. Also SFT with the KL term is not used in any of the experiments.", "was_revised": true, "final_justification": "Results are mostly valid with new insights of SFT", "rebuttal": "Thank you for taking the time to read our work. Your judgment about the importance of KL is very accurate. There may be some aspects we haven't expressed clearly, so we would like to restate our method here:\n- The post-training process typically includes multiple learning methods such as SFT and RL, and their theoretical connections are underexplored. To deeply understand what the SFT process is actually doing, we return to the maximum entropy imitation learning optimization objective and make corrections that align with language modeling, adding a cross-entropy term to the entropy term. The goal is to keep the model from deviating from the language distribution while maintaining explorability.\n- We follow the derivation approach of non-adversarial imitation learning and obtain a series of imitation learning objectives. When the f-divergence is chosen as total variance, it exactly equals the commonly used SFT objective function. We also obtained some new insights about the SFT process, which we list as follows\n- The above derivation process proves that SFT is actually also learning implicit reward. According to Equation 2, SFT is equivalent to searching for the most suitable reward in the optimal reward-policy space based on the given expert demonstrations. At the same time, DPO also proves that in reference learning, we can learn implicit rewards to directly fit preference signals. So implicit reward can naturally serve as a unified perspective to connect with DPO.\n- We found that when taking total variance, the original KL derivative term -log π_ref is a zero-order term in stochastic gradient descent and does not contribute to the gradient. In RL setting, the role of KL is mainly to constrain the magnitude of single-step updates. We constrain the step size of single-step optimization in the same way by reducing the learning rate.\n- Since the derivation process has the same mathematical structure as DPO, we simultaneously extend the connection between LLM logits and Q-function from the DPO process to the SFT context. We found that the logits of LLM during the SFT process can also serve as a Q function. They represent the expected future return starting from the current state and can serve as a judgment of the quality of the current state-action pair.\n\n## About the cross-entropy term in KL will be small\n\nYou proposed a thoughtful question! In fact, this value is relative. When LLM pre-training converges, the cross-entropy loss typically ranges from 1.x to 2.x(https://arxiv.org/pdf/2407.21783). For the original maximum entropy, taking Qwen(https://arxiv.org/abs/2309.16609) as an example, the entire vocabulary size is 128,256, so the maximum entropy at uniform distribution is log(1/128,256) = 5.11, which is significantly larger than the cross-entropy loss. At this point, it completely loses the characteristics of language distribution. Therefore, although the SFT stage changes output probabilities, as long as the output is natural language, the cross-entropy loss constraint is reasonable.\n## What is $V^*$\n\nFor any given reward, there must be a corresponding optimal policy. The expected future return that the current policy will obtain in a certain state is defined as $V^*$.\n\n## The KL deficiency problem in SFT\n\nYour judgment about KL deficiency in SFT is absolutely correct! This is also one of the core contributions of our work. We found that the KL term, which originally plays a huge role in the imitation learning objective, becomes ineffective due to the corresponding term under total variance, i.e., -log π_ref in Equation 5, being a zero-order term. Therefore, we made some corrections to common SFT tricks, including small learning rate correction and other imitation loss corrections. We found that these effects all show significant improvements.\n\nFinally, thank you again for your insights, and we look forward to further discussion with you!"}, {"review_id": "tEk5krGPpQ", "reviewer": "Reviewer_SPrv", "summary": "This paper presents a unified theoretical framework that connects Supervised Fine-Tuning and Direct Preference Optimization in the post-training of Large Language Models. The authors argue that both SFT and DPO can be viewed as learning under an implicit reward function within the same policy-reward subspace. They derive the SFT objective from f-divergence-based distribution matching and show it is a special case of implicit reward learning when total variation is used. Based on this insight, the paper proposes two practical improvements: (1) reducing the learning rate during SFT to compensate for the loss of gradient information from the KL term, and (2) using alternative f-divergence formulations that preserve KL structure. Empirical evaluations on standard instruction-following benchmarks demonstrate that these simple changes sometimes improve post-DPO performance.", "questions": "Q1: Why do the benefits of different SFT variants only emerge after post-alignment training? The paper observes that certain SFT configurations perform better after DPO but not immediately after SFT. Are there theoretical or empirical insights into why this discrepancy exists? Why would SFT effectiveness not manifest directly in post-SFT performance?\n\nQ2: Line 178–180: The authors argue that omitting the KL term leads to a “substantial distance” between the policy and reference model, implying this is undesirable. However, prior works have shown that SFT alone is effective and a larger distance may offer greater expressivity. Why is substantial distance necessarily a negative factor in this case?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength.\n\n- **Theoretical contribution.** The paper offers a rigorous and insightful theoretical framework that unifies SFT and DPO through the lens of implicit reward learning. This clarifies an often empirically observed but poorly understood connection.\n- **Practical utility.** The proposed improvements are easy to implement and demonstrate consistent gains across different models and datasets, making them applicable in real-world settings.\n\n---\n\n## Weaknesses.\n\n- **W1: Insufficient justification for selected learning rate**\n\nIn Section 4.2, the authors claim that using a smaller learning rate during SFT leads to better post-DPO performance. However, the choice of 5e-6 as the \"small\" learning rate seems arbitrary. A more thorough hyperparameter study would be helpful to validate this claim. Would further reducing the learning rate continue to improve results, or is there an optimal range?\n\n- **W2: Limited model evaluation**\n\nAll experiments are conducted on 7B-level models. It remains unclear whether the observed phenomenon generalizes across smaller (e.g., 0.5B, 1.5B, 3B) or larger (e.g., 13B) models. Additionally, the authors did not experiment with state-of-the-art open-source models such as the Qwen series, which may yield more robust or contrasting results.\n\n\n- **W3: Lack of discussion on Table 3**\n\n Table 3 contains several outlier values, and the proposed SFT variants show inconsistent performance across benchmarks. However, the authors provide no analysis or explanation for these inconsistencies, leaving unclear when and why the proposed methods succeed or fail.\n\n- **W4: No representation-level analysis**\n\nThe claims regarding KL divergence and its effect on alignment could be further substantiated by analyzing changes in the learned representation space. For instance, examining whether the regularization leads to more structured or aligned latent spaces would strengthen the argument.\n\n\n- **W5: Limited generalizability beyond DPO**\n\nThe paper does not provide sufficient analysis on the sensitivity of key hyperparameters, such as the clipping bounds in pre-clip, the threshold in advantage filtering, number of training data samples, or the weights in the final reward aggregation. Without such studies, it is difficult to assess the robustness and reproducibility of the proposed StableReinforce algorithm.\n\n- **W6: Presentation issue**\n\nThe reference to a figure on line 45 appears incorrect.", "was_revised": true, "final_justification": "I find the theoretical framework of this paper interesting. The authors addressed some of my questions, but most of the experiment-related concerns remain unaddressed. However, I still have doubts about the practical value of the claim that “a smaller learning rate generally leads to better performance.” Although the authors added a new model, I do not see its results on \"Other Imitation Loss\" experiments. Overall, I still consider the practical applicability of this work to be limited, and the current experiments are insufficient to fully support its theoretical framework (align with reviewer 4L4H). Therefore, I will keep my score unchanged", "rebuttal": "Thank you for your kind suggestions.\n## About learning rate selection\nRegarding the first point in the weaknesses, we also tried other learning rates. For the Llama model, with a learning rate of 1e-6, the DPO results were 18.83 and 17.26, which are also better than the commonly used 2e-5 learning rate. Our SFT learning rate adjustment is a trick based on theoretical properties, and the results with other learning rates on Llama hopefully demonstrate the universality of small learning rates for performance improvement.\n## About model selection\nWe followed SimPO's experimental settings to facilitate using their results as baseline comparisons. SimPO uses exactly the same settings as ours, and such model configurations are very common in preference learning, as can be seen in SimPO, DPO, ORPO, and GEM. So, maybe this setting is enough?\n## About SFT and DPO relationship\nOur work mainly focuses on the connection between SFT and DPO. As we mentioned in the experimental section L268, a good SFT starting point may not necessarily score high on benchmarks, so SFT performance has variability, but we achieved significant improvements on algorithms like DPO and SimPO.\n## About representation-level analysis\nIn Section 4.4, our analysis of logits changes, and in Section 4.5, our visualization of logits value changes during the SFT process, are both demonstrations of what the model is doing at the representation level.\n## About Key parameters\nOur work does not involve the key parameters you mentioned, such as clipping bounds in pre-clip and the threshold in advantage filtering. Your question has left us very confused.\n## About Typo in Line 45\nThe figure reference on line 45 is incorrect, and we sincerely apologize.\n## About why increments emerge after post-alignment training?\nSFT mainly provides a better initial point for DPO in the post-training process. As we limited the update magnitude during the SFT stage via kl or a small learning rate to ensure the exploratory nature of the SFT checkpoint, it may not perform well on specific downstream tasks. However, it maintains output diversity and acts as a better starting point for alignment algorithms, which contributes to better results after performing alignment algorithms.\n## About why KL is suitable\nThe distance relationship between the SFT model and the base model is also relative. When LLM pre-training converges, the cross-entropy loss is around 1. However, for the original maximum entropy, taking Qwen as an example, the entire vocabulary size is 128,256, so the maximum entropy at uniform distribution is log(1/128,256) = 5.11. It is significantly larger than the cross-entropy loss. When the maximum entropy is achieved, it completely loses the characteristics of language distribution. Therefore, although the SFT stage changes output probabilities, as long as the output is natural language, the cross-entropy loss constraint is reasonable. We constrain the relationship with the base model to maintain the rich natural language modeling in the base model."}, {"review_id": "85i1p8Jqoc", "reviewer": "Reviewer_6WqT", "summary": "Previously, [12,17] have established the relationship between DPO and implicit reward learning and how LLM logits can function as a Q-function with regard to that implicit reward model. This paper extends this relationship to SFT, and discovered that KL term is irrelevant to the policy in its mathematical form. \n\nA learning rate adjustment scheme is proposed and the KL div is extended to tother f-divs.", "questions": "- L46, do you mean Figure 4 or Figure 1? Elsewhere, Figure 1 is never mentioned. Thus I wonder how it is made? \n- Figure 2 a, consider flipping y axis and make image texts larger?", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength:\n\n- This paper offers deep theoretical insights, filling an important gap in previous analysis in RLHF. \n- The insight gleaned from theoretical analysis is converted into a learning rate tuning insight which turns out well empirically. The f-div SFT experiments showed weaker results. \n- Various observational experiments show convincing results on the theoretical insights.\n\nWeakness:\n\n- The multi-objective results in Appendix D provides empirical results that joint optimization fails, revealing a practical inconsistency that undermines the central theoretical claim. It should be considered that more efforts be spent on looking at this.", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your recognition and encouragement of our work. We are deeply touched. Regarding your two questions:\n\n## Typo in L46\n\nThere is indeed a typo on line 46. It should be Figure 1. We made an error when referencing the figure, which caused confusion in understanding. We sincerely apologize.\n## About Figure suggestion\n\nRegarding 2a, you provided a very sincere suggestion. We will correct the figure according to your recommendation in the subsequent version. We apologize that the figure is too small, causing reading inconvenience.\n\n## About the failure of joint optimization\n\nWe also feel it is very regrettable. There are many possible reasons that lead to the failure of joint optimization. We think the problem mainly lies in data quality. We will dive into this question further and try to solve this in future work.\n\nThank you again for your support and recognition of our work!"}]

Author rebuttal:

Meta-review: The authors proposed a unified framework to bridge SFT and DPO, the two training paradigms in current foundation model training. Reviewers agree the theoretical contribution is crucial to the community. There are some argumentative discussion during rebuttal, we ask the authors to extend the experiments as required by different reviewers.
Decision: Accept (poster)