Title: Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
Venue: NeurIPS 2025
Accepted: yes
Keywords: Natural Language Processing

Large Language Model

Reinforcement Learning

Markov Decision Processes

Post-training
Abstract: Post-training processes are essential phases in grounding pre-trained language models to real-world tasks, with learning from demonstrations or preference signals playing a crucial role in this adaptation. We present a unified theoretical framework bridging Supervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training. Through rigorous mathematical derivation, we demonstrate that both SFT and preference learning methods like Direct Preference Optimization (DPO) operate within the same optimal policy-reward subspace, with SFT representing a special case of implicit reward learning. Our analysis reveals a critical limitation in conventional SFT: the KL divergence term in distribution matching becomes constant with respect to the policy during optimization, failing to constrain model updates. To address this, we propose a simple yet effective learning rate reduction approach that yields significant performance improvements (up to \textbf{25\%} relative gain and \textbf{6\%} absolute win rate increase in instruction following tasks. Additionally, we derive alternative SFT objectives from various f-divergence functions that preserve the KL term during optimization, further enhancing post-DPO model performance. Finally, we extend the theoretical relationship between LLM logits and Q-functions from preference learning to the SFT context, providing mathematical derivations and experimental validation.


Peer reviews:
comments: SUMMARY:
Previously, [12,17] have established the relationship between DPO and implicit reward learning and how LLM logits can function as a Q-function with regard to that implicit reward model. This paper extends this relationship to SFT, and discovered that KL term is irrelevant to the policy in its mathematical form. 

A learning rate adjustment scheme is proposed and the KL div is extended to tother f-divs.

STRENGTHS AND WEAKNESSES:
Strength:

- This paper offers deep theoretical insights, filling an important gap in previous analysis in RLHF. 
- The insight gleaned from theoretical analysis is converted into a learning rate tuning insight which turns out well empirically. The f-div SFT experiments showed weaker results. 
- Various observational experiments show convincing results on the theoretical insights.

Weakness:

- The multi-objective results in Appendix D provides empirical results that joint optimization fails, revealing a practical inconsistency that undermines the central theoretical claim. It should be considered that more efforts be spent on looking at this.

QUESTIONS:
- L46, do you mean Figure 4 or Figure 1? Elsewhere, Figure 1 is never mentioned. Thus I wonder how it is made? 
- Figure 2 a, consider flipping y axis and make image texts larger?

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a unified theoretical framework that connects Supervised Fine-Tuning and Direct Preference Optimization in the post-training of Large Language Models. The authors argue that both SFT and DPO can be viewed as learning under an implicit reward function within the same policy-reward subspace. They derive the SFT objective from f-divergence-based distribution matching and show it is a special case of implicit reward learning when total variation is used. Based on this insight, the paper proposes two practical improvements: (1) reducing the learning rate during SFT to compensate for the loss of gradient information from the KL term, and (2) using alternative f-divergence formulations that preserve KL structure. Empirical evaluations on standard instruction-following benchmarks demonstrate that these simple changes sometimes improve post-DPO performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength.\n\n- **Theoretical contribution.** The paper offers a rigorous and insightful theoretical framework that unifies SFT and DPO through the lens of implicit reward learning. This clarifies an often empirically observed but poorly understood connection.\n- **Practical utility.** The proposed improvements are easy to implement and demonstrate consistent gains across different models and datasets, making them applicable in real-world settings.\n\n---\n\n## Weaknesses.\n\n- **W1: Insufficient justification for selected learning rate**\n\nIn Section 4.2, the authors claim that using a smaller learning rate during SFT leads to better post-DPO performance. However, the choice of 5e-6 as the \"small\" learning rate seems arbitrary. A more thorough hyperparameter study would be helpful to validate this claim. Would further reducing the learning rate continue to improve results, or is there an optimal range?\n\n- **W2: Limited model evaluation**\n\nAll experiments are conducted on 7B-level models. It remains unclear whether the observed phenomenon generalizes across smaller (e.g., 0.5B, 1.5B, 3B) or larger (e.g., 13B) models. Additionally, the authors did not experiment with state-of-the-art open-source models such as the Qwen series, which may yield more robust or contrasting results.\n\n\n- **W3: Lack of discussion on Table 3**\n\n Table 3 contains several outlier values, and the proposed SFT variants show inconsistent performance across benchmarks. However, the authors provide no analysis or explanation for these inconsistencies, leaving unclear when and why the proposed methods succeed or fail.\n\n- **W4: No representation-level analysis**\n\nThe claims regarding KL divergence and its effect on alignment could be further substantiated by analyzing changes in the learned representation space. For instance, examining whether the regularization leads to more structured or aligned latent spaces would strengthen the argument.\n\n\n- **W5: Limited generalizability beyond DPO**\n\nThe paper does not provide sufficient analysis on the sensitivity of key hyperparameters, such as the clipping bounds in pre-clip, the threshold in advantage filtering, number of training data samples, or the weights in the final reward aggregation. Without such studies, it is difficult to assess the robustness and reproducibility of the proposed StableReinforce algorithm.\n\n- **W6: Presentation issue**\n\nThe reference to a figure on line 45 appears incorrect.\n\nQUESTIONS:\nQ1: Why do the benefits of different SFT variants only emerge after post-alignment training? The paper observes that certain SFT configurations perform better after DPO but not immediately after SFT. Are there theoretical or empirical insights into why this discrepancy exists? Why would SFT effectiveness not manifest directly in post-SFT performance?\n\nQ2: Line 178–180: The authors argue that omitting the KL term leads to a “substantial distance” between the policy and reference model, implying this is undesirable. However, prior works have shown that SFT alone is effective and a larger distance may offer greater expressivity. Why is substantial distance necessarily a negative factor in this case?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to connect SFT and preference learning through the idea of the implicit reward and describes SFT as distribution matching using the TV distance without a KL term. Through this, they describe SFT as searching the optimal policy-reward subspace and suggest using smaller learning rates for SFT. They also provide alternatives to standard SFT derived from different f-divergences. They claim that the logits after SFT correspond to state quality. Experiments are performed to verify these claims.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Background on distribution matching and the problem setting was clearly presented with definitions and explanations. \n- The paper provides a novel analysis that provides an understanding of the connection between SFT and DPO which is a common part of the language model training pipeline. \n- The paper overall clearly states results and assumptions with empirical verification of results. \n\nWeaknesses:\n- There are some steps that need further justification. Mainly Assumption 1 which makes a rather strong assumption about the value function and the claim about the KL term in the derived SFT being treated as a constant with little effect. It seems unclear how true or practical these claims are and seem to be an important part of being able to draw the connection using the implicit reward. This is a large weakness especially since the claim about SFT searching the optimal policy-reward completely depends on this assumption and following results build on this. Justification for replacing entropy with KL and why the cross-entropy term is small is currently weak. \n- Using the identity function as the conjugate function for TV only holds over a small range so if pi starts to deviate more from the initial model, this connection is no longer valid and also further suggests that the KL term in SFT could have a non-trivial effect. \n- It seems in the comparisons with other objectives from f-divergences, SFT is used without the KL term which seems like an unfair comparison. Also SFT with the KL term is not used in any of the experiments.\n\nQUESTIONS:\n- Can you provide some justification for Assumption 1? as well as for why you expect the cross-entropy between the policy and the reference to be small? Output probabilities do change significantly over the course of SFT especially if introducing a chat format. \n- Can you provide some clarification on eq. 4? What is $s_t$? and what is meant by $V^*$ is the value function of the optimal policy?\n- Could you provide experimental results for SFT with the KL term? \n\nJustification on the assumptions and steps taken are the current main concern and better explanations or experiments demonstrating that these steps are reasonable would increase the score.\n\nLIMITATIONS:\nThe assumptions necessary may be a limitation that needs to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCan we understand supervised fine-tuning (SFT) and direct preference optimization (DPO) as part of a unified reward-based framework? The paper present argues that SFT is in fact optimizing an implicit reward function, just like DPO does explicitly. This reframes both SFT and DPO as reward-based methods under a shared mathematical umbrella. The authors show that SFT corresponds to inverse reinforcement learning (IRL) by matching expert distributions. They point out a limitation of SFT, where the KL term in distribution matching becomes constant during training—making SFT unable to constrain model updates effectively. To address this, they advocate using alternative divergences (e.g., Pearson χ²) to recover stable reward shaping and show that this leads to better downstream DPO performance. They also generalize prior work connecting LLM logits and Q-functions in preference learning, extending it to SFT as well. Experiments with LLaMA-3 and Mistral on UltraChat/UltraFeedback validate their theory.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n* The paper theoretically unifies two central components of LLM alignment — SFT and DPO — under a shared framework based on implicit rewards and distribution matching.\n* It clearly identifies a flaw in conventional SFT training: the constant KL term fails to regulate model updates and can harm downstream performance. This is a non-trivial insight with actionable implications. \n* It goes beyond theory and proposes two concrete techniques backed by experiments: (1) reducing the SFT learning rate, and (2) switching to divergences (χ², Squared Hellinger) that preserve gradient signal.\n* Empirical results validate theory across multiple models (Mistral, LLaMA-3) and evaluation sets (AlpacaEval, Arena-Hard, MT-Bench), showing large relative win rate improvement post-DPO.\n* The extension of the Q-value/logit interpretation to SFT broadens its usefulness for reward estimation and could inspire further work on early-stage reward shaping diagnostics.\n\n# Weaknesses\n\n* The experimental validation is focused on a narrow domain: instruction tuning with UltraChat and UltraFeedback. It's unclear if the same effects hold in tasks with fundamentally different reward structures (e.g., code generation, tool use, multi-turn dialog).\n* The Q-function interpretation of logits in SFT is supported mostly by correlation experiments rather than rigorous causal or ablation-based testing. No direct test of how useful this proxy is in practice.\n\nQUESTIONS:\n# Questions\n\n1. Could the unified framework potentially support more complex multi-turn feedback protocols beyond binary preferences (e.g., fine-grained ratings, trajectory-level reward models)?\n2. Is there a formal characterization of when the logits-to-Q-function approximation in SFT breaks down? Under what training regimes or tasks does this interpretation become unreliable?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reframes the relationship between SFT and DPO, arguing they both exist in an \"optimal policy-reward subspace\" where an implicit reward is being learned. The core of the work is deriving the SFT objective from an f-divergence minimization problem, showing that SFT corresponds to using the total variation distance. This viewpoint highlights that SFT lacks the explicit KL-regularization present in DPO and other RL algorithms. The authors propose that this missing regularization can be compensated for by using a small learning rate or by adopting alternative f-divergence objectives for SFT. Experiments show these modifications to the SFT stage improve the final performance of a model after it undergoes DPO.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper does a good job of building a conceptual and mathematical bridge between SFT and DPO, which are often treated as separate, sequential steps. The \"implicit reward\" framing is interesting.\n- The analysis pinpoints the theoretical weakness in the standard SFT objective of lacking a KL constraint, which explains some of the known drawbacks of SFT.\n- The improvements from reducing the learning rate are clearly demonstrated. The finding that alternative f-divergences can also improve post-DPO performance (Table 3) is a compelling piece of evidence for the overall theory.\n- The paper is well-written and well-structured. Figure 1 provides an excellent schematic for the core idea, and the mathematical arguments are sound.\n\nWeaknesses:\n- The extension of the logits-as-Q-function theory to SFT (Theorem 2) relies on Assumption 2. This assumption is quite strong and is not directly validated. A more thorough discussion of when this assumption might hold or fail would strengthen the claim.\n- A fascinating result in Table 3 is that the Pearson and SH-SFT models, which perform worse than the standard SFT model on their own, produce better post-DPO models. This strongly supports the paper's thesis about regularization but is a non-obvious finding. The paper could benefit from a slightly more detailed discussion of this implication: that the best SFT model (by imitation metrics) may not be the best foundation for preference tuning.\n- The main practical takeaway is to use a smaller learning rate for SFT. While the paper provides a new theoretical justification for this, using smaller learning rates for more stable fine-tuning is a common heuristic. The novelty is therefore more in the \"why\" than the \"what,\" and the practical impact could be seen as an incremental improvement on existing best practices.\n- The framework provides a unified perspective but does not result in a unified algorithm. The SFT->DPO pipeline remains sequential. The paper shows that doing SFT \"better\" helps DPO, but the two objectives are still optimized separately. The failed multi-objective experiments underscore that a true unification is still an open problem.\n\nQUESTIONS:\n1. The paper frames DPO as navigating along a \"chosen-rejected\" direction and SFT as moving along an \"average direction\" (Figure 1). Could you formalize what this \"average direction\" means in the context of your implicit reward framework? Is it the gradient of the expected reward over the expert demonstration distribution?\n2. How does your framework relate to other methods that modify the SFT objective, such as ORPO, which combines the SFT and preference-learning objectives into a single loss function? Does your theory offer any insight into why such approaches might work?\n3. You show that better regularization during SFT leads to a better starting point for DPO. Does this imply that the reference model used in DPO should ideally be the final, regularized SFT model rather than the original pre-trained base model?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}