Title: UFT: Unifying Supervised and Reinforcement Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reasoning

SFT

RFT
Abstract: Post-training has demonstrated its importance in enhancing the reasoning capabilities of large language models (LLMs). The primary post-training methods can be categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). SFT is efficient and well-suited for small language models, but it may lead to overfitting and limit the reasoning abilities of larger models. In contrast, RFT generally yields better generalization but depends heavily on the strength of the base model. To address the limitations of SFT and RFT, we propose Unified Fine-Tuning (UFT), a novel post-training paradigm that unifies SFT and RFT into a single, integrated process. UFT enables the model to effectively explore solutions while incorporating informative supervision signals, bridging the gap between memorizing and thinking underlying existing methods. Notably, UFT outperforms both SFT and RFT in general, regardless of model sizes. Furthermore, we theoretically prove that UFT breaks RFT's inherent exponential sample complexity bottleneck, showing for the first time that unified training can exponentially accelerate convergence on long-horizon reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Unified Fine-Tuning (UFT), a novel post-training paradigm that combines traditional supervised fine-tuning (SFT) with reinforcement fine-tuning (RFT) in a single integrated process. The authors argue that SFT (learning from correct solutions) can lead to memorization/overfitting in large models, while RFT (learning via trial-and-error reward feedback) improves generalization but struggles when the base model is weak or rewards are sparse. UFT addresses these issues by blending supervised signals with reinforcement learning: during training, the model is guided with partial solutions (“hints”) to explore correct reasoning paths more frequently, and a hybrid objective optimizes both the reward of final answers and the log-likelihood of these hints. As training progresses, the hint length is gradually annealed to zero, smoothly transitioning from a fully supervised regime to pure reinforcement learning. The paper provides a theoretical analysis showing that UFT can break the exponential sample complexity bottleneck of standard RFT, achieving polynomial sample complexity in the length of the reasoning horizon. Empirically, UFT is evaluated on various reasoning-intensive tasks (arithmetic puzzles, math word problems, logic puzzles) across multiple model scales (0.5B to 3B parameters). The results demonstrate that UFT consistently outperforms both SFT and RFT alone, effectively obtaining the best of both: for smaller models UFT performs on par with SFT (which was superior to RFT in that regime), and for larger models UFT matches or exceeds RFT’s performance (which was superior to SFT for stronger models). Overall, the paper introduces a unified training framework that improves the reasoning capabilities of LLMs by combining the strengths of instruction-based supervised learning and reward-driven reinforcement learning in a principled way.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n\nStrength: The paper’s technical quality is generally high. It proposes a clear training algorithm that integrates supervised and reinforcement signals, and backs it up with both theoretical guarantees and extensive experiments. The theoretical contribution is notable: the authors prove that in a long-horizon reasoning setting, pure RFT requires an exponential number of explorations in the worst case, whereas the UFT approach achieves a polynomial sample complexity – an exponential improvement in efficiency. This analysis (with formal theorems and proofs in the appendix) strengthens confidence that UFT’s performance gains are not just empirical luck but rooted in principle. The experimental evaluation is thorough: models of two different families (Qwen2.5 and Llama 3.2) and multiple sizes (0.5B, 1.5B, 3B) are fine-tuned on three distinct tasks. Results are consistently in favor of UFT, and the paper includes informative figures (e.g. training curves, accuracy vs. steps) illustrating how UFT avoids the pitfalls of each separate method (e.g. overcoming RFT’s stagnation on small models and SFT’s overfitting on large models). \n\nWeaknesses: While the experiments support the claims, the scope of tasks and model scales is somewhat limited. The tasks (math puzzles, logic riddles) are all domains with a known ground-truth answer and a well-defined reward signal (answer correctness). It remains unclear how UFT would perform on more open-ended tasks or typical language model alignment settings (e.g. dialogue with human preference rewards), where “instruction tuning” usually refers to aligning with human instructions rather than solving puzzles. Additionally, the largest models tested are 3B parameters, which are relatively small by modern LLM standards – it would strengthen the paper to see if UFT’s benefits hold for truly large models (e.g. 7B, 13B or beyond) and on more diverse, real-world benchmarks. Another quality concern is that the method’s practicality relies on having high-quality solution trajectories (hints) for supervision; the authors used human-annotated step-by-step solutions from the dataset, but in general, obtaining or generating such intermediate solutions might be non-trivial for arbitrary tasks. The paper would benefit from a discussion or experiment on how noise or inaccuracies in the provided hints affect performance. Despite these points, the overall execution is solid – the algorithms are described in detail, and the claims are well-supported by evidence.\n\nStrengths: The paper is clearly written and structured. The motivation is well-established with a relatable analogy (“learning by memorizing solutions vs. thinking through problems”), and background on SFT vs RFT is provided to orient the reader. The method is explained with both a conceptual figure and a concrete example of the UFT prompt format (including how the hint is inserted into the prompt), which makes the procedure easy to follow. Key components such as the hint-length annealing schedule and the combined loss function are described in a logical order. The inclusion of a “theoretical justification” section and an explicit list of contributions also aids clarity. \n\nWeaknesses: A minor clarity issue is that Algorithm 1, which outlines the UFT procedure, is not discussed in much detail in the main text. A brief high-level explanation of its steps—such as how hints are sampled and how loss terms are scheduled—would help readers.\n\nSignificance: \n\n Strengths: This work tackles a timely and important problem in the fine-tuning of LLMs. As models grow and tasks demand reasoning, the limitations of purely supervised or purely RL-based fine-tuning have become apparent. A method that successfully combines the strengths of both has significant practical implications – it could lead to more data-efficient and capability-enhanced model training pipelines. The empirical gains demonstrated by UFT (outperforming SFT and RFT across tasks and scales) suggest that the approach can improve model reasoning without requiring larger models or prohibitively many RL training samples. The fact that UFT adapts to model capability (favoring memorization for weaker models and exploration for stronger ones) is particularly compelling – it hints that this single paradigm could simplify the fine-tuning strategy for a wide range of model sizes, rather than needing very different approaches for small vs. large LMs. The theoretical result is also significant in that it formally confirms an intuition about RL in long-horizon tasks and shows that providing intermediate feedback (hints) can dramatically accelerate learning.\n\nWeakness: \nThe impact of UFT is somewhat limited by its reliance on curated intermediate solutions, which are more feasible for structured tasks like math but less available in broader domains like QA or dialogue. The chosen benchmarks, while suitable, are relatively narrow; showing gains on more widely-used or complex tasks would strengthen the significance. Still, the approach offers meaningful insights for improving LLM fine-tuning, with broader impact depending on future extensions.\n\nOriginality: \n\nStrengths: The idea of unifying supervised and reinforcement fine-tuning is novel in the context of LLM training. Prior work has typically run SFT and RL sequentially (e.g. fine-tune on instructions, then apply RLHF), or chosen one based on context, but merging them into one framework with a seamless transition is a fresh approach. The use of partial solutions as hints effectively bridges the gap between “imitating correct reasoning” and “exploring new reasoning”, which is an original way to tackle the trade-off. The paper also introduces a smooth hint annealing technique (using a binomially sampled hint length that decays via cosine schedule) to avoid distributional shock – this is a thoughtful improvement over earlier “stagewise” or uniform sampling strategies (citing Xi et al., 2024) and appears to stabilize training. The theoretical analysis provided is another aspect of originality: it’s the first time (to my knowledge) that a unified training regime is proven to improve sample complexity for reasoning tasks, giving a new theoretical insight into why combining methods is beneficial. \n\nWeaknesses: Some elements of UFT build on existing ideas, such as guiding RL with hints or mixing supervised and reinforcement learning, which makes the contribution more evolutionary than revolutionary. While the approach doesn’t introduce a fundamentally new paradigm, it offers a well-integrated and effective combination of known techniques. The novelty lies more in execution and analysis than in algorithmic invention, but the work remains sufficiently original for NeurIPS.\n\nQUESTIONS:\n1. How would UFT apply to open-ended tasks like dialogue or preference alignment, where step-by-step solutions aren’t available? Could model-generated hints or proxy signals be used? Clarifying this would help assess UFT’s broader applicability.\n\n2. How critical are the hybrid loss and cosine hint annealing schedule? Would performance drop if we removed the supervised loss on hints or used a simpler schedule? Any ablation results would help clarify the importance of these components.\n\n3. Can you clarify the assumptions behind Theorem 4.3? For instance, how realistic is the reward gap assumption, and do you use partial rewards in practice? Also, could UFT fail if the model becomes too dependent on hints?\n\n4. How does UFT’s training cost compare to standard SFT or RFT? Does hint usage significantly increase sequence length or training time? Did UFT reach performance targets in fewer steps, as theory predicts? Any discussion on scalability would be useful.\n\nLIMITATIONS:\nPartially discussed. UFT depends on high-quality intermediate steps, which may not exist in open-ended tasks. Broader applicability to instruction tuning or RLHF tasks is not demonstrated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UFT, a post-training method that unifies SFT and RFT into a single integrated process. The authors argue that SFT suffers from overfitting while RFT depends heavily on base model strength, and UFT addresses both limitations by using partial solutions as \"hints\" to guide exploration while incorporating supervision signals. They provide theoretical analysis claiming exponential sample complexity improvements over standard RFT and empirical validation on reasoning tasks using various language models.\n\nSTRENGTHS AND WEAKNESSES:\npros\n\n- The approach tries to combine SFT and RFT in a principled manner, which is an interesting topic.\n- The paper provides clear motivation.\n- The paper is well written.\n\ncons\n\n- I'm not fully convinced by the theoretical analysis. See the questions below.\n\nQUESTIONS:\n- Hint length is sampled from a Binomial distribution. What is the rationale for this specific distributional choice? How would performance be affected by using a simpler scheme?\n- Figure 4 (right) indicates that hints are phased out entirely after 300 steps. Does this imply the hint mechanism's primary role is as a warm-up, with the final performance advantage stemming solely from the modified objective function?\n- The objective in Eq3.2 includes the term $KL(\\pi^\\ast\\|\\|\\pi)$, which you estimate via a single monte carlo trace using the annotated trajectory $(s_h^\\ast, a_h^\\ast)$. This seems to be an exceptionally high-variance estimator. How does this not introduce significant instability during training?\n- Could you provide a detailed ablation study that isolates the performance contribution of the $\\sum KL(\\pi^\\ast\\|\\|\\pi)$ term? Specifically, what is the performance of UFT if this term is removed, leaving only the GRPO objective and the standard log-likelihood on the hint?\n- How do you justify the direct application of theoretical results derived from a deterministic, uniform tree-search model to the stochastic and unstructured nature of LLM text generation?\n- The informal statement of Theorem 4.3 requires $\\beta$ be \"small enough.\" Could you formalize this condition and comment on whether the empirically optimal value of $\\beta$ satisfies this theoretical constraint?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Unified Fine-Tuning (UFT), a novel post-training paradigm that integrates supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT) into a single process. The authors address the fundamental trade-off in LLM post-training: SFT excels at knowledge acquisition but suffers from overfitting, while RFT provides better generalization but struggles with sparse rewards and depends heavily on base model capabilities. UFT resolves this by using gradually shortened \"hints\" and a hybrid objective function that combines reinforcement learning with log-likelihood maximization on these hints.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n1. The paper addresses a genuine limitation in current fine-tuning approaches. The observation that SFT works well for small models but may lead to overfitting for larger ones, while RFT generalizes better but depends heavily on base model capabilities, is well-supported and practically relevant.\n2. The core idea is intuitive. The cosine annealing schedule for hint length provides a smooth transition from supervised to reinforcement learning, avoiding the distribution shifts that plague staged approaches.\n3. The evaluation across multiple model sizes (0.5B-3B), architectures (Qwen2.5, Llama-3.2), and tasks (Countdown, MATH, Logic puzzles) demonstrates the generality of the approach.\n\n## Weakness\n1.  The paper doesn't discuss the computational cost of UFT relative to baselines. The hybrid objective and hint processing likely increase training time.\n2. The method introduces additional hyperparameters (hint length schedule, β weighting). While the paper shows UFT is effective, more analysis of sensitivity to these choices would help practitioners.\n3. It would strengthen the paper to include other domains like code generation, scientific questions, or other reasoning domains to demonstrate broader applicability.\n\nQUESTIONS:\n1. While UFT demonstrates superior performance on most benchmarks, SFT-RFT occasionally outperforms UFT on certain tasks (e.g., Countdown and Logic), and Table 2 reveals that UFT doesn't universally dominate across all evaluation metrics. This raises critical questions about when practitioners should choose SFT-RFT versus UFT. What task characteristics, model properties, or dataset features determine the optimal choice between these methods?\n2. The extensive mathematical formulation in lines 88-101 introduces numerous tree-based definitions and notations that appear underutilized in subsequent theoretical development. Notably, the core objective functions in Equations 3.1 and 3.2 seem largely independent of the elaborate tree structure definitions.\n3.  The paper lacks clear guidance on how to strategically allocate training data between SFT and RFT phases. Should practitioners prioritize difficult examples for the SFT stage to provide strong foundational knowledge, or should simpler examples be used for supervised learning while reserving complex cases for reinforcement learning?\n4. A comprehensive analysis comparing computational overhead between UFT, pure SFT, and pure RFT is notably absent.\n5. The decision to begin zero-hint training at step 300 in Figure 4 appears empirically motivated rather than theoretically grounded. How can the optimal timing for beginning zero-hint training be determined across different model sizes, task complexities, and dataset characteristics?\n6. Several theoretical claims lack sufficient justification, most notably the assertion on line 185 that a particular estimator is unbiased without providing formal proof. While the authors present extensive theoretical analysis in the appendix, key intermediate results like the optimal training step count remain underdeveloped.\n7. The evaluation is primarily concentrated on mathematical and logical reasoning tasks using relatively small language models (≤3B parameters). This narrow scope limits the generalizability of findings to broader reasoning domains, or larger model scales.\n8. Algorithm 2's Q-value computation (line 7) appears oversimplified. Additionally, unclear definitions of key metrics (e.g., \"Average Score\" in Figure 4) and missing comparative analyses (e.g., standard RFT performance versus RFT with cosine annealing in Figure 5) limit the comprehensiveness of the experimental evaluation.\n\nLIMITATIONS:\nYes. However, there are only one limitation about human-annotated solutions. I suggest the author could add more limitation analysis and training cost of UFT.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper brings together supervised and reinforcement fine tuning into a single framework. The authors claim that the combination of SFT and RFT can actually break the exponential sample complexity. SFT+RFT hold promise of combining curriculum memorization with reward-driven exploration, thereby enabling thinking/reasoning and memorizing at the same time. The authors contrast it with approaches that use SFT or RFT alone, or combine them sequentially.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* smooth transition from SFT to RFT during fine tuning is intuitive, making the exploration task progressively harder and reducing the dependency on supervision gradually over time.\n* The authors consider previous work thoroughly, citing R^3\n* The paper offers intuitive motivation for unifying SFT with RFT based on human intelligence (though some may find drawing these parallels as a disadvantage)\n* theoretical proof is offered for polynomial sample complexity for UFT. \n* information-theoretic intuition/explanation is offered (sec 3.2) for why RFT's sparse reward signal (correct/incorrect) is inferior to injecting new knowledge through partial SFT hints. Thus, UFT is shown to leverage the strengths of both fine tuning approaches. \n* UFT reduces to SFT as a special case, mathematically (remark 3.1)\n\n**Weaknesses**\n* Exploration was done on non-SOTA models that are quite small Qwen2.5-0.5/1.5/3B. (only an ablation study for hint length predictors was done on LLama 3.2-1)\n* UFT may not generalize (or not offer any benefits) for larger reasoning models.\n\nQUESTIONS:\n* Does UFT unification proposed by authors generalize to larger models that have enough capacity to just train on RFT alone? \n* My main concern is lack of in-depth experimentation with a broader set of models and larger model sizes (e.g. 70B). If the authors convince me that UFT generalizes to more and larger models -- this can be convincing evidence of desired generalization (and could help +1 my score). Otherwise, UFT's contributions might be very constrained. \n* it would be good to show how UFT compares to larger SOTA models given it's better convergence and sample efficiency (i.e. within some time budget or upper limit on the number of steps). While larger SOTA models may eventually converge to better accuracy, UFT may offer competitive performance within a time or step limit.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-04"}