UFT: Unifying Supervised and Reinforcement Fine-Tuning
Authors: Mingyang Liu
Gabriele Farina
Asuman E. Ozdaglar
Venue: NeurIPS 2025 poster ()
Keywords: Reasoning
SFT
RFT
Primary area: deep_learning
TL;DR: Unify supervised & reinforcement fine-tuning, and outperforms both of them. Together with theoretical justifications.

Abstract: Post-training has demonstrated its importance in enhancing the reasoning capabilities of large language models (LLMs). The primary post-training methods can be categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). SFT is efficient and well-suited for small language models, but it may lead to overfitting and limit the reasoning abilities of larger models. In contrast, RFT generally yields better generalization but depends heavily on the strength of the base model. To address the limitations of SFT and RFT, we propose Unified Fine-Tuning (UFT), a novel post-training paradigm that unifies SFT and RFT into a single, integrated process. UFT enables the model to effectively explore solutions while incorporating informative supervision signals, bridging the gap between memorizing and thinking underlying existing methods. Notably, UFT outperforms both SFT and RFT in general, regardless of model sizes. Furthermore, we theoretically prove that UFT breaks RFT's inherent exponential sample complexity bottleneck, showing for the first time that unified training can exponentially accelerate convergence on long-horizon reasoning tasks.

UFT: Unifying Supervised and Reinforcement Fine-Tuning
Mingyang Liu, Gabriele Farina Asuman Ozdaglar LIDS, EECS
Massachusetts Institute of Technology
Cambridge, MA 02139, USA
{liumy19, gfarina, asuman edu
Abstract
Post-training has demonstrated its importance in enhancing the reasoning capabili- ties of large language models (LLMs). The primary post-training methods can be categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). SFT is efficient and well-suited for small language models, but may lead to over- fitting and limit the reasoning abilities of larger models. In contrast, RFT generally yields better generalization but depends heavily on the strength of the base model. To address the limitations of SFT and RFT, we propose Unified Fine- Tuning (UFT), novel post-training paradigm that unifies SFT and RFT into a single, integrated process. UFT enables the model to effectively explore solutions while incorpo- rating informative supervision signals, bridging the gap between memorizing and thinking underlying existing methods. Notably, UFT outperforms both SFT and RFT in general, regardless model sizes. Furthermore, we theoretically prove that UFT breaks RFT's inherent exponential sample complexity bottleneck, showing for the first time that unified training can exponentially accelerate convergence on long-hod long-horizon reasoning tasks.
Introduction
When humans learn a new subject, we typically practice with problem sets (thinking) and try to understand the solutions when encounter difficulties (memorizing). There are also counterparts in fine-tuning LLMs, which is
Supervised Fine- TaninTunig (SFT). Memorizing the collected reasoning trace (solution) by maxi- mizing the log-likelihood of it. Reinforcement Fine- Tine-Tunigg (RFT). Exploring the reasoning space of L.M and improving the performance according to the signal from a verifier the final answer (thinking).
However, unlike humans, learning and thinking are disentangled during the training of language
models. Specifically, prior work [DeepSeek-AL al. 2025 Zhou al. 2023
Muennighoff et al..
2025 Liu et al., 2025 Zeng et al. [025 typically applies either SFT or RFT throughout the fine-
tuning phase, applies RFT only after SFT completes (cf. Figure D]. The choice of the proper fine-tuning algorithm depends on the LLM's capacity and the task's complexity. Specifically, when the LLM weak, SFT typically works better since the LLM cannot explore the correct answer during reinforcement learning [Pan al.) 2025), due to the sparse reward caused by the verifier-based reward model. On the other hand, when the L.MM strong, RFT generalizes better [ie et al. 2025
Chu
al. 2025).
'The source code is available at https//gitnb.ccm//uim/0010007700
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

SFT
Differer
RFT
ASDoRaIION Usge
Question: How to get 24 with (3,377,33)
Question:
Solution:
x 13
74 65
24 with
(3,57,133)
Steps
SFT-RFT
Question:
Solution:
Question: How
Solution:
13 =
03,,2,133?
72 + 3 24
(3,57,133)
UFT (ours)
Question: How to with (3,5,7,13
Solurom:
Question:
Solution: 5 13 = 65
Question: Hew 035,7,13
Solution: x 13 es
Question:
Solution:
5 13 65 72 24
How with (3577,13)?
How to (3,577133)
72
Figure 1: (top left, top right, middle, bottom). The illustration of SFT, RFT, SFT-RFT, and UFT. respectively. SFT-RFT refers to applying RFT after an initial SFT stage [DeepSeek-AL al. 2025 Zeng et al. 2025]. (Top, center). shows the annotation usage of different algorithms over training. Curves are slightly shifted for better visibility.
To get the best of both worlds, we propose Unified Fine-Tuning (UFT), which unifies SFT and RFT and enriches the reinforcement learning signal with supervised feedback. enabling the model to acquire new knowledge during fine-tuning more efficiently. In Figure [ SFT-RFT refers to the common practice of initiating reinforcement learning from supervised fine-tuned model, widely adopted the literature [DeepSeek-A) Al et al., 2025 Zeng et al. 2025). As shown Figure T (top left), SFT uses full annotations (solutions) throughout training, whereas RFT does not any annotations at all (Figure D top right). Similarly, SFT-RFT begins with SFT using full annotations, but once the RFT phase starts, it discards all annotations and relies entirely exploration. In contrast, method, UFT, offers a smooth transition from SFT to RFT, preserving the annotation signal early on and gradually reducing it the model becomes capable of self-guided reasoning.
The most relevant work to UFT is Learning Reasoning through Reverse Curriculum Reinforcement Learning (R²) [Xi et al., 2024], which proposes curriculum learning method that concatenates the problem with a slice of the solution (hint, cf. Figure left). While R treats hints primarily exploration aids, UFT further integrates them part of the supervision signal. This unification enables reinforcement learning not just to search, but to learn from existing solutions, effectively raising the performance ceiling imposed by the model's pretraining capacity (cf. Figure detailed comparison with related work is postponed to Appendix[A)
Figure 2 shows the accuracy of different algorithms over time, while the training set is Countdown Wiikipedia contributors 2025 Pan et al.] [025], MATH(3,4.5) (levels 3-5 only) [Hendrycks et al.
2021 Zeng et al. 20255] and the Knights and Knaves logic puzzle (Logic) [Xic et al. 2025 Basc

---

Average Accuracy Qwen2.5-0.5/11553B
Countdown
MATH(3.4.5)
Logic
0.4
Accuracy
0.4
0.3
0.2-
0.1
0.0
2500
600
230
200
Steps
Steps
Steps
Figure 2: Presentation for different algorithms' accuracy when trained on Countdown
Wikipedia
contributors, 2025], MATH(3,4.5) (level 3-5 only) [Hendrycks et al. 2021 Zeng al. 2025], and the Knights and Knaves logic puzzle (Logic) [Xie et al. 2025]. Accuracy is averaged over Qwen2.5 models of sizes 0.5B, 1.5B, and 3B [Qwen 2025]. Base refers the model without fine-tuning, and R² is the curriculum reinforcement learning baseline [Xi et al., 2024). The figure shows that UFT outperforms both SFT and RFT, while the relative performance of SFT and RFT varies depending on task complexity.
refers to the model before fine-tuning, and Rt represents the curriculum reinforcement learning baseline [Xi et al. 2024]. shown the figure, UFT generally outperforms all other algorithms. Furthermore, we provide the evaluation on various benchmarks, and the results are shown in Table B
Moreover, theoretically prove that RFT [DeepSeek-Al et al. 2025 Zeng al., 2025 Liu al. 2025] suffers from an inherent sample complexity bottleneck, which is exponential in the length of the reasoning. In contrast, the unified training paradigm in UFT can improve the sample complexity to polynomial dependence on the reasoning length, which is an exponential improvement over RFT.
1.1 Contribution
We state the contribution of this paper in the following.
Integration of Supervision and Reward Signal. UFT provides a general framework that integrates the supervision from SFT and reward from RFT into a single training paradigm. UFT blends reward optimization with log likelihooo maximization on hints (partial solution), and smoothly transitions from fully supervised to fully reinforcement learning. Such inte- gration allows models explore and learn simultaneously, addressing the trade-off between memorization (SFT) and generalization (RFT) in principled way.
2. Theoretical Justification. We provide a theoretical analysis of UFT, proving it achieves polynomial sample complexity dependence on reasoning length, compared to the exponential complexity required by standard RFT. This result formally establishes the efficiency gains from unifying learning (cf. Section
3. Empirical Validation Across Model Scales and Tasks. We evaluate the algorithms by training Qwen2.5-0.5/ 5...55.05..///33 [Qwen al. 2025 and Llama3 2- 1/3B (Grattation al.) 2024
on Countdown /Wikipedia contributors/ 2025 Pan et al.] 2025), MATH [Hendrycks et al.
2021], and the Knights and Knaves logic puzzle (Logic) [Xie al. 2025 UFT consistently outperforms previous methods, showing robustness across domains and models (cf. Section
2 Preliminaries
Notation. For any integer n > 0, let [n] = 21, .n} and === [0, Σn x, 1}
be the n 1-dimenional probability simplex. For any two distribution x, y E A", let KL (x+ yy ==
Ex I; log denote the KL-divergence between and y. For any discrete set S, let [ be its cardinality.

---

3,5,7. 7, 18.22
5 x 13
3
7- =
7+ 65 72
8-7=1
8 13 21
4+ 65
69
66
4 = 61 72+3= = 24 72+3 216 13 1 = 14 13+ = 13 21 + 7= 28 21 7
Figure 3: An illustration the Countdown game, where the goal is to obtain 24 by applying basic arithmetic operations (+, -, X, +) the numbers (3, 5, 13). The green path represents the correct solution.
Search Tree. The problem-solving process can be represented as a search tree, as illustrated in Figure Except for the leaf nodes, each node (also referred state interchangeably) in the search tree has B children, where B is the branching factor. Each child represents a different next token (or next sentence) be generated, node corresponds to a complete solution to the problem. The tree has height of H, with the root at height and each node's height equal to its parent's ' height plus one.
state
the terms node and
path from the
a leaf
Let S₁ denote the set of nodes with height h E 1, H) and S = - SR. Note that [Sp]
since it only contains the root Sroot. and + B-[Sk]. Therefore, there are Σ Bh B nodes in total. Once reaching a leaf node E SF,, the model will receive reward R(s) [0. 1]. policy can be written th child of For any state-action pairs (s,a) E S [B], let T (s,a) S be the child at the branch a of state 8, and T (s) = for 8 E Sh. The value function of policy ा is written as the trajectory starting from and sampled
B-1
U Sp. A where xaa 8) is the probability of selecting the
V=: S + [0, 1]. We write Sho s, (sn) according to æ. i.e., On (((1sh), sh T(sh, ah). For any hg (0, 1.... .H) and Skn. [R (sH)]. which the expected reward obtained by following define (s) = E,, B (o.n)) policy æ starting from node 8.
Let x- argmax, V (Smoot) denote optimal (deterministic) policy that achieves the highest expected reward] Let V* V=a (Sroot) be the expected reward of the optimal policy x Since n is deterministic, let (s aū si, si, ... ,HH) represent the path from the root leaf node by following n², where sú Stoot.
3 Unified Fine- Tane-Tnnig (UFT)
In this section, we introduce the two key features of UFT: (i) an exploration mechanism guided by hint, which improves sample efficiency by mitigating the sparse reward problem common in rule-based reinforcement learning [DeepSeek-Al et al. 2025): and (ii) hybrid training objective that combines reinforcement learning with a log-likeldhood term on hints, which provides a more informative learning signal and enables the model to acquire knowledge more effectively during fine-tuning.
3.1 Exploration with Hint
Although RFT is beneficial for training large models [DeepSeek-Al et al. 2025 several recent studies [Pan et al., 2025) report that small models often fail to reason effectively, as they may never explore the correct answer once due to the sparse reward. Additionally, other work has found that RFT's final performance constrained by base models' capabilities [aandh et al. 2025).
'here exists least deterministic optimal policy, and choose such policy as x.

---

To address the sparse reward issue, UFT guides exploration using a hint, that is, trajectory sampling starts from the concatenation of the problem description and hint, which is a partial solution the problem (cf. Figure 4. In this way, models will explore the correct answer more frequently.
RFT can be modeled as the task of finding a path from the root of the problem-solving tree leaf node that represents the correct answer. As shown in Figure 3] RFT needs to identify the green path. However, the problem-solving tree for real-world tasks, such as math problems, typically contains an enormous number of nodes, making it difficult for an LLM to discover the correct path through exploration alone. To make matters worse, under the rule-based reward model proposed in DeepSeek-A Al et al. [025], only small fraction of the leaf nodes correspond to correct answers, resulting in the well-known sparse reward problem [Ladosz et al., [022].
We address this challenge by concatenating the problem with a partial solution, referred to as the hint, to guide the model towards the correct answer. Figure 4 (left) provides example of UFT's prompt.
LFFT Prompi
A converaaiinn in ferween Uhe and Aocita saae. The oser eiks pxoppptioon and Au
Average Score on Training Batch (Qwen2.550.55)
wikoos it The ascituee firs thikk afoer the renuneng proces the ried and than
1.0
UFT
tobyy dhe med tuin mary piges as pounsbly If dwe wores to mod lilf of the meraring pages renorme, your rurs papes shaakl dhe isad" Skiw your sook
User
book Yearedyy
ollo rea
Stage
0.2
cumple gasingi slaanarr
Average Score
Assinan Let tne valve tho stae by stap.
0.6
Setation
0.4
Hial
Maiie road 2 in porliig--p pe0002 puga loday. the abli reads
we12 24 20 pages dioce yarrrday
0.2
540 4400moo0440400ioo pages.
Siave sle seue eoad hail
0.0
0
1000
2000
300
400
500
Steps
Figure 4: (left). An illustration of the UFT prompt. We adopt the prompting template from TinyZero [Pan et al., 2025), which is similar to that used in Deepseck-RI [DeepSeck-AL] et al., 22025). The hint consists of slice of the full solution. During training, the question prompt and the hint are concatenated and fed to the model. (right). An illustration of the training curve of Qwen2.5-0.5B. Stage and UFT keep zero hint since step 300.
3.1.1 Hint Length Sampling
Since we are ultimately interested in the LLM's performance when the hint length is zero, the hint must be gradually shortened during training. A natural idea is to subtract the hint length by constant amount regularly, which is referred to as the staged reinforcement learning [Xi et al. 2024]. However, because solutions typically consist of no more than 10 sentences, changes in hint length can cause significant distribution shift, leading to unstable training (cf. Figure 4 right).
To avoid distribution shift during training, Xi et al. 2024) samples the hint length uniformly from all possible values throughout the training. However, relying on hints throughout training introduces significant distribution mismatch between training and evaluation. This often leads to performance collapse at test time, where no hints are available. To address this, UFT employs a smoothed reduction of hint length to zero, which (i) avoids drastic distribution shifts and (ii) better aligns the training distribution with the evaluation distribution.
Specifically, we maintain a variable p E [o. 1), representing the proportion of the solution revealed to the LLM as a hint. The value of p gradually descends during training according to cosine annealing (cf. (B.I) [Loshchloov and Hutter 2017). Let be the random variable indicating the hint length, require
and let L be the total length of the solution (e.g., number of sentences). By definition, I E {0, 1..... L. and E [ p - L,, so that the expected hint length matches the proportion p. To achieve this, we sample I Binomial( (T, from binomial distributior It is straightforward to
P = ()) = (t)p1'(1 - for any lo E (0. 1,...... L}. In other words, / the number of heads obtained when tossing L independent coins, each landing heads with probability p.
5

---

We remark that the optimal policy n² is unknown and cannot compute - KL (m-(-1 (- sn)) directly. However, thanks the annotations contained in verify that = -B In Appendix B.3 we further show that UFT is robust to the choice of hint length distribution, and we choose binomial distribution due its simplicity.
Compared to stage-wise hint length reduction, UFT provides a smoother transition from long to short hints. The training curves of these algorithms are shown in Figure | (right). We can that the training curve of UFT is smoother and converges faster than that of the staged reinforcement learning. Note that staged reinforcement learning and UFT do not use any hint since step 300.
As shown in F although RFT (cosine), which is RFT equipped with the cosine annealing hint length scheduler, outperforms R$ (uniform sampling), it is still worse than SFT-RFT. Furthermore, for Llama-3.2-18, 1B, RFT (cosine) is even worse than SFT alone. This im- plies that the model's performance hindered by its knowledge gained through pretraining [Gandhi et al. 2025), which motivates the sec- ond modification of UFT introduced in Section 3.2 an additional log- likelihood term in the objective function.
Acceracy Triined Countion with Owen25-0.5
Traised Coundoww Llans 3.3B
377
Sk.p0
Skege
Figure 5: An ablation study of different hint length schedulers. RFT (cosine) refers to reinforcement learning with our cosine annealing hint length scheduler proposed in this section.
3.2 Objective Function Modification
The hinted RFT only enables LLMs to explore the correct solution more frequently, but remains inefficient at injecting new knowledge into the LLMs. This inefficiency arises because each sampled trajectory provides limited information, essentially signal (correct/incorreet), which provides far less information than the supervision signal in SFT. In contrast, SFT enables more efficient knowledge acquisition, but suffers from poor generalization [Xie et al. [025 Zeng et al. 2025]. To get the best of both worlds, UFT introduces an additional log-likelihood term to the objective function of RFT, allowing the model to learn from the informative supervision signal and still benefit from the generalization of RFT.
B-1 For notational simplicity, let 8q = Sroot, (sh, an h=
denote the shorthand for on ~ x(: (8))
and 86+1
T(h,, an), i.e., (sh, on) a=) H-
represents a trajectory sampled according to starting at denote the objective function associated with the expected
Sroot. Formally, let Jualue ((s). an h=
H-1
reward Then, let B > 0 be the hyperparameter controlling the KL divergence, we have
JRFT
gralus (mur:n))hs H-1) -
H-1
Σ (x( (o((1en))l (.1m))) Amo
(3.1)
mo (2h an Nn
JUFT
=E
1,00 1,0022000 (e) (1), 17
goalon (W M-!) - > KL (nt ( - E KL (ar (19001((111))))
(3.2)
Compared to the objective function of GRPO. UFT adds an additional term Ei KL (-1=s=)))) (s))), the KL divergence between the optimal policy and the current policy. Compared to I Jvauee this term explicitly guides the policy towards optimality, and thus results in faster convergence rate.
'In GRPO [hao et al. 2024 we have Jralue (()) an ) = = - min { mayy ا,ي Aw, xíarr 1ew) Toto " 1 e, +c) Aw) where is the current policy, n7 old is the policy at the previous step. Au is the estimated advantage value in GRPO.
clip

---

the dataset, we have access to trajectory sampled according to πτ, i.e., (5), "9 ~ '², which can be used to estimate the KL-divergence, According to the definition of KL-divergence, minimizing
KL (a" ( (- (-((s)(())))))))) is equivalent to minimizing Σa, =1 ' (a sf log [[an] (omit terms irrelevant to x), and log =(a;, 1x
is an unbiased estimator of it, since oj π a= (-1s)) Therefore, 3.2 can be equivalently written as
5 GUFT
gralue ((s) (x)ml))
Σ ( (1Io)) + Σ log xaa (5))
(3.3)
Ninrma] (ex, 19
Therefore, the UFT objective 3.3 can be interpreted as (i) maximizing the expected reward while (ii) staying close to the reference policy and (iii) memorizing the hint by maximizing the log-likelihood of producing the hint.
Remark 3.1. The name of Unified Fine-Tuning (UFT) comes from the fact that when p = 0 for all
steps during training. (3.3) is equivalent to RFT, since 3 2 log n ((u 87, = 0. When 1, then
Jvalue (sh, (1h) ==1 3 Σズー
H-1 KL (T( [sx]]l (- (-1 (k)) 0, so that (33) degenerates to SFT.
An illustration can be found Figure (top middle).
is noteworthy that after adopting the additional log-likelihood term, UFT's performance matches that of SFT-RFT for small models (cf. Figure This suggests that UFT improves the ceiling of RFT by enabling the model to acquire new knowledge during post-training.
3.3 Algorithm Outline
Overall, the UFT algorithm proceeds as follows:
Sample a batch of problems B along with their corresponding solutions.
For each problem, sample a hint length according to Section 3.1.1
Concatenate each problem with its solution prefix of length I.
Train the model on D using reinforcement learning algorithm with the objective defined in
3.3).
The corresponding pseudocode is provided in Algorithm !
4 Theoretical Justification
In this section, provide theoretical justification for UFT. First, we show that the lower bound of RFT's sample complexity grows exponentially (O (O(B#) )) as the tree height (reasoning length) increases. Second, we show that UFT may find the solution within polynomial number of samples (O (BH log B)), representing an exponential improvement tree height in sample complexity.
Next, we define the sub-optimality gap in reward, which is the difference between the rewards for correct and incorrect solutions.
Definition 4.1 "Sub-Optimality Gap). There sub-optimality gap A between the reward of optimal and suboptimal nodes. Formally, for any leaf node 8 E Sh with reward R(s) < maxves, R(s'), we have
R(s)
S max R(s') A. e'ESH
(4.1)
In this paper, there only three possible outcomes for R(s), i.e., reward (incorrect format), format reward, and accuracy reward. Therefore, the sub-optimality gap
= (accuracy reward) (format reward) = 1.0 0.1 = 0.9.
(4.2)
Next, we will give the lower bound on the RFT's sample complexity to achieve 50% pass@1 success
'The probability of reaching the correct answer when sampling single trajectory.
7

---

Theorem 4.2 (Lowerbound). For any integers H 2 1, 2. and any RFT algorithm, there exists problem with height H and branching factor B, that satisfies the following: achieve 50% pass success rate, the algorithm needs to explore at least
D
(4.3)
4
nodes in Sy. Moreover, when there are multiple nodes in Sy representing the correct solutions, e.g.. K 1, any algorithm needs to explore at least 8" nodes Sy- 1K
The proof constructs a set of problems with different correct solutions, which cannot be distinguished before exploring sufficient nodes in Sy. The details can be found in Appendix [ Furthermore, the traditional lower bounds in reinforcement learning (Jin et al., 2018 Domiinuees al.] [202T] are built on the stochastic transitions of the Markov decision process, but the search tree's transition is deterministie, which requires a different construction.
Theorem 4.2 implies that when the reward is sparse, such as when K is a constant, learning the
optimal policy takes number of iterations exponential in the height of the tree. This also justifies why long reasoning is generally difficult (Chai et al. [2025] Chen et al.) 2025]. In the following, we will show that UFT exponentially improves the sample complexity. The full algorithm can be found in Algorithm 2
Theorem 4.3 (Informal), When 3 is small enough, Algorithm oobaains 50% pass @1 success rate when the algorithm explores
B H (log B)² O
(4.4)
A
nodes in Sh
The formal version is deferred to Appendix Note that the 50% pass 1 in both Theorem 4.2 and
Theorem 4.3 can be arbitrarily adjusted, and it only affects the sample complexity by constant factor. From Theorem 4.3 observe that the dependence on H is reduced from Bdd to HS representing an exponential improvement enabled by the use of hints. Moreover, A in the denominator implies that the difference between accuracy reward and format reward should be large for fast convergence,
which is also supported by empirical studies [Shao et al. 2024 Pan et al. 2025
Zeng et al.) 2025
5 Experiments
In this section, we present the experimental results of UFT. We demonstrate several key properties of UFT: (i) When the model is small (S 1B) and SFT outperforms RFT, UFT's performance matches that of SFT. (ii) When the model is large (~> 3B and RFT outperforms SFT due to better generalization, UFT's performance matches that of RFT (and sometimes even outperforms it, cf. Table S).
In experiments, we train Qwen2.5-0.55 Qwen2.5-1.5B Qwen2.5- 3B [Qwen et al. 2025]. Llama-3.2-1BB and L.ama-3.2-3B
Grattafiori et al. 2024
on Countdown [Wikipedia contributors,
Canndainc Arcme Exihention Secccn Bak Logk
2025 Pan et al. 2025
(3TH(3,445) (only level 3-5 included) Zeng et al. [2225] and the Knights and
Henddrycks et al. 2022
Knaves logic puzzle (Logic) Xie al. [025]]
5.1 The Memorization of UFT
As shown in Figure 7 we can see that when the model is small, the improvement from RFT is marginal, since the model rarely explores the correct answer. As shown in Figure 6 when training Qwen2.5-0.5p on Logic, RFT rarely explores the correct answer, while UFT finds it at every single timestep.
Figure 6: Owen2.5-0.5B's cu- mulative average success rate for exploring the correct answer at each step when trained on Logic.
Compared to R where hints are also applied, UFT outperforms it since UFT (i) gradually shifts the distribution toward hint length zero, and (ii) maximizes the log-likes leeelleelhood on hints to encode information about the solution in gradients. The proximity between the performance of UFT and SFT-RFT also supports the conclusion that UFT helps the model to memorize the solution when the model's initial capacity is not enough to solve it.
8

---

Accuracy Trained with Qwen2.5-0.5B
Countdown
MATH(3,4.5)
Logic
0.2
Accuracy
0.4-
0.2
0.1-
0.0
0.0
20.0
200
200
400
Steps
Steps
Steps
Figure 7: An illustration of the accuracv on the test dataset of Qwen2.5-0.5B Base is the base model without fine-tuning mne-ttning. R3 [Xi et al. 2024 trained the model with RFT and uniform distribution over all hint lengths. SFT-RFT refers to training a supervised fine-tuned model with RFT, and UFT is our algorithm.
Accuracy Trained with Qwen2.5-3B
Countdown
MATH(3,4,5)
Logic
0.6
Accuracy
0.5
0.4-
0-
0.4
0.2-
0.2
0.3
200
100
200
1001
200
Steps
Steps
Steps
UPT
Figure 8: An illustration of the accuracy on test dataset of Qwen2.5-3B. Base refers to the base model without fine-tuning.
5.2 The Generalization of UFT
As shown in Figure 8 when the model is larger and its prior knowledge gained from pertaining is enough for reasoning, UFT generalizes well as RFT. contrast, SFT and SFT-RFT are worse, since SFT leads to overfitting. These experiments show that UFT will automatically adapt to model size and enjoy the advantage of both SFT and RFT.
As shown in Figure when the model is larger and its prior knowledge gained from pretraining is sufficient for reasoning, UFT generalizes well RFT. In contrast, SFT and SFT-RFT perform worse, since SFT leads to overfitting. These experiments show that UFT automatically adapts to model size and benefits from the advantages of both SFT and RFT.
5.3 UFT Helps LLMs Learn New Knowledge
In Gandhi et al. 2025 was found that 1ama-332233B improvement through RFT is marginal compared to that of Qwen2.5-3B. This is because Llama gains less sessoning-relatee knowledge from pertaining, e.g., backtracking and subgoal setting. In Figure 9 we can see that UFT significantly improves the performance Llama-3.2. In Countdown, even Llama-3.2-11 outperforms Llama- 3.2-3B fine-tuned by RFT after the same number of steps (250 steps). This supports the claim that UFT introduces new knowledge the model, whereas RFT only helps the model utilize its existing knowledge (Yue et al. 2025

---

Average Accuracy of Llama-3.2-1/3B
Countdown
MATH(3.455
Logic
0.10
Accuracy
0.4
Accuracy
Accuracy
0.1
0.0
0.00
0.0
100
200
100
200
100
200
Stops
Steps
Stops
-- (1R(
(18)
(3B)
UPT (38)
(38, Gundhhi (2025))
Figure 9: The comparison of Llama-3, 1.2ma-3.2-111 12-1B/BB's behavior in Comtdown/MATH/LLogii when applying
RFT/UFT. Countdown, the dotted line is the accuracy Llama-3.2-33 after 250 steps RFT reported in Gandhi et al. 2025
Time per Step Actoos Models
Tire
Idees 4.
Rollnt Time per Stop Aeme Modelk
Hosinen liee por Mkep
deeat Su.
Lenon 1t
Figure 10: The computational cost per step of UFT and RFT.
5.4 Computational Cost of UFT
The computational costs of UFT and RFT are shown Figure 10 Interestingly, UFT is faster than RFT, as it begins reasoning from a partial solution (a hint) rather than starting from scratch, thereby reducing the rollout cost during training.
6 Conclusion and Limitations
This paper proposes novel fine-tuning framework, UFT, which unifies SFT and RFT. Empirically, we show that UFT outperforms both SFT and RFT in general. Specifically, by adopting UFT, small models tend memorize while large models generalize. Theoretically, achieves an exponential speed-up compared to RFT. However, throughout the paper, we use only the human-annotater solutions in the dataset and GRPO as the reinforcement learning algorithm. Moreover, we do not employ state-of-the-re (e.g.. 70B-scale) models in our experiments due to computational constraints, but provide evidence that UFT remains beneficial under these settings (see Appendix(B.2 B.4). In the future, it would be interesting to explore the incorporation of advanced SFT and RFT techniques into UFT. For instance, using long Shain-of-thought generated by large models Muennighoff et al.) 2025 Gandhi et al. [0255] for SFT, and choosing other reinforcement learning algorithms such as REINFORCE++ [Hu 2025 and DAPO YYu et al.) 2025 as the reinforcement
prove that UFT
learning algorithm for UFT.
10

---

7 Acknowledgement
The authors would like thank Jacob Andreas, Chanwoo Park, and Kaiqing Zhang for their valuable discussions. Mingyang Liu was supported by the Siebel Scholarship, and Gabriele Farina was supported part by the CCF-2443068, ONR grant N000142512296, and an AI2050 Early Career Fellowship.
References
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Yekun Chai, Haoran Sun, Huang Fang, Shuohuan Wang, Yu Sun, and Hua Wu. Ma-rlhf: Reinforce- ment learning from human feedback with macro actions. International Conference on Learning Representations (ICLR), 2025.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.0956) 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: comparative study of foundation model post-training. arXiv preprint arXiv:2501.17766 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and Aixin Liu et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic rein- forcement learning in finite mdps: Minimax lower bounds revisited. 2021.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01300 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and Arun Rao et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Annual Conference on Neural Information Processing Systems (NeurIPS), 2021.
Jian Hu. Reinforcet+: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03266 2025.
Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Annual Conference on Neural Information Processing Systems (NeurIPS), 2018
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. International Conference Machine Learning (ICML), 2002.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.

---

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, alya Sutskever, and Karl Cobbe. Let's verify step by step. International Conference on Learning Representations (ICLR), 2024.
Mingyang Liu. On solving larger games: Designing new algorithms adaptable to deep reinforcement learning. Master's thesis, Massachusetts Institute of Technology, 2025.
Mingyang Liu, Asuman E. Ozdaglar, Tiancheng Yu, and Kaiqing Zhang. The power of regularization in solving extensive-form games. International Conference Learning Representations (ICLR), 2023.
Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. A policy-gradient approach to solving imperfect-information games with iterate convergence. arXiv preprint arXiv:2408. 0075/, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang. Chao Du, and Min Lin. There may not be aha moment in rl rl-zero-like training pilot study. https://oatlnn www.iinssite/oat-zrrrro 2025. Notion Blog.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. International Conference on Learning Representations (ICLR), 2017.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprini urXiv:2406.06592. 2024.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. International Conference on Machine Learning (ICML), 2020.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint wxXiv:2501.19399 2025.
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. http:://eitubb.co//aaiiiii Pan.a/iaaaaaaaaiiiieeeoooo 2025. Accessed: 2025-01-24.
An Yang Qwen, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, and Keming Lu et al. Qwen2.5 technical report. arXiv preprint arXiv:2412 15115, 2025.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. International Conference on Learning Representations (ICLR), 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXhv:2402.03300. 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning. arXiv preprint arXiv:2504.05520 2025.
Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang. Xuan Luo. Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training ri-like reasoning models. arXiv preprint arXiv:2503. / /7287, 2025.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wa, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, arxiv preprint arXiv:2312.08933. 2023.
12

---

Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and for long cot from scratch and beyond. arXiv preprint arXiv:2503. /04660 2025.
Wikipedia contributors. Countdown (game (game_shoop show)] 2025.
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, al. Training large language models for reasoning through reverse curriculum reinforcement learning. International Conference on Machine Learning (ICML), 2024.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing I1m reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 14768, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.01981, 2024.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint 2111:22308.01855 2023.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar (tt.::2553..8899)) 2025.
Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404. 18922, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less more for alignment. Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Sections and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 0
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
See Section 4 and Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
The code can be found in https//ggihubbcomm//mmm220000777
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https ssionPolicy) for more details.
7 //nips. www.ul/icgguiee//ooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: To show the robustness of our algorithm, we run experiments on different models and training dataset. However, training LLMs is relatively costly, and we cannot
afford to run the experiments with multiple random seeds. This is also common in other
papers in this area, such as DeepSeek-A et al. (2025). Gandhi et al. (2025 Yu et al. 2025
Zeng al. [2025].
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp/ss/aaarrssss.cc////ccccts
Answer:
[Yes]
Justification:
We conform to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, |ppper swi thcode. com/ conasdatsets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer our LLM policy (tteap////eerrrooocccoooffrrerrs for what should or should not be described.
20

---

A Related Work
In this section, we introduce related work about SFT, RFT, and curriculum learning for reasoning.
Supervised Fine- Tine-Tuning (SFT) for Reasoning. Different SFT methods for enhancing reasoning
capability usually differ in the source of the collected reasoning trace. [Zeng et al. 2025 uses
traditional SFT, i.e. learning from the human-annotater problem solutions. In contrast, Gandhi
et al. [2025), Muennighoff et al. [2025] utilize long chain-of-thoughts solutions generated by some large models, such as Claude and Deepseek-RI [DeepSeek-Al et al., 2025). On the other hand,
Yuan et al. [2033] Xie et al. 2025 utilizes rejection sampling fine-tuning. Specifically, the model will generate multiple reasoning traces, and the one that leads to the correct answer is selected for human annotations the SFT data (traditional SFT), it is sufficient for our purpose and keeps the focus on our main contribution (unifying SFT and RFT).
further fine-tuning. In this paper,
Reinforcement Fine- Tunnng (RFT) for Reasoning. RFT for reasoning can be categorized into process supervision and outcome supervision. Process supervision assigns reward to each step of long reasoning trace [Lightman et al. 2024), which evaluates whether each step is correct or not. The main drawback of process supervision is that it is costly to prepare step-by-step feedback data. On the other hand, outcome supervision assigns single reward to the entire trace [DeepSeek-A] AI itt al. 2025] Zeng et al. 2025 Yu et al. 2025], e.g., whether the trace yields the correct answer to a math problem. Furthermore, Wang et al. [0233 Yuan et al. [2024], Zhong et al. [2024] Luo et al. [2024), Setlur et al. (2025) learn a step-by-ster reward model from a collection of reasoning traces
with outcome rewards, which avoids the cost of preparing step-by-see by--eppte data. In this paper, due to the efficiency and simplicity of outcome supervision, we focus on the comparison with RFT using
outcome supervision.
Curriculum Learning for Reasoning. Existing curriculum reinforcement learning for reasoning mainly focuses on utilizing a collection of problems with varying difficulties (Wen et al.) 2025 Shi
al. 2025] Song al. 2025). These methods train the model with problems gradually increasing difficulty, where the difficulty is determined by predefined criteria, such as the length of the successful reasoning trace (Song et al. 2025) or the success rate of baseline models (Shi et al. [025 Wen et al., 2025]. However, such methods fail when the problems in the dataset are homogeneous in difficulty. In contrast, Xi et al. 2024) proposes a curriculum learning method that concatenates the problem with slice of the solution (hint). The difficulty is determined by the hint length. However, Xi et al. [2024) uses uniform distribution over all possible hint lengths, which misaligns with the distribution of interest (zero hint length). On the other hand, UFT designs hint length scheduler that smoothly reduces the hint length to zero. Furthermore, UFT adds an additional log-likelihooe term for the hint in the objective function, which helps the model to acquire knowledge more efficiently and increases the ceiling of reinforcement learning (cf. Figure
B Experiment Details
In this section, we describe experimental setup and results. We present the pseudocode for UFT (Appendix [B. detail the hyperparameters used (Appendix [B22), and conduct an ablation study (Appendix B.3) We also provide analysis of the model's generalization to larger scales and report additional experimental findings (Appendix B.5).
B.1 Algorithm
The pseudo-code of UFT presented in Algorithm T In lines 4-9: we sample the hint length for each (question, solution, answer) pair in the sampled data batch B3. In lines 11-13, we concatenate the question with the partial solution of length (such as GRPO), with the objective function (3.3) Section discusses the theoretical properties of UFT. To facilitate concrete convergence analysis, we specify the update rule of UFT, and the theoretically grounded variant is provided in Algorithm 2
((L) and feed it into reinforcement learning algorithm
21

---

Algorithm 1: Unified Fine-Tuning
Hyperparameters: KI.-penalty coefficient B, total number of steps T, number of steps with hint Thint lowhigh probability plow /pligh for hint sampling, and hint length I.
Input: Reference policy parameter free
Initialization: ((0) - fref
for / 0, T 1,..,... do Sample batch of problems
.....
2
for (,)) do // For each (question, solution, answer) pair
if < Tiit then
((0)
t poo + (phigh p (1 cos
t +
π
(B.1)
Tiint
// Cosine annealing, π As 3.14159 is the Pi constant Sample ((0) ~ Binomial (min {L, (en(S)) p(0))
else
( =
end
DU {Q + S|: |/ Concatenate the question with the partial solution (hint) and add D
end
13
Run reinforcement learning algorithm on with the objective function 3.3
14 end
B.2 Cost and Implementation Details
The project costs roughly $10,000 GPU hours. The experiment is based VERL (Sheng al. 2024 and TinyZero [Pan et al. 2025). The hyperparameters for training on different datasets are listed in Table T The omitted hyperparameters follow the default values of VERL (Sheng al. 2024
Additionally, provide illustration of hint in MATH(3,4.5), Basically, divide the whole solution by sentences, then uniformly divide the sentences into buckets. Then, during training, we will sample to decide the hint length (how many buckets included).
"For the piecewise function to be continuous, the cases must \"meet\" at $2$ and $-25. ", "For example, Sax+3$ and $x-55 must be equal when $x=2$. " "This implies $a=2)===2-55 which we solve to get $2a=-6 Wiighharrow a=-3$. " "Similarly, $x-5$ and $2x-b$ must be equal "Substituting, we get $-2-5=2(-2)-bb, which implies $b=3$. "So 3+mb==3+3=(Mbxxed(00
Sx=-2s. ",
B.3 Ablation Study
Ablation on Hint Length. We conducted study on Qwen2.5-0.5B (MATH(3,455)., testing dividing
the solution into L, = 4/5/6 pieces uniformly and sampling hint length among {0, 1, L.. We choose MATH(3,4.5) instead of Countdown because the solution length MATH(3,4.5) is relatively longer. The results are shown in Table 2
22

---

Data
Training Batch Size
256
Validation Batch Size
1312
Mini-batch Size
Hint Length
5
Training
Learning Rate
10 6
0.001
500
hhin
300
Number of Rollouts
4
Countdown: 256
Context Window (Prompt)
MATH(3.4.5); 1024
Logic: 1024
Context Window (Response)
1024
plow
0.05
p whigh
0.95
SFT Epochs
5
Reward
Accuracy Reward
Format Correctness Reward
0.1
Incorrect Reward
0.0
Table 1: The hyperparameters for training on different datasets. The other parameters follow the default parameters of VERL [Sheng et al. 2024
Hint Length Setting
Accuracy (%)
Hint length
27.44
Hint length L 5
27.15
Hint length = 6
25.20
Table 2: Ablation hint length for Qwen2.5-0.5B (MATH(3,455).
This suggests that UFT is relatively insensitive the total number of pieces we divided the solution into.
Ablation on 8. shown T
We evaluated different 3 values Qwen2.5-0.5B (Countdown). The results
3
Accuracy (%)
3 = 0.0005
46.09
B 0.001
50.39
B 0.002
59.95
Table 3: Ablation on for Qwen2.5-0.5B (Countdown).
higher 3 amplifies the impact of the supervised log-likelihood term, helping the model learn more from hints. For all main experiments, we adopt 3 = 0.001, the default in VERL.. For larger models, our preliminary results show that varying 3 yields minimal performance changes, likely due their stronger pretrained priors. Table presents the results of Qwen2.5-3B (Countdown).
23

---

B
Accuracy (%)
3 0.0005
77.93
3 0.001
75,59
B 0.002
78.22
Table 4: Ablation on 3 for Qwen2.5-3B (Countdown).
Ablation on Hint Phase Length. (Countdown), as shown in Table
We test different durations for the hint phase on Qwen2.5-0.5B
Hint Phase Length
Accuracy (%)
Thiu 200
52.15
hiat = 300
50.39
T hint 400
50,00
Table 5: Ablation on hint phase length (Thins) for Qwen2.5-0.5B (Countdown).
These small variations confirm that UFT robust to the hint phase length.
Ablation Hint Length Distribution. We also evaluate UFT under two-point hint length sampling scheme: for any expected hint length L E [n, + 1), we set = n with probability + 1 p L and + 1 with probability p L n. p is still determined by the cosine-annealing scheduler. Further details are provided in Table
Hint Length Distribution Accuracy (%)
Two-point
48.63
Binomial
50.39
Table Ablation on hint length distribution for Qwen2.5-0.5B (Countdown).
We can see that UFT is also robust to different choices of hint length distributions.
B.4 Generalization Larger Models
Even though cannot afford training larger model, believe UFT will be helpful for the state-of-the-ar models on tasks out of its capacity. For instance, some tasks involving knowledge of special sub-field or extremely complex reasoning problems.
The following empirical results (Table may hint at this. For instance, on the logic puzzle, which requires more abstract reasoning, Qwen2.5-1.5B performs poorly with RFT alone (0.00%) but achieves 23.00% with UFT:
Task (Qwen2.5-1.5BB
RFT
OFT
Countdown
71.48%
71.48%
MATH(3,4.55
46.68%
47.95%
Logic
0.00%
23.00%
Table 7: Performance comparison between RFT and UFT across tasks with Qwen2.5-1.5B.
In conclusion, although Qwen2.5-1.50 has enough capacity for Countdown and MATH(3,4.5), it is insufficient in solving the logic puzzle and UFT outperforms RFT by a large margin. Therefore, is likely that for larger models, UFT will be helpful when solving extremely hard tasks.
This stark difference Logic suggests that even when a model appears to have "enough capacity" (as Qwen2.5-1.55 1.5.BBBBBBBBBBB does for Countdown and MATH), it can still benefit significantly from UFT when the task challenges reasoning ability. Hence, for much larger models facing extremely complex tasks, anticipate that UFT will continue to provide meaningful ad advantages.

---

Qwon2.5-4.50
Qwen2.5-31
Use 50,
Use
SUT-IFT Rogane
chinko 20 T 55--ar ae Thanctiv apartice otteaee
Thedlas be cquatios -3- (49-2144-)000 20g
IFT Reqpene
Shink 30 -S. kh-cke
<Beel 10-
50
30-32 18
32: 31
UPT Respens
53- 20x 103 nso kylo
Slankl 54
(10
wast)
Real
"aateer (1 453-132 21) Kanawarr
Figure 11: Responses of Qwen2.5-0.5/3B trained by different algorithms.
B.5 Additional Results
Figure 11
shows the response the model trained via different algorithms. For Qwen2.5-0.55B UFT's response aligns with the solution better than RFT's. For Qwen2.5-3B, UFT generates longer reasoning trace and presents skills such as verification [Gandhi et al. 2025 while SFT-RFT does not.
Table shows the accuracy results across different datasets. For clarity, we report the average accuracy over models trained on three datasets: Countdown, MATH(3,4.5) and Logic.
For smaller models such Qwen2.5-0.5B, SFT-RFT achieves an accuracy of 7 7.28%, compared to only 3.25% for RFT. In contrast, UFT achieves 9.45% accuracy, outperforming both.
For larger models such as Qwen2.5-3B, SFT-RFT achieves 17.34% accuracy, which is significantly lower than RFT's 32.15%. However, UFT still performs competitively, reaching 30.93% and closely matching RFT.
In summary, UFT combines the strengths of both SFT and RFT. When the model is small and memorization plays a key role, UFT matches or exceeds SFT's performance. When the model is large and generalization becomes more important, UFT benefits similarly to RFT, achieving comparable accuracy.
C Proof of Theorem 4.2
Theorem 4.2 (Lowerbound). For any integers H 2 1, 2 2, and any RFT algorithm, there exists problem with height H and branching factor B, that satisfies the following: achieve 50% pass success rate, the algorithm needs to explore at least
BH
(4.3)
nodes in Sy. Moreover, when there are multiple nodes in Sy representing the correct solutions, e.g. 8
K 1, any algorithm needs to explore at least nodes in Sh- IK
Proof. Proving the lower bound of exploration is equivalent to the following. Find the maximum T > 0, such that any algorithm will fail learn the optimal policy with probability at least 0.5 within
explorations. Consider the (B#)
K possible trees, each associated with distinct subset of Sy of size K, where that subset represents the correct solution for that specific tree. At the beginning, pick an instance from all those possible trees uniformly at random.
During each exploration, the algorithm requests the reward at node Sy. Let (1), (22)) the leaf node reached at timestep 1, 1.2. which are random variables depending the randomness of the algorithm. Let S {s E S R(s) max R(s')) the set of nodes representing correct solutions. Note that given the construction of the instances, | K. Then, the probability
(T) be
25

---

Model
Algorithm
MATH(31.5)
Counddown
Logic
Olympiad
Avg.
Base
3.03
0.00
0.00
0.00
0.00
1.73
0.74
0.30
7.66
1.55
SET
4.92
0.00
1.61
11.20
1.87
2.13
2.08
1.35
13.07
4,46
RFT
3.78
3.21
8.30
0.00
3.80
2.57
3.87
Qwea2.5-0.5B
SET-ET
8.69
0.00
3.61
17.45
7.07
2.07
4,41
2.12
16.45
7.28
R-
9.86
0.00
6,43
9.99
4.30
3.33
5,02
3.11
20.10
7.36
Base
24.31
3.33
4.82
0.20
2.20
18.27
4.41
5.48
60.96
14.29
SFT
12.47
0.00
5.62
13:48
5,33
6.40
4.53
2.62
29.74
9.36
RFT
Qwaa2.5-1.5B
SFT-RFT
15.72
1.11
6.83
20.51
11.13
5.00
4.41
4.59
30.02
11.70
RJ
28.12
2.22
13.65
23.57
11.47
14,93
7.48
9.43
49.79
18.65
UFT
24.54
10.07
Base
31.45
0.00
13.25
3.81
5,60
24.53
4.78
7.70
51.85
17.13
SFT
24.52
0.00
10.04
15.07
10.30
16.80
5.27
5.19
45.54
15,25
RFT
Qwen2,5.3B
SPT-RITT
26.50
1.11
9.54
17.61
19.60
14.07
5.76
6.77
48.22
17.34
R
44.01
2.22
21.29
27.12
24.50
28.00
10.91
14.57
70.20
28.02
74.63
Basc
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.08
0.01
SFT
1.07
0.00
0.80
13:41
3.67
0.00
0.74
0.25
1.87
2.49
RFT
Llama-3.2-1B
SPT-RITT
0.42
0.00
0,00
18.68
8.33
0.00
1.23
0.20
0:48
3,29
R
1.53
0.00
1.61
9,90
0.13
0.33
2.94
0.99
1.49
2.20
17.87
7.40
Basc
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0,00
SFT
2.54
0.00
0.40
14,68
6.13
0.00
1,72
0.54
7,08
3.85
RFT
Llama-3.2-3B
SPT-RIT
3.16
0.00
2.41
16.05
8.87
0.07
3.92
0.89
5.76
4,79
R
2.93
0.00
3.21
17.55
9.93
0.87
3.06
1.04
5,16
5,03
1.10
Table Average performance of Qwwn2.5500.51155/3 and Llama-3.2-1/3B across all three training datasets, Countdown, MATH(3,4.5), and Logic.
of reaching one of the correct solutions in S is
Pr (f010) S NSH
{s(0) 1-1
ns 0) Pr ((((0))
#0
Σ
(0) E SS
n
s=1
s=1
T
Pr
((0) E Si
[s(0)] 1-1
ns = 0)
Given that pick Sj, uniformly random, Pr Therefore,
{ ( Eir
n 0)
1s
B2
((0)) S 1=1
T
S
Pr
n +
B - -++1
When 7 <
B 4
we have
((10)]
T S Σ
S
(
T
2 2
2 2
Pr
nsn #
1=1
B
1
BH
BH
2
1=1
1=1
ii) uses the fact that S
8
8 2-
Therefore, within
8 s[s)]
exploration, the algorithm will
4 S
fail to find the correct answer with probability at least 0.5.
D Extended Theoretical Justifications
In this section, we introduce some additional notations in Appendix D.I and then present the theoretically sound UFT in Appendix D.2
26

---

D.1 Extended Preliminaries
Notation. For any vector IR", let z; be its 1th element and ||>/|p be the Ly-norm, where ||=|||
denotes the L--norm by default. For any two vectors æ, y E IR", let (xr, y) == -i= Ty yr denote their inner product.
Softmax Parameterized Policy. Formally, the policy TO controlled by 0 F x such that for any E S and E [B],
Algorithm assumes the policy follows softmax parameterization.
exp( a))
(a E exp (xp(((s,a'))
(D.1)
The softmax parameterizee policy is also widely adopted in the literature [Mei et al. 2020 Agarwal al. 2021] Ding al, 2020] sidestep the complexities of analyzing non-convex neural networks
and to keep the focus on the learning algorithm itself.
D.2 Theoretically Sound UFT
The full algorithm is shown in Algorithm 2 In lines 2-3: sample the hint length and trajectory starting from the hint. In lines 6-10, we estimate Q-values by sampling an additional trajectory for each state-action pair, which the objective function and update the parameters by gradient ascent. In lines 16-17, we estimate the expected reward of each intermediate policy and return the best one.
greatly reduce the variance of sampling. In lines 13-14, we compute
Note that Algorithm 2 differs slightly from the UFT shown Algorithm While Algorithm| | leaves the choice of the reinforcement learning algorithm unspecified, Algorithm (ecorihm@explicii explicitly defines the trajectory rolling mechanism and update rule for concrete theoretical analysis. Further, Algorithm P assumes ceftmax--paramterrizee policy, whereas Algorithm [ipossss constraints the policy network architecture.
E Proof of Theorem
In this section, for notational simplicity, we use z(1) to denote n r( for any E {0, 1, T). Moreover, for any t E sampled path H
[T). we define A (st="(sso)) () 0 for those nodes off the
h=(()) at timestep i.
Theorem E.1 (Formal). Consider Algorithm
A When B < 12(H+1) (oog++ooo0000 2+2 the pass @ 1
) satisfies
accuracy Pr a0th (pass @ 1) of policy n Pr
o (pass @ 1) 0.5,
(E.1)
when
T= (1I (1++)² (log B + 2 @ref 11 + +7)
(E.2)
A/12
and explores no more than (BH + NYT leaf nodes in Sh-
Proof. The update rule can be divided into two steps: (i) Use the concentration bound to get a (ef. Appendix E. (ii) Convent
(),,) the difference in each node the V Vz 1) (Sroot) by the regret decomposition lemma (cf.
high-probability bound on (Qx(t-)) (s,-), "h (:1 (---1)
Appendix E.2 (iii) Convert the bound on expected reward to success rate (cf. Appendix E.3)
E.1
Concentration Bound
For any height h E (0) U [H 1), state E Sh, and action [B], we can define the Q-value of the state-action pair (8,a) E x [B] when following policy æ as
Q (s, = E, (.. [R (sn))
(E.3)
27

---

Algorithm 2: Theoretically Sound Unified Fine-Tuning
Hyperparameters: Learning rate 1/ KI.-penatty coefficient (3, and total number of steps T Input: Reference policy parameter fref
Initialization: ( <- g
for t 0, - do Sample ((t) ~ I H,H)
// In fact, any distribution with full support on {0. 1.......... , - H is fine. We choose the uniform distribution for simplicity
Sample trajectory ($) (1)
π 0(1)
where (00 $10
h=10
for h (1)),(0) + for a = 1,2, ...
H do do // Group sampling
Sample trajectory
(5/)
(0),0
T00
starting from sh1 T a)
1))
(0)
W=h+1
(1(1)
hh a) - R
(t),a)
88
end
for
1,2, ... B do 110) h
10
(+)
h
(1)
:2)
- Ea m00h (al (1) ((t) (010
11
end
12
end
(1(0) H
// ((0 =0 for any 8 off the trajectory
h=((()
13
H-1 B
((0)
aa
(0)
A
5/h .a)
8h
((1) a=1
H-1
Σ h=1((1)
KL
70100 (-1s%² (1)
o (-10
Σ
log r (())
(a 17)
14
( -- (()) + v
(D.2)
15 end
16
Estimate Vio
(Broot) N Σn
R (z(t).n) by sampling trajectories 2(1,,n
Sroot and
(1)))
H
n 0i0 where N 12log(14(T+H)
A²
(0.1, Ty oi (Sroot)
E
argmax,
Return: (()
Then, for any E S SH and E [T], we have
E (i
(s, a)] = Pr (sE ( (
(SE (t) H
Qmil-)) (s, a).
(E.4)
where the expectation is taken over the probability of sampling trajectories Algorithm 2 Next, we will introduce Lemma 5.3 in Liu al. 20044)
Proposition E.2. Let M, 2 be the constants such that (x)) (x)) M and
((0()) - (xx) M for any [T] and x, x' E C, where is a convex set. for any
28

---

E C, we have
E (x)) (it) fio) f = (x).
and (1) is deterministically influenced by f(t) f(2),... ..... fi(-1), then for any d E (0, 1) and a E C, we have
Pr Σ (10()) (() = Σ (x)) ((())) + (M + M) V 21 log
1 {
2 21-8 8.
For any h < H and E Sr. let flr (x) = Pr EE 5b S n-mi 17 .)) (Qri (s,, )), ),z)) where
{ (1-1)] H
f() A [0, 1] since each element of (() (s. is bounded by [0, 1] by definition. Therefore,
M in Proposition E.2 is 1. Similarly, let F(() (x) ()(-1)(s, -) xx and we have M- = 1. Therefore,
by E.4 Proposition E.2 and Lemma E.3 for any E (0, 1). with probability at least - 6, we have
T
(1) Pr 8 sh ) (Q
(s, )) x- (-1s) x x0-1) (s)
(s,-)) )), (. (.1s) 2) (-1s)) + +2 2T log 1
8
(1(-1) (s, (), (-18) 2-- 22 27 log 8
ii) is because
A (s, )) x 2. -11)) x((11((11)) 18)
@
- rl (- 8))
B
(a () (s, a) Σ
(T s) x00-1 s))
- (s, ). (.1s) 1(" (-1s))
By the update rule of Algorithm Lemma E.3, Consider Algorithm
2
we have the following lemma.
For any node s E ) SH, we have
T
1
-), n (-1s) s) - (1 (-1ss) ss)
<
3T
KL T (-1 s) (.1s) + 2nTT.
The proof is postponed to Appendix E.4 Lemma E.3 gives us an upper bound on the accumulated difference between our policy (?(-1) 1) and the optimal policy n². Therefore,
T
(1) "1 s/ ) ) (Qr (s, (. (-1s) - (-
Pr s S
[ ([)-))(ss,)) (s.-), x (-|s) (-))) + 2 2T log 8
+ BT KL 7 (-(=)) + 2TF + +2 2T log 8
29

---

E.2 Difference Decomposition
Let (s) be the probability of reaching state s from the root by following policy и. Hence, μ² (Sroot) = 1. For any s E S \ Sn and action a E [B], μ² (T(s, a)) can be recursively defined as 4² (T(s, = J (s) as)
(E.5)
In the following, we will introduce Lemma E.4 which is special case of the regret decomposition lemma (Lemma 5.1) in Liu et al. (2023). Specifically, it is the regret decomposition lemma for a two-player zero-sum extensive-form game without chance node and the second player's action sets at all nodes are of size 1.
Lemma E.4. For any sequence of policies x01,, (T()), ,,,, (T)) and policy x, we have
E (sroat) - V* (s
E sES
1 (s)
(...),=(-133 (s:))t)))))) - (')(ss))
Lemma E.4 can also be viewed as the performance difference lemma in reinforcement learning (Kakade and Langford| 2002] for tree-shape Markov decision process. For completeness, also provide the proof at the end of this section.
By letting (1)) (t-11)
)) for any E [T] and x = x², we have
T
y 1) (5r00t)
V"
T
(.) - T (--1(((-s))
1=1
T
i
oT a (1 (.1 (.1s))
1= (s)
(0) "1
Pr s SE
8/ }
(()
E (laini. 1 158 t=l Pr
s 8/ Sn
@ xT" (: s) (1 (-1s) ss)
uses the fact that x² is deterministic such that n² (s) > 0 only when s E {s si
. Since
sampled from {s si.... . uniformly, for any s E {sg. si. ... s }. we have
,
Pr
{ (8) E { shh k=
2 Pr (s=
$60)
(t)
1
H 1
1= (s) sE (E((()))
Therefore,
< H 1 and we have
( (1)"" "
T
Vall 1) (S200t)
H-1
1² (8%)
(C,+ ST) KL (T'(-177))) + 2n + 2 2T log
sh {
(1)
h=0
Pr
8/ h
H-1
KI. (T'( (-185))17000 (-105)) + 21/T + 2 2T + 3T
2T log 1
<(H+1)
Next, we can bound KL
(-1
(-1s)) (+I by the following lemma.
'Chance nodes represent the randomness of the game, such as rolling dice.
30

---

Lemma E.5. For any h E {0,1, H 1}, we have
KL
a ( (1)
0 (-1 (-1s)) = log B + @
The proof is postponed to Appendix E.4 Therefore, by taking 1 V have
Σ V yoll (sroot)
< SH +12
(log B + rr
VT + 22T + 2 2T log 1
H-1
(H + 1)
Σ h=0
KI. x
(.1s%)
Because V* (=(l-1) (Sroot
2 0 for any t E [T]. according to pigeon hole principle, there must
exist E {0,1, T3 such that
v Valh"
(srott
H + 1)²
(log + +2| oo)) + 2/ log
VT
H-1
+ +HH + 11
Σ
KL (-1)) |T (-18%))
h=0
For any <> 8(1 + h=0 KL (- |T]][ rrsf (. (.1$%) it takes
2
(H 11) log + Oref
+ 2 2 log _
00
c- B(H + 1)
1 Ei=o KL (-18)) | 0 (.1s)))
iterations satisfy V*_ (z((") (Sront) S.
Recall that A > 0 is the sub-optimality gap. By picking
A 6
5
8. and B S
A
get accuracy with probability
need
12(H+1)) (log B+ 2 21(er=((1,))
(H (H 11) (log B + gref +7)
T
A/12
iterations, which implies T < O (H'rlog By²
Since O(B H) leaf nodes are explored at each
iteration, the number of leaf nodes explored during training O(B H TT) s 50 (BB H (log B)²
E.3 Compute Probability
To find 1", we need estimate Vx" (t) for all / (0,1,...,77 ,) by sampling trajectories. By sampling trajectory from .CC get random variable from Bernoulli (Preand (pass @ 1)
representing whether the trajectory reaches the correct solution. Then, by Hoeffding's inequality, by sampling N trajectories, have
Pr ( 7 (() (Sroot) - Vn (Sroot)
2
(6)
<
S eexp
(E.6)
12
72
7(T + +1)
31

---

ii is by definition of N in Algorithm 2 By union bound, for any t E Vnolt)
(front) - Vn (Sront)
S
A
holds with probability at least 7(++)) 7. Therefore,
T+1
12
Vx (Sroot) 2 Vnoces (Sroot)
(5root
12
12
A
(8rot)
V
6
3
Recall that Pr (rr) (pass @ 1) is the pass @ I accuracy of policy (P"), In the following, we will use Prod as shorthand of Pr (-1 (1Iv=( (Sroot) 2 V
Proond (pass @ 1) Poud
(((sH)=
max R(('H)
8g (8) ho (10)
cond
(R(8H)=VV
Furthermore,
3 S VF(() (8100t) =B (or (t)) [R(HH)]
cond
(R(sH) V*) V*
Ap = 9ro0s (1)
Proond
sn scont (Na)) h (R(sH) = V*) (V* -A).
By combining all pieces together, we have
Proond (pass @ 1) + (1 11-Pr moodd (pass @ 1)) AA)
A
3
which implies that pocond (pass @ 1) 3.4
m((*)
Finally,
Prace (pass 1)
Pr cond (pass 1) Pr ((ro()()rrot)) (* --- - (TT (Sroot) > px(("))
(8root)
(Sroot)
6 1 2 (1 8)
3
2
E.4
Omitted Proofs
Lemma F.3. Consider Algorithm
For any node & E S Sh we have
T
1 11 (s. ), n (- (-1s) - n (- (-1s))
1=1
BT KL (-)ss))) 0" (.1s) + 2nT.
<
Proof. We will introduce the following one-step analysis of the update rule first. Lemma E.6. For any node E S SH and E [T'. have
1
(- - x (.1s))
SKL ( ||] (t-2)
KL (- (=1) KL (-11(-1)s) (x(()-))
+ 7KKL (-18))]17 0' (-1s)).
32

---

The proof is presented later in this section. Therefore,
"7 A (s, x (-1 - (.1s)
SKI. (x (=1(1))11(--)((111)) KI. x (1)1))) - KI. (1)) (1 (-18)
+ + (- rrar (-18))
By adding 17 (A(-1)(s.)) .), (.1s) x(t (. [s] on both sides, we have
1 1 (8, ) n." (-1s) - -x x.-1)(.16)) 16))
KL
(-1s) (11))))))(( KL (I KL (((1s))=((00)))
(7' (. ())
+ 1) ( ) (z (.16) T (.)3)) + m/KKI (T' 10 (116)
By Hölder's inequality, we have
A 11 (s.
(t) (-1s) - x( (-1s))
π (0) (-1s) - x (-1s))
<2n A
1 | --s)) (-
+ 87
(9)
1 KL (
2n + 4n
(11))
(i) uses 1( ))
I 1 and Pinsker's inequality. Therefore,
x(-1) ) nt" (-|ss) - x (.1)))
<KL
(: [ (+)))) KL
+ +2n² + n3KL (T'(11))]l (0))s))((((()))) (-1s)).
By telescoping, we have
T - - (+)),
7
t=1
<KL
(.)
KL
(s)) +21/T + n19KL (1))) grot (.15) T
(1)
SKL
+ 21/ + nKK (-)s (-1s)) T.
(i) uses the non-negativity of KL-divergence. By dividing on both sides, we have
(s, - n (-1s))
< KL (0) (.1s)) + +2nIT + +K3KL (T" (-1s) gust (-1s)) T
10)
(6) KL (-)ss()]l rra (-1s) + 2r/T' + 3KL 7 (-1s)] (-1s) T.
(i) is because nr (-) s) n Trr (.1s) by the initialization of Algorithm
Lemma E.4. For any sequence of policies (1),, (?),,...,,,7)) and policy x, we have
T (Txh) (s, (.)))))())))))) = (.1)))
(Sroot) - -v*") (Froot) =
Σ 1r
SES
1=1
33

---

Proof. The lemma can be proved by induction. When H = 1, Lemma E.4 holds since ²=a (froot: a) R (T (Sroot: a)) Q" (Sroot: a) for any action E [B] and [2]. There- fore,
T Σ E (0 (s,-), - π (-1s))
SE t=1
T
1 (Froot) (10" -) (T( (root)) - (Qrr (Stoots )z (,
Broot)) (-)rot)))) ))
V (8100t) Vx(l) (9root))
For any two nodes 8, s' we write s' if is an ancestor of s' in the search tree. Consider when Lemma E.4 holds for any search tree of height H I Ho- Then, for H = Hu + 1, we have
E Σ (s) (0 (s.)) ((.
n (.1s)
SES
T
(Sroot) (0)(0) (Sroots :), T(- STont) 7()) (-1 (((sroo)))
B
T (s) ((is)(or)))
Σ
Σ
(0))))())((())))))))))))))) (s,), - (-))()
+
SES)SH a=1
T(sroot [s
Then, according the induction hypothesis, for any a E [B], since the subtree rooted ((Srott: 0) is a tree of height Ho, we have
Σe (s) (s) (s.), ( (-1s)
SES al)
=T(a| Sroot)
VT (T(3rootss)) a)) v*(() (T (T(sroot: a)))
Moreover, by definition, have Qm! (sroot, Vx(1) (T(rooos a)). Therefore,
Σ
th (s) (Qxx" (8.-). (-1s))
*ES'SSH
t=1
T
(Sroot) (Qrit)) (Sroper ) ((. Sroot) (0) Spoot))
xaa Spoot) Σ 4al (TT a)) - Vx(*)) (T (T a))
+
T(a Sroot) (00 (a Stoot)) v= (T (Sroot: a))
B
T
+ V" (Sroot YT
Σ
n raa Sroot)
Σ
vall) (T(sroot, a))
1=1
T
4d (Sract) - (t(()) (Srout)
Therefore, Lemma E.4 also holds when H Ho + 1 and thus can conclude the proof.
34

---

Lemma E.6. For any node E S SH and t E [T]. we have
A (A--)/ -). x (-1 x (.1s)
SKL (-'(|s)|[T(0=))
KL x (1(1s))])(((s()
KL (1)
KL
+ (T" (- (-13)]17 (-16))
Proof. Let h be the height of 8. There are three possibilities on V I() (I) ((1-1)(s..)) +
3 log T (- 3 log π ond (-1=) + 81; (II) A one-hot vector with only index aj be 5 x) (III) 0.
Then, will show that (.2) is equivalent to the following in different cases. Lemma E.7. For any t E equivalent to the following.
21, .T), h 10,1,-.. , .. - 1}, and node e E Sh. D.2) is
π ( = argmin
(..)),((111) + 3KI. (I( [-][1=)]=== (-18)
=(1 1)EA
KL (T-11)))77(0-1)
(I)
(-(ss ()) argmin
(- ( ( (s))
+ n
1 KL ( (T(15))[(((
(II. II))
x(1 ")
where (I), (II), (III) stand for the cases when
(s,-)sss.) Blog (- - 3 log π ~ (-1s) + 81 (1)
one-hot vector with only index oj be 11-1)) 0.
(III)
Then, will introduce special case of Lemma 3,0.3 from Liu 2025 Lemma E.8. For any node s, vector g RR. n > 0, Po 0, policy (0) A and reference
policy ref E EAB let
+ _ KL. (201(x10)) ) 1
x argmin #EAB
(g.xx) 3OKL (x)lx""*
Then, for any (2) E A B have
m3 KL
(o (1 - 313KKL (222) + (g, (1)
x0(2) -x(2)
(E.7)
x (2) (((0) - 7 (2(2) - KL. (2=0)
<KL
Consider (I) first. For any node s S) Sn and E [T]. by taking ax xt( (-(1))),
(- (0) = x (t--1)( x = rr (.1 (18),g = - -)t and By = 3, we have
n8KL (1) [1))s (-1s) n8KL (-))))) (-1ss)
+ + A (s. ) (.1s) - (.1ss)
<KL 7 ()) (- (8) (1 + KL. ( ((1s))](((((((1)))))) KI. KI.
Further, by the non negativiiy of KL-divergence, we have
[ 1) (8, n (.1s) - (.
<KL
(5 (: [s][ (0-))
KL ( (-)))))))()) - KL (-) (-])) (-1s))
+ 1K3LL (-'(11)]1=> 17 gra (-1s))
35

---

Consider (II). For any node s E S Sh and t E [77, by taking x(2) = x x*(1ss) =()))
(-1ss, x0(0) (: s), wrtf x (-1s),g- = v. 12) and 8o in Lemma E.8 we have
([x(113) x (:16) - (-1( (s)
<KL TT (- 8) (.11s))))))) - KI. (-'))))]((()((11) KL (.1)))
Moreover,
=9 <" 18%) - T (a (hh)
((((l))(((11) (-(1ss - x.0((
(a) (s))
20
(ic)
A (x, ) x =-11(())))))))))<{)) (-) (.1b))
(i) uses the fact that xt "a(a; [88) 1 and (ii) uses ( (s,-) - by definition. Therefore,
() ), x (-1s) (-)ss
<KL (T (-16) (-) ()) (-16)
KL (-)))))]7((( KL ( (-1 (1)1) (.16)) (E.8)
For (III, which is 8 off the sampled trajectory at step - 1, by definition we have (s,-)) = O. Then,
(Vr019) T- (-1s) - (- = 0 =
(-1s) (-)s-)))
and E. also holds.
Lemma E.S. For any h {0, 10.,, HH 11-1,, 1}, we have
KL n (: (.15%))) |l gres (-16%) I log B + 2 |r Il
Proof. For any h E {0} U [ - 1]. since x" is deterministic. let aj be the action such that T² (a7, si) Then,
B
T'aal (%))
KL n (- s grot
gret (-11))
Σ a=1
(a s log o (a s9))
= log
n Braf (nj, (8)))
By definition, we have
n graf (1) (8))
exp (fref (sh a)
exp
Orof
exp (-2 @ref
2
Dn exp (Ozel (a)
B Bexp (10roof (lao)
B
Therefore,
KL π (. s1)]1 grn (+1s5)) S log (B Beexppp (2 22100ref| )l=)) = log B + 2 2||0rec11
Lemma E.7. For any t E {1,2, ......TTTp. equivalent to the following,
.T.. h E 10,1,..... .H - 1}, and node s E Sn,
D.2) is
(-18) argmin =(-1))EA
(- (s.)),((.13)) + 3KL (I(( (z)(== (-1x))
(0.
1 KL T(- [T][ (t-()-)))))) (((11)))(((0(((11)))
(I)
x ((18) = argmin (- (-Va(1a))(tt a (1)) + KL (T)(15)
(II, II))
x(- e)
n7
where (I), (II), (III) stand for the cases when
Ä (s. 3log n(( (-1))) - 3 log o (-|s)
+ 181 (I)
V V
one-hot vector with only index oj be [-]]]
(II
(III)
36

---

Proof. The Lagrangian of (-A (. a(-1s)) + 3KL (I+|œ*) + " KK ( | [))[ (-1s))
L (-1)())))) = (s. + 3KL (-1)()))) 1T (-11s)
B
1 KL (T {>]| 1 (.18))
Σ
(a 8) -
[B], by setting (-(x())--1s)) (al))
For any action
0, we have
(a 8)
n (a 8)
(s, + B log
+ 8 + log
+ 1
+ x = 0,
Drai (a s)
-18-1-n24t7 "x: (e:)+ log
pre 14n0
(a (al=)) (lo()))
which implies that π (a| exp
exp
By further setting (c(=))(.15)
0, we have
0x
Σ (a s)= 1.
Therefore, by combining all pieces together, we have
11( (s. + n log (0 (als)) + log (a {s|
(a|s) exp
12
1+13 1/3
(st :) 3llog T (a (als) + log (a(tt))))aa))
x exp
1 + n13
1) -(t- 11 (s, +
n79
gree (8,0) () + 1 nB
1 glt (0-111))))))
x exp
1 + n/33
1 + n/3
( (a.)) log orat (a +log (1-)))())
In (1), 2 = >e exp
14
For (II) (III, the proof can be concluded by setting 3 0 and changing Á As.-) to
V 7 [((-1))((111))
37


Reviews:
[{"review_id": "YJfOPJvXvQ", "reviewer": "Reviewer_KErd", "summary": "This paper brings together supervised and reinforcement fine tuning into a single framework. The authors claim that the combination of SFT and RFT can actually break the exponential sample complexity. SFT+RFT hold promise of combining curriculum memorization with reward-driven exploration, thereby enabling thinking/reasoning and memorizing at the same time. The authors contrast it with approaches that use SFT or RFT alone, or combine them sequentially.", "questions": "* Does UFT unification proposed by authors generalize to larger models that have enough capacity to just train on RFT alone? \n* My main concern is lack of in-depth experimentation with a broader set of models and larger model sizes (e.g. 70B). If the authors convince me that UFT generalizes to more and larger models -- this can be convincing evidence of desired generalization (and could help +1 my score). Otherwise, UFT's contributions might be very constrained. \n* it would be good to show how UFT compares to larger SOTA models given it's better convergence and sample efficiency (i.e. within some time budget or upper limit on the number of steps). While larger SOTA models may eventually converge to better accuracy, UFT may offer competitive performance within a time or step limit.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* smooth transition from SFT to RFT during fine tuning is intuitive, making the exploration task progressively harder and reducing the dependency on supervision gradually over time.\n* The authors consider previous work thoroughly, citing R^3\n* The paper offers intuitive motivation for unifying SFT with RFT based on human intelligence (though some may find drawing these parallels as a disadvantage)\n* theoretical proof is offered for polynomial sample complexity for UFT. \n* information-theoretic intuition/explanation is offered (sec 3.2) for why RFT's sparse reward signal (correct/incorrect) is inferior to injecting new knowledge through partial SFT hints. Thus, UFT is shown to leverage the strengths of both fine tuning approaches. \n* UFT reduces to SFT as a special case, mathematically (remark 3.1)\n\n**Weaknesses**\n* Exploration was done on non-SOTA models that are quite small Qwen2.5-0.5/1.5/3B. (only an ablation study for hint length predictors was done on LLama 3.2-1)\n* UFT may not generalize (or not offer any benefits) for larger reasoning models.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer KErd\n\nWe sincerely appreciate your thoughtful review and constructive feedback. Below, we address your primary concern regarding the generalization of UFT to larger models and large-scale experiments.\n\n### UFT's generalization to larger models\n\nEven though we cannot afford training a 70B model, we believe UFT will be helpful for 70B models on tasks out of its capacity (unseen tasks during pretraining / extremely hard reasoning problems).\n\nWhile we are currently unable to run experiments on 70B-scale models due to computational constraints, we believe that UFT will be beneficial even for such models, especially on tasks that exceed their pretraining capabilities. For instance, some tasks involving specific domain knowledge or extremely complex reasoning problems.\n\nThe following empirical results may hint at this. For instance, on the logic puzzle, which requires more abstract reasoning, Qwen2.5-1.5B performs poorly with RFT alone (0.00%) but achieves 23.00% with UFT:\n\nTask (Qwen2.5-1.5B)|  RFT | UFT|\n|------------------|-----|----|\n| Countdown        |    71.48\\% |71.48\\%  |\n| MATH(3,4,5)      |      46.68\\% | 47.95\\% |\n| Logic            |     0.00\\%  | 23.00\\%   |\n\nIn conclusion, although Qwen2.5-1.5B has enough capacity for Countdown and MATH(3,4,5), it is insufficient in solving the logic puzzle and UFT outperforms RFT by a large margin. Therefore, it is likely that for larger models, UFT will be helpful when solving extremely hard tasks.\n\nThis stark difference in Logic suggests that even when a model appears to have “enough capacity” (as Qwen2.5-1.5B does for Countdown and MATH), it can still benefit significantly from UFT when the task challenges its reasoning ability. Hence, for much larger models facing extremely complex tasks, we anticipate that UFT will continue to provide meaningful advantages.\n\n### Experiments on large models\n\nWe fully agree that broader experiments would offer more insight. However, as noted in Appendix B.2, our current experiments already incurred a cost of approximately $10,000, and scaling to 70B models would be prohibitively expensive for us at this stage.\n\nThat said, we emphasize that absolute model size is not the sole driver of UFT's benefits. What matters more is the relative complexity between the model's capacity and the task. Our findings show that UFT outperforms RFT in settings relatively hard for the model (such as the logic puzzle for Qwen2.5-1.5B), which is a likely scenario even for large models on particularly difficult or unfamiliar problems."}, {"review_id": "LtTADCXwTL", "reviewer": "Reviewer_MekW", "summary": "This paper introduces Unified Fine-Tuning (UFT), a novel post-training paradigm that integrates supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT) into a single process. The authors address the fundamental trade-off in LLM post-training: SFT excels at knowledge acquisition but suffers from overfitting, while RFT provides better generalization but struggles with sparse rewards and depends heavily on base model capabilities. UFT resolves this by using gradually shortened \"hints\" and a hybrid objective function that combines reinforcement learning with log-likelihood maximization on these hints.", "questions": "1. While UFT demonstrates superior performance on most benchmarks, SFT-RFT occasionally outperforms UFT on certain tasks (e.g., Countdown and Logic), and Table 2 reveals that UFT doesn't universally dominate across all evaluation metrics. This raises critical questions about when practitioners should choose SFT-RFT versus UFT. What task characteristics, model properties, or dataset features determine the optimal choice between these methods?\n2. The extensive mathematical formulation in lines 88-101 introduces numerous tree-based definitions and notations that appear underutilized in subsequent theoretical development. Notably, the core objective functions in Equations 3.1 and 3.2 seem largely independent of the elaborate tree structure definitions.\n3.  The paper lacks clear guidance on how to strategically allocate training data between SFT and RFT phases. Should practitioners prioritize difficult examples for the SFT stage to provide strong foundational knowledge, or should simpler examples be used for supervised learning while reserving complex cases for reinforcement learning?\n4. A comprehensive analysis comparing computational overhead between UFT, pure SFT, and pure RFT is notably absent.\n5. The decision to begin zero-hint training at step 300 in Figure 4 appears empirically motivated rather than theoretically grounded. How can the optimal timing for beginning zero-hint training be determined across different model sizes, task complexities, and dataset characteristics?\n6. Several theoretical claims lack sufficient justification, most notably the assertion on line 185 that a particular estimator is unbiased without providing formal proof. While the authors present extensive theoretical analysis in the appendix, key intermediate results like the optimal training step count remain underdeveloped.\n7. The evaluation is primarily concentrated on mathematical and logical reasoning tasks using relatively small language models (≤3B parameters). This narrow scope limits the generalizability of findings to broader reasoning domains, or larger model scales.\n8. Algorithm 2's Q-value computation (line 7) appears oversimplified. Additionally, unclear definitions of key metrics (e.g., \"Average Score\" in Figure 4) and missing comparative analyses (e.g., standard RFT performance versus RFT with cosine annealing in Figure 5) limit the comprehensiveness of the experimental evaluation.", "limitations": "Yes. However, there are only one limitation about human-annotated solutions. I suggest the author could add more limitation analysis and training cost of UFT.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n1. The paper addresses a genuine limitation in current fine-tuning approaches. The observation that SFT works well for small models but may lead to overfitting for larger ones, while RFT generalizes better but depends heavily on base model capabilities, is well-supported and practically relevant.\n2. The core idea is intuitive. The cosine annealing schedule for hint length provides a smooth transition from supervised to reinforcement learning, avoiding the distribution shifts that plague staged approaches.\n3. The evaluation across multiple model sizes (0.5B-3B), architectures (Qwen2.5, Llama-3.2), and tasks (Countdown, MATH, Logic puzzles) demonstrates the generality of the approach.\n\n## Weakness\n1.  The paper doesn't discuss the computational cost of UFT relative to baselines. The hybrid objective and hint processing likely increase training time.\n2. The method introduces additional hyperparameters (hint length schedule, β weighting). While the paper shows UFT is effective, more analysis of sensitivity to these choices would help practitioners.\n3. It would strengthen the paper to include other domains like code generation, scientific questions, or other reasoning domains to demonstrate broader applicability.", "was_revised": true, "final_justification": "After careful consideration of the rebuttal and discussions, I recommend accepting this paper. The authors have successfully addressed my primary concerns through comprehensive experimental evidence: (1) computational cost analysis showing UFT is actually faster than RFT due to shorter generation lengths during the hint phase, (2) extensive ablation studies demonstrating robustness to hyperparameter choices, and (3) clear practical value in providing a reliable unified approach when the optimal training strategy is unknown beforehand. While some limitations remain—particularly the evaluation scope being limited to mathematical/logical reasoning tasks on smaller models (≤3B) and the lack of theoretical convergence guarantees—the core contribution of seamlessly integrating SFT and RFT through gradually shortened hints represents a meaningful advance in LLM post-training. The method's consistent performance across diverse settings, especially avoiding catastrophic failures like RFT's 0% on Logic tasks, combined with its computational efficiency, makes this a valuable contribution to the community that merits publication.", "rebuttal": "# Response to Reviewer MekW\n\nThank you for your comprehensive and thoughtful review. We deeply appreciate your questions and suggestions. Below, we address each of your points in detail.\n\n### When to use UFT\n\nUFT’s main advantage lies in its robustness across tasks and model capacities. While SFT, RFT, or SFT-RFT may perform slightly better in specific scenarios, it is often difficult to determine the optimal fine-tuning strategy beforehand. UFT offers a strong, consistent alternative that adapts across regimes.\n\nAs demonstrated in the table below (Qwen2.5-1.5B), RFT surpasses SFT-RFT on Countdown and MATH(3,4,5), but fails entirely on Logic. UFT, by contrast, maintains competitive or superior performance across all tasks:\n\nTask (Qwen2.5-1.5B)| SFT-RFT | RFT | UFT|\n|------------------|---------|-----|----|\n| Countdown        |   57.13\\%      |   71.48\\% |71.48\\%  |\n| MATH(3,4,5)      |   36.91\\%      |   46.68\\% | 47.95\\% |\n| Logic            |   29.6\\%      |   0.00\\%  | 23.00\\%   |\n| Average          |   41.21\\%    |   39.39\\%    |   47.48 |\n\nThis illustrates that while task complexity and model capacity are important, their interaction is nontrivial. UFT offers a reliable choice when the best training scheme is unknown.\n\n### Computational cost comparison\n\nWe evaluated the average training step time across models. Surprisingly, UFT is faster than RFT, due to shorter outputs in the early training phase (thanks to hints):\n\n| Model          | RFT    | UFT    | \n|---------------|---------|--------|\n| Qwen2.5-0.5B       | 63.62  | 57.18  | \n| Qwen2.5-1.5B          | 98.25  | 89.21  | \n| Qwen2.5-3B           | 127.14 | 118.36 | \n| Llama-3.2-1B    | 96.61  | 74.63  | \n| Llama-3.2-3B     | 126.90 | 108.18 | \n| Average            | 102.50  | 89.51   | \n\nThis improvement stems from shorter generation lengths during the hint phase. We also report the rollout time per step below, showing clear gains:\n\n| Model | RFT | UFT |\n|-------|-----|-----|\n| Qwen2.5-0.5B | 26.75 | 20.47 | \n| Qwen2.5-1.5B | 41.34 | 32.40 | \n| Qwen2.5-3B | 63.98 | 55.35 | \n| Llama-3.2-1B | 52.30 | 32.13 | \n| Llama-3.2-3B | 65.22 | 48.28 | \n| Average | 49.92 | 37.72 |\n\nPure SFT remains the most efficient (about 6-9x faster than RFT/UFT), as it does not involve trajectory sampling or reward computation.\n\n### Hint length ablation\n\nWe conducted a study on Qwen2.5-0.5B (MATH(3,4,5)), testing dividing the solution into $L=4/5/6$ pieces uniformly and sampling hint length among $\\{0,1,\\dots, L\\}$. We choose MATH(3,4,5) instead of Countdown because the solution length of MATH(3,4,5) is relatively longer.\n\n- Hint length $L=4$: 27.44\\%\n- Hint length $L=5$: 27.15\\%\n- Hint length $L=6$: 25.20\\%\n\nThis suggests that UFT is relatively insensitive to the total number of pieces we divided the solution into.\n\n### $\\beta$ ablation\n\nWe evaluated different $\\beta$ values on Qwen2.5-0.5B (Countdown):\n\n- $\\beta=0.0005$: 46.09\\%\n- $\\beta=0.001$: 50.39\\%\n- $\\beta=0.002$: 59.95\\%\n\nA higher $\\beta$ amplifies the impact of the supervised log-likelihood term, helping the model learn more from hints. For all main experiments, we adopt $\\beta$ = 0.001, the default in VERL. For larger models, our preliminary results show that varying $\\beta$ yields minimal performance changes, likely due to their stronger pretrained priors. Here are the results of Qwen2.5-3B (Countdown).\n\n- $\\beta=0.0005$: 77.93\\%\n- $\\beta=0.001$: 75.59\\%\n- $\\beta=0.002$: 78.22\\%\n\n### Length of hint phase ablation\n\nWe tested different durations for the hint phase on Qwen2.5-0.5B (Countdown):\n\n- $T_{hint}=200$: 52.15\\%\n- $T_{hint}=300$: 50.39\\%\n- $T_{hint}=400$: 50.00\\%\n\nThese small variations confirm that UFT is robust to the hint phase length.\n\n### Data allocation strategy\n\nWe appreciate your insightful suggestion. We agree that placing harder examples in the supervised phase and easier ones in the RL phase may further enhance performance. Moreover, adaptively adjusting hint length based on the hardness of each data point is also promising.\n\nHowever, the current paper’s goal is to demonstrate the value of integrating supervising signals to reinforcement learning, and we leave adaptive data allocation to future work.\n\n### Clarification of theory\n\n- Tree Structure (Lines 82–101): These definitions are necessary to formally define the trajectory sampling and expected return in Section 3.\n- Optimal Step Count: Since GRPO is a convergent algorithm, the more steps we train, the higher the final accuracy is. The accurate value of $T$ of achieving $50\\%$ accuracy is given in Lemma E.1.\n- In Algorithm 2, using reward as the Q-value follows the practice in GRPO [1], where advantage is computed as $\\tilde A(s,a)=\\frac{r(s,a)-mean({\\bf r})}{std({\\bf r})}$. Our implementation omits normalization for simplicity, in line with GRPO variants [2].\n\n### References\n\n[1] Shao Z, Wang P, Zhu Q, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models[J]. arXiv preprint arXiv:2402.03300, 2024.\n\n[2] Liu Z, Chen C, Li W, et al. Understanding r1-zero-like training: A critical perspective[J]. arXiv preprint arXiv:2503.20783, 2025."}, {"review_id": "c2SUrwaUsj", "reviewer": "Reviewer_BALi", "summary": "This paper proposes UFT, a post-training method that unifies SFT and RFT into a single integrated process. The authors argue that SFT suffers from overfitting while RFT depends heavily on base model strength, and UFT addresses both limitations by using partial solutions as \"hints\" to guide exploration while incorporating supervision signals. They provide theoretical analysis claiming exponential sample complexity improvements over standard RFT and empirical validation on reasoning tasks using various language models.", "questions": "- Hint length is sampled from a Binomial distribution. What is the rationale for this specific distributional choice? How would performance be affected by using a simpler scheme?\n- Figure 4 (right) indicates that hints are phased out entirely after 300 steps. Does this imply the hint mechanism's primary role is as a warm-up, with the final performance advantage stemming solely from the modified objective function?\n- The objective in Eq3.2 includes the term $KL(\\pi^\\ast\\|\\|\\pi)$, which you estimate via a single monte carlo trace using the annotated trajectory $(s_h^\\ast, a_h^\\ast)$. This seems to be an exceptionally high-variance estimator. How does this not introduce significant instability during training?\n- Could you provide a detailed ablation study that isolates the performance contribution of the $\\sum KL(\\pi^\\ast\\|\\|\\pi)$ term? Specifically, what is the performance of UFT if this term is removed, leaving only the GRPO objective and the standard log-likelihood on the hint?\n- How do you justify the direct application of theoretical results derived from a deterministic, uniform tree-search model to the stochastic and unstructured nature of LLM text generation?\n- The informal statement of Theorem 4.3 requires $\\beta$ be \"small enough.\" Could you formalize this condition and comment on whether the empirically optimal value of $\\beta$ satisfies this theoretical constraint?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "pros\n\n- The approach tries to combine SFT and RFT in a principled manner, which is an interesting topic.\n- The paper provides clear motivation.\n- The paper is well written.\n\ncons\n\n- I'm not fully convinced by the theoretical analysis. See the questions below.", "was_revised": true, "final_justification": "All my questions and concerns are resolved.", "rebuttal": "# Response to Reviewer BALi\n\nThank you for your thoughtful and detailed feedback. We sincerely appreciate your insights, and we address your comments point by point below.\n\n### Ablation study on distributions\n\nWe chose the binomial distribution for hint length sampling due to its simplicity and desirable properties: (i) it is discrete with support over integers (hint lengths), and (ii) the expected hint length can be precisely controlled, which is crucial for implementing the cosine-annealing schedule.\n\nTo assess the sensitivity of UFT to this choice, we also experimented with a two-point distribution: for any expected hint length $p\\cdot L\\in [n, n+1)$, we set $l=n$ with probability $n+1-p\\cdot L$ and $l=n+1$ with probability $p\\cdot L-n$. $p$ is still determined by the cosine-annealing scheduler. Using this scheme, the final accuracy of Qwen2.5-0.5B (Countdown) is:\n\n- Two-point: 48.63\\%\n- Binomial distribution: 50.39%\n\nThis suggests that the choice of distribution's impact on overall performance is relatively small.\n\n### KL-Divergence estimation in (3.2)\n\nAs we discussed in Line 180-186, $KL(\\pi^{\\*}|| \\pi)$ (red term) is equivalent to the log-likelihood on the hint (the blue term in Eq 3.3) by definition of the KL-divergence.\n\nSpecifically, $KL(\\pi^{\\*}|| \\pi)=\\sum_a \\pi^{\\*}(a|s)\\log \\frac{\\pi^{\\*}(a|s)}{\\pi(a|s)}=\\sum_a \\pi^{\\*}(a|s)\\log \\pi^{\\*}(a|s) + \\sum_a \\pi^{\\*}(a|s)\\log \\frac{1}{\\pi(a|s)}$. \n\nSince only $\\sum_a \\pi^{\\*}(a|s)\\log \\frac{1}{\\pi(a|s)}$ is related to our policy $\\pi$, minimizing the KL is equivalent to minimizing $\\sum_a \\pi^{\\*}(a|s)\\log \\frac{1}{\\pi(a|s)}$. Lastly, the log-likelihood on the hint, $\\log \\frac{1}{\\pi(a|s)}$ with $a\\sim \\pi^{\\*}(\\cdot|s)$, is an unbiased estimator of $\\sum_a \\pi^{\\*}(a|s)\\log \\frac{1}{\\pi(a|s)}$.\n\n### Performance gain\n\nThe objective function of UFT is equivalent to that of RFT (GRPO) when $\\sum KL(\\pi^{\\*}||\\pi)$ is removed (c.f. Equation (3.1) and (3.2)). The only difference after removing $\\sum KL(\\pi^{\\*}||\\pi)$ is that UFT starts reasoning from the hint, while RFT starts from scratch.\n\nIn Figure 5, we did an ablation study on using hint but **with/without** the modified objective function (the additional log-likelihood on hint, equivalent to the KL as stated above). It shows that the additional log-likelihood contributes to the final performance gain and helps the model to learn new knowledge.\n\n### Implication of theory.\n\nModeling LLM text generation as a uniform tree (each node representing the generation of the next token) is a standard and widely used abstraction in prior work [1][2].\n\nFor LLM text generation, if we take each branch as the next token, then the whole text generation forms a path in the tree. Since the token table is fixed for LLMs, the branching factor $B$ is the same for all nodes. In other words, the search tree is uniform.\n\nFinally, our theoretical results serve to justify that integrating the supervised signal during RL helps improve sample complexity, as noted by Reviewer N2ms. While our analysis assumes uniform trees, the results can extend to imbalanced trees with minor adjustments. For instance, the lower bound in Theorem 4.2 can be understood more generally as requiring exploration of at least $\\frac{\\text{Number of Leaves}}{4}$ to achieve 50\\% accuracy.\n\n### $\\beta$ value\n\nWe acknowledge the reviewer’s request for a more detailed explanation regarding $\\beta$. The formal version of Theorem 4.3, provided in Appendix E, shows that $\\beta$ should satisfy: $\\beta\\leq O(\\frac{\\Delta}{(H+1)^2\\log B})$.\n\nThis is a conservative upper bound that holds under worst-case for arbitrary reference policies $\\pi^{ref}$. A tighter (and more practical) condition depends on the KL divergence between the and the reference policy, which is $O(\\frac{\\Delta}{(H+1) KL(\\pi||\\pi^{ref})})$.\n\nIn practice, due to pretraining, $\\pi^{ref}$ might be close to $\\pi^{\\*}$, making it feasible to choose a larger $\\beta$. In our experiments, we choose $\\beta=0.001$, which satisfies the bound when $\\pi^{ref}$ is not far from $\\pi^{\\*}$.\n\n### References\n\n[1] Xie Y, Goyal A, Zheng W, et al. Monte carlo tree search boosts reasoning via iterative preference learning[J]. arXiv preprint arXiv:2405.00451, 2024.\n\n[2] Chen G, Liao M, Li C, et al. Alphamath almost zero: process supervision without process[J]. Advances in Neural Information Processing Systems, 2024, 37: 27689-27724."}, {"review_id": "VWSy11cLKe", "reviewer": "Reviewer_N2ms", "summary": "This paper proposes Unified Fine-Tuning (UFT), a novel post-training paradigm that combines traditional supervised fine-tuning (SFT) with reinforcement fine-tuning (RFT) in a single integrated process. The authors argue that SFT (learning from correct solutions) can lead to memorization/overfitting in large models, while RFT (learning via trial-and-error reward feedback) improves generalization but struggles when the base model is weak or rewards are sparse. UFT addresses these issues by blending supervised signals with reinforcement learning: during training, the model is guided with partial solutions (“hints”) to explore correct reasoning paths more frequently, and a hybrid objective optimizes both the reward of final answers and the log-likelihood of these hints. As training progresses, the hint length is gradually annealed to zero, smoothly transitioning from a fully supervised regime to pure reinforcement learning. The paper provides a theoretical analysis showing that UFT can break the exponential sample complexity bottleneck of standard RFT, achieving polynomial sample complexity in the length of the reasoning horizon. Empirically, UFT is evaluated on various reasoning-intensive tasks (arithmetic puzzles, math word problems, logic puzzles) across multiple model scales (0.5B to 3B parameters). The results demonstrate that UFT consistently outperforms both SFT and RFT alone, effectively obtaining the best of both: for smaller models UFT performs on par with SFT (which was superior to RFT in that regime), and for larger models UFT matches or exceeds RFT’s performance (which was superior to SFT for stronger models). Overall, the paper introduces a unified training framework that improves the reasoning capabilities of LLMs by combining the strengths of instruction-based supervised learning and reward-driven reinforcement learning in a principled way.", "questions": "1. How would UFT apply to open-ended tasks like dialogue or preference alignment, where step-by-step solutions aren’t available? Could model-generated hints or proxy signals be used? Clarifying this would help assess UFT’s broader applicability.\n\n2. How critical are the hybrid loss and cosine hint annealing schedule? Would performance drop if we removed the supervised loss on hints or used a simpler schedule? Any ablation results would help clarify the importance of these components.\n\n3. Can you clarify the assumptions behind Theorem 4.3? For instance, how realistic is the reward gap assumption, and do you use partial rewards in practice? Also, could UFT fail if the model becomes too dependent on hints?\n\n4. How does UFT’s training cost compare to standard SFT or RFT? Does hint usage significantly increase sequence length or training time? Did UFT reach performance targets in fewer steps, as theory predicts? Any discussion on scalability would be useful.", "limitations": "Partially discussed. UFT depends on high-quality intermediate steps, which may not exist in open-ended tasks. Broader applicability to instruction tuning or RLHF tasks is not demonstrated.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Quality:\n\nStrength: The paper’s technical quality is generally high. It proposes a clear training algorithm that integrates supervised and reinforcement signals, and backs it up with both theoretical guarantees and extensive experiments. The theoretical contribution is notable: the authors prove that in a long-horizon reasoning setting, pure RFT requires an exponential number of explorations in the worst case, whereas the UFT approach achieves a polynomial sample complexity – an exponential improvement in efficiency. This analysis (with formal theorems and proofs in the appendix) strengthens confidence that UFT’s performance gains are not just empirical luck but rooted in principle. The experimental evaluation is thorough: models of two different families (Qwen2.5 and Llama 3.2) and multiple sizes (0.5B, 1.5B, 3B) are fine-tuned on three distinct tasks. Results are consistently in favor of UFT, and the paper includes informative figures (e.g. training curves, accuracy vs. steps) illustrating how UFT avoids the pitfalls of each separate method (e.g. overcoming RFT’s stagnation on small models and SFT’s overfitting on large models). \n\nWeaknesses: While the experiments support the claims, the scope of tasks and model scales is somewhat limited. The tasks (math puzzles, logic riddles) are all domains with a known ground-truth answer and a well-defined reward signal (answer correctness). It remains unclear how UFT would perform on more open-ended tasks or typical language model alignment settings (e.g. dialogue with human preference rewards), where “instruction tuning” usually refers to aligning with human instructions rather than solving puzzles. Additionally, the largest models tested are 3B parameters, which are relatively small by modern LLM standards – it would strengthen the paper to see if UFT’s benefits hold for truly large models (e.g. 7B, 13B or beyond) and on more diverse, real-world benchmarks. Another quality concern is that the method’s practicality relies on having high-quality solution trajectories (hints) for supervision; the authors used human-annotated step-by-step solutions from the dataset, but in general, obtaining or generating such intermediate solutions might be non-trivial for arbitrary tasks. The paper would benefit from a discussion or experiment on how noise or inaccuracies in the provided hints affect performance. Despite these points, the overall execution is solid – the algorithms are described in detail, and the claims are well-supported by evidence.\n\nStrengths: The paper is clearly written and structured. The motivation is well-established with a relatable analogy (“learning by memorizing solutions vs. thinking through problems”), and background on SFT vs RFT is provided to orient the reader. The method is explained with both a conceptual figure and a concrete example of the UFT prompt format (including how the hint is inserted into the prompt), which makes the procedure easy to follow. Key components such as the hint-length annealing schedule and the combined loss function are described in a logical order. The inclusion of a “theoretical justification” section and an explicit list of contributions also aids clarity. \n\nWeaknesses: A minor clarity issue is that Algorithm 1, which outlines the UFT procedure, is not discussed in much detail in the main text. A brief high-level explanation of its steps—such as how hints are sampled and how loss terms are scheduled—would help readers.\n\nSignificance: \n\n Strengths: This work tackles a timely and important problem in the fine-tuning of LLMs. As models grow and tasks demand reasoning, the limitations of purely supervised or purely RL-based fine-tuning have become apparent. A method that successfully combines the strengths of both has significant practical implications – it could lead to more data-efficient and capability-enhanced model training pipelines. The empirical gains demonstrated by UFT (outperforming SFT and RFT across tasks and scales) suggest that the approach can improve model reasoning without requiring larger models or prohibitively many RL training samples. The fact that UFT adapts to model capability (favoring memorization for weaker models and exploration for stronger ones) is particularly compelling – it hints that this single paradigm could simplify the fine-tuning strategy for a wide range of model sizes, rather than needing very different approaches for small vs. large LMs. The theoretical result is also significant in that it formally confirms an intuition about RL in long-horizon tasks and shows that providing intermediate feedback (hints) can dramatically accelerate learning.\n\nWeakness: \nThe impact of UFT is somewhat limited by its reliance on curated intermediate solutions, which are more feasible for structured tasks like math but less available in broader domains like QA or dialogue. The chosen benchmarks, while suitable, are relatively narrow; showing gains on more widely-used or complex tasks would strengthen the significance. Still, the approach offers meaningful insights for improving LLM fine-tuning, with broader impact depending on future extensions.\n\nOriginality: \n\nStrengths: The idea of unifying supervised and reinforcement fine-tuning is novel in the context of LLM training. Prior work has typically run SFT and RL sequentially (e.g. fine-tune on instructions, then apply RLHF), or chosen one based on context, but merging them into one framework with a seamless transition is a fresh approach. The use of partial solutions as hints effectively bridges the gap between “imitating correct reasoning” and “exploring new reasoning”, which is an original way to tackle the trade-off. The paper also introduces a smooth hint annealing technique (using a binomially sampled hint length that decays via cosine schedule) to avoid distributional shock – this is a thoughtful improvement over earlier “stagewise” or uniform sampling strategies (citing Xi et al., 2024) and appears to stabilize training. The theoretical analysis provided is another aspect of originality: it’s the first time (to my knowledge) that a unified training regime is proven to improve sample complexity for reasoning tasks, giving a new theoretical insight into why combining methods is beneficial. \n\nWeaknesses: Some elements of UFT build on existing ideas, such as guiding RL with hints or mixing supervised and reinforcement learning, which makes the contribution more evolutionary than revolutionary. While the approach doesn’t introduce a fundamentally new paradigm, it offers a well-integrated and effective combination of known techniques. The novelty lies more in execution and analysis than in algorithmic invention, but the work remains sufficiently original for NeurIPS.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer N2ms\n\nThank you very much for your detailed and encouraging review. We greatly appreciate your recognition of both the theoretical and empirical contributions of UFT, as well as your thoughtful questions. Below, we address each of your points.\n\n### Model scale and generalization to larger LLMs\n\nWe fully agree that demonstrating UFT on larger models would further strengthen our claims. However, due to resource constraints (our current experiments already cost approximately $10,000), we were unable to scale up the experiments further.\n\nThat said, our current results reveal key trends that may generalize. For instance, on Qwen2.5-1.5B:\n\n| Task (Qwen2.5-1.5B)| RFT | UFT|\n|------------------|---------|-----|\n| Countdown        |     71.48\\% |71.48\\%  |\n| MATH(3,4,5)      |   46.68\\% | 47.95\\% |\n| Logic            |    0.00\\%  | 23.00\\%   |\n\nWe observe that RFT performs well on Countdown and MATH(3,4,5) but fails on Logic, where the model does not get enough knowledge through pretraining. UFT, by contrast, significantly outperforms RFT in this setting. We expect similar benefits on larger models when fine-tuning for extremely complex reasoning tasks or tasks requiring specific domain knowledge.\n\n### Use of model-generated hints\n\nUFT is not limited to human-annotated hints. It only requires:\n\n- A reasoning trace. It can either be human annotations or chain-of-thoughts generated by other models.\n- A reward signal (*e.g.*, a rule-based reward model).\n\nThe “hints” used in our work are not curated step-by-step annotations, but simply prefixes of existing solutions contained in the dataset. For each solution, we will divide it by sentence, and the hint length is the number of sentences. Here is an example from the MATH dataset:\n\n```python\n[\n\"For the piecewise function to be continuous, the cases must \\\"meet\\\" at $2$ and $-2$. \",\n\"For example, $ax+3$ and $x-5$ must be equal when $x=2$. \",\n\"This implies $a(2)+3=2-5$, which we solve to get $2a=-6 \\\\Rightarrow a=-3$. \",\n\"Similarly, $x-5$ and $2x-b$ must be equal when $x=-2$. \",\n\"Substituting, we get $-2-5=2(-2)-b$, which implies $b=3$. \",\n\"So $a+b=-3+3=\\\\boxed{0}$.\"\n]\n```\n\nTherefore, this approach is easily extendable to LLM-generated traces or even imperfect intermediate outputs.\n\nIn fact, prior work [3] has shown that LLM-generated explanations often outperform human-written ones for supervision. Recent work s1 [4] also utilized LLM-generated responses for fine-tuning. Hence, we expect UFT to perform even better when paired with high-quality model-generated responses.\n\n### Description of the algorithm\n\nThank you for highlighting this. We will update the main text to briefly explain Algorithm 1 at a high level, describing how hints are sampled and how the hybrid loss is scheduled during training.\n\n### Ablation study on cosine annealing and supervised loss\n\nFigure 5 already includes an ablation where RFT is combined with cosine-annealed hints but without the supervised log-likelihood loss. The drop in performance in this setting demonstrates that the supervised loss on hints plays a key role in UFT’s effectiveness.\n\n### Ablation on the hint schedule\n\nIn Figure 5, we can see that $R^3$ (the hint length is sampled from a uniform distribution throughout the training) performs worse than RFT with cosine-annealed hints. Both of them adopt the standard RFT objective (3.1) without the log-likelihood on hints. This result also justifies the necessity of gradually annealing the hint length.\n\nOn the other hand, the specific choice of hint length distribution is not important. To assess the sensitivity of UFT to this choice, we also experimented with a two-point distribution: for any expected hint length $p\\cdot L\\in [n, n+1)$, we set $l=n$ with probability $n+1-p\\cdot L$ and $l=n+1$ with probability $p\\cdot L-n$. $p$ is still determined by the cosine-annealing scheduler. Using this scheme, the final accuracy of Qwen2.5-0.5B (Countdown) is:\n\n- Two-point: 48.63\\%\n- Binomial distribution: 50.39%\n\nThis suggests that the choice of distribution's impact on overall performance is relatively small.\n\n### Clarifications on Theorem 4.3\n\nFor any rule-based reward model, the assumption holds because there is always a gap between the reward for the correct answer and the incorrect answer. For instance, Deepseek R1 [1] and DAPO [2].\n\nRegarding “partial rewards,” if this refers to assigning intermediate rewards (e.g., for correct reasoning steps), we currently do not use partial rewards in our setup. All rewards are terminal (binary correctness). However, extending UFT to settings with partial rewards is a promising direction for future work.\n\n### Avoiding overreliance on hints\n\nIndeed, overreliance on hints is a real concern, and the cosine annealing strategy is specifically designed to mitigate this. As training progresses, the expected hint length smoothly decays, ensuring that the model learns to reason without external help, aligning training with the final evaluation distribution (no hints at inference time).\n\n### Computation time\n\nUFT is faster than RFT per training step. Here is the training time per step averaged across tasks:\n\n| Model | RFT | UFT |\n|-------|-----|-----|\n| Qwen2.5-0.5B | 63.62 | 57.18 | \n| Qwen2.5-1.5B | 98.25 | 89.21 | \n| Qwen2.5-3B | 127.14 | 118.36 | \n| Llama-3.2-1B | 96.61 | 74.63 | \n| Llama-3.2-3B | 126.90 | 108.18 | \n| Average | 102.50 | 89.51 |\n\nThis improvement stems from shorter generation lengths during the hint phase. We also report the rollout time per step below, showing clear gains:\n\n| Model | RFT | UFT |\n|-------|-----|-----|\n| Qwen2.5-0.5B | 26.75 | 20.47 | \n| Qwen2.5-1.5B | 41.34 | 32.40 | \n| Qwen2.5-3B | 63.98 | 55.35 | \n| Llama-3.2-1B | 52.30 | 32.13 | \n| Llama-3.2-3B | 65.22 | 48.28 | \n| Average | 49.92 | 37.72 |\n\nWhile UFT introduces a hybrid loss, this overhead is offset by a more efficient trajectory rollout procedure.\n\n### References\n\n[1] Guo D, Yang D, Zhang H, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning[J]. arXiv preprint arXiv:2501.12948, 2025.\n\n[2] Yu Q, Zhang Z, Zhu R, et al. Dapo: An open-source llm reinforcement learning system at scale[J]. arXiv preprint arXiv:2503.14476, 2025.\n\n[3] Ren X, Wu B, Liu L. I learn better if you speak my language: Understanding the superior performance of fine-tuning large language models with LLM-generated responses[J]. arXiv preprint arXiv:2402.11192, 2024.\n\n[4] Muennighoff N, Yang Z, Shi W, et al. s1: Simple test-time scaling[J]. arXiv preprint arXiv:2501.19393, 2025.\n\n[5] Agarwal A, Kakade S M, Lee J D, et al. On the theory of policy gradient methods: Optimality, approximation, and distribution shift[J]. Journal of Machine Learning Research, 2021, 22(98): 1-76.\n\n[6] Jin C, Liu Q, Wang Y, et al. V-Learning--A Simple, Efficient, Decentralized Algorithm for Multiagent RL[J]. arXiv preprint arXiv:2110.14555, 2021."}]

Author rebuttal:

Meta-review: **Summary**

This paper tackles the important challenge of effectively combining SFT and RL in LLM post-training. The core contribution is a new training scheme that applies SFT loss to initial tokens and RL loss to later tokens within a single generation. This transition is managed by an annealing schedule that gradually shifts focus from SFT to RL. The authors provide both empirical results and theoretical justification for their approach.

**Strengths**

The paper’s primary contribution is a principled method for integrating SFT and RL. The proposed method with annealing schedule provides a mechanism for a gradual transition from SFT to RL, addressing a known challenge in the field.

**Weaknesses**

The main weakness of the paper lies in the limited generalizability of its findings.
* **Empirical Scope:** The experimental validation is restricted to smaller-scale models (< 3B) on a narrow range of tasks. This makes it difficult to assess whether the observed benefits will transfer to the larger, more capable models, as performance trends at smaller scales do not always generalize predictably. 
* **Theoretical Assumptions:** The practical relevance of the theoretical guarantees is questionable, as some of the underlying assumptions do not align with standard practice. For example, key theorems rely on an extremely small $\beta$ value (lines 266, 516), a condition that is not met in typical LLM training configurations.
* **Problem Setting:** The method assumes that a ground-truth prefix (the initial tokens of a desired response) is available for every prompt to apply the SFT loss (line 181). This is a strong assumption that deviates from the standard RL setting for LLMs.

**Recommendation**

Despite the concerns regarding generalizability, the paper introduces an interesting method for a timely problem. It represents a notable attempt to better integrate SFT and RL, and its findings, while preliminary, could serve as a useful data point for future research in this direction.
Decision: Accept (poster)