SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
Authors: Yuqian Fu
Tinghong Chen
Jiajun Chai
Xihuai Wang
Songjun Tu
Guojun Yin
Wei Lin
Qichao Zhang
Yuanheng Zhu
Dongbin Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Reasoning
Reinforcement Learning
Supervised Reinforcement Fine-Tuning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose SRFT, an entropy-aware single-stage framework that unifies SFT and RL for LLM reasoning.

Abstract: Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet optimally integrating Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains a fundamental challenge. Through a comprehensive analysis of token distributions, learning dynamics, and integration mechanisms from an entropy-based perspective, we reveal key differences between these paradigms: SFT induces coarse-grained, global shifts to policy distributions, while RL performs fine-grained, selective optimizations.
Our analysis further establishes entropy as a critical indicator of training efficacy. 
Building on these observations, we introduce **S**upervised **R**einforcement **F**ine-**T**uning (**SRFT**), a single-stage framework that unifies both fine-tuning paradigms through entropy-aware weighting mechanisms. 
SRFT simultaneously applies SFT and RL to directly optimize LLMs using demonstrations and self-exploration rollouts rather than through two-stage sequential methods.
Extensive experiments show that SRFT outperforms zero-RL baselines by **9.0%** on five mathematical reasoning benchmarks and by **10.9%** on three out-of-distribution benchmarks.
Moreover, by leveraging demonstration data, SRFT maintains a more stable policy entropy, facilitating sustained policy improvement.

Published as a conference paper at ICLR 2026
SRFT: A SINGLE-STAGE METHOD WITH SUPERVISED AND REINFORCEMENT FINE-TUNING FOR REASONING
Yuqian Fu Tinghong Chen " Chai Xihuai Wang W Songjun Tu
Guojun Yin Wei Lin Qichao Zhang * Yuanheng Zhu Dongbin Zhao
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA
School of Artificial Intelligence, UCAS Meituan & SJTU
[fuyuqian2022, yuanheng wwweaia.aaaa ac.cn
ABSTRACT
Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet optimally integrating Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains fundamental challenge. Through a comprehensive anal- ysis of token distributions, learning dynamics, and integration mechanisms from entropy-based perspective, we reveal key differences between these paradigms: SFT induces coarse- ponsse-rrained global shifts to policy distributions, while RL performs fine-gree ane-grained selective optimizations. Our analysis further establishes entropy as a critical indicator of training efficacy. Building on these observations, we introduce Supervised Reinforcement Fine- Tuning (SRFT), single-stage framework that unifies both fine-tuning paradigms through entropy- aware weighting mechanisms. SRFT simultaneously applies SFT and RL to directly optimize LLMs using demon- strations and self-exploration rollouts rather than through two-stage sequential methods. Extensive experiments show that SRFT outperforms zero-RL baselines by 9.0% on five mathematical reasoning benchmarks and by 10.9% three out- of-distribution benchmarks. Moreover, by leveraging demonstration data, SRFT maintains more stable policy entropy, facilitating sustained policy improvement.
INTRODUCTION
Recent advances in Large Language Models (LLMs) for reasoning (OpenAI, 2025; Guo et al., 2025; Anthropic, 2025) have demonstrated remarkable capabilities in complex moblemm-solving tasks. Despite these remarkable achievements, fine-tuning strategies for enhancing reasoning capabilities remain an active area of research, presenting both opportunities and challenges.
Initial approaches often treat Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) as distinct, sequential phases. For instance, SFT might be applied for instruction-folloiing, followed by RL for policy optimization. However, this separation presents several challenges: SFT can lead to models that memorize patterns without developing true reasoning abilities, potentially overfitting to the training dataset (Chu et al., 2025; Chen et al., 2025a). Conversely, RL methods, while promising for reward optimization, can be sample-inefficient, struggle with effective exploration in vast solution spaces (Schmied et al., 2025), or suffer from issues such as mode collapse, where models repeatedly generate similar, suboptimal outputs (Cai al., 2025).
Instead of simple sequential approaches, recent work (Yan et al., 2025; Wu et al., 2025; Liu et al., 2025a;b; Chen al., 2025b; Zhang et al., 2025a) has shown movement towards integrated frame- works that unify SFT and RL paradigms, or dynamically switch between the two fine-tuning methods during the training process. As shown in Figure 1(a), SFT guides the LLM policy toward demonstra- tion distributions, while RL enables the policy to explore improved solutions in the neighborhood of the base policy distribution. Our illustration demonstrates a key limitation: when the base policy is positioned near suboptimal policy, the RL online rollouts alone cannot effectively navigate to
'Work was done during an internship at Meituan. 'Corresponding authors. I This work was supported in part by the Strategic Priority Research Program of Chinese Academy of Sciences under Grant XDA0480302, in part by National Natural Science Foundation of China under Grants 62136000 and 62293541, and in part by Beijing Nova Program under Grant 20240484514 and Beijing Natural Science Foundation QY25186

---

Published as a conference paper at ICLR 2026
FP Femonstarto
THI Explee
Teatore Neight hortol
(or, yr. Mi., mn.le-a....
Rollour Buffer Prett
Demonstration Di
Policy Model
Reward Model
SFT&RL
Reugaroe
Seugaros
Update Policy Model πT with Entropy Indicator (2)
SXFT Palicy
Paicy
Rollout(s)
Poliyy
Re
Revard
Wneel
startyy
Lie deteouteess
(a) Toy illustration SFT, RI, and SRFT for LLM (b) Framework of SRFT. Our method leverages demon- strations to improve reasoning capabilities.
reasoning training on a single prompt.
Figure I: Overview of SRFT's motivation and framework.
the optimal policy. Beyond applying SFT and RL individually, unified integration of SFT and RL within single-stage method (e.g., our proposed method SRFT) enables the LLM policy directly optimize toward better solutions across an expanded space. However, critical challenge remains in determining the balance between SFT's knowledge distillation and RL's policy optimization: on the one hand, insufficient integration risks error propagation and limits RL improvements; on the other hand, excessive reliance on demonstrations leads to overfitting that constrains exploration beyond the base policy distribution. This trade-off creates uncertainty for practitioners when deciding between SFT, which leverages demonstration data, and RL, which supports online policy exploration.
To address these issues, in this work, we study how build single-stage LLM fine-tuning algorithms that are not only effective for LLM reasoning from SFT datasets but also well-suited to continuous improvement with RL rollouts. We conduct comprehensive analysis of the roles that SFT and RL play in fine-tuning LLM reasoning. Through our analysis in Sec. 3, we obtain the following key findings that guide subsequent algorithm design:
Key Findings
Policy distribution effects (Secs. 3.1.1 and 3.1.2): SFT induces coarse- sounse-grained global changes to the policy distribution, while RL performs fine-grained selective modifications.
Single-stage optimization (Secs. 3.1.2 and 3.2.2): Single-stage integration of SFT and RL enables direct optimization for reasoning capabilities and achieves superior training efficiency compared to sequential SFT-RL approaches.
Entropy as an indicator (Sec. 3.2.1): Entropy dynamics reveal the underlying mechanisms of training processes, enabling balanced weighting between the two paradigms.
Based these insights, we propose Supervised Reinforcement Fine-Tuning (SRFT), single-stage method for LLM reasoning. As shown in Figure 1(b), we integrate SFT into RL and use entropy dynamic indicator to control the balance between these paradigms. Specifically, for samples generated by LLM policy rollouts, we employ different RL training losses based on whether sample rewards are positive or negative. For samples from demonstration datasets, we simultaneously apply both SFT and RI. objectives. This unified approach enables sample-efficient learning from demonstrations at multiple granularities while bridging the complementary strengths SFT and RI. We evaluate our method on five comperition-level mathematical reasoning benchmarks and three -i----i-ittiiutiinn benchmarks. SRFT achieves an accuracy 59.1% based on Qwen-2.5- 2-h-Math- 7B (Yang et al., 2024). Moreover, SRFT demonstrates superior generalization capability, achieving an average improvement of over 4.7% compared to other demonstration based methods.
Overall, our key contributions
We conduct comprehensive analysis of SFT and RL in LLM reasoning, examining their differential effects on I.M policy distributions and learning dynamics. We further analyze the integration of SFT and RL. through an entropy- -aase perspective.
We propose SRFT, a single-stage fine-tuning approach that combines supervised fine-tuning and reinforcement learning with entropy-award weighting mechanisms, enabling effective utilization of demonstrations while maintaining stable exploration dynamics.
We demonstrate SRFT's superior performance across eight challenging benchmarks, achiev- ing substantial improvements of 9.0% and 10.9% over Re baselines on mathematical reasoning and out-of-distribution tasks, respectively.
2

---

Published as a conference paper at ICLR 2026
Queetae: Ewery nonting Aan porr for 1etoboneeiiinggg 1o ank and expe at ccitinn shop aferacide When the wation an conetur connt klonstrro
her
val tskee her 2 hour und 24 minuaaa roluding SI5 renviia isanin the colfee shop Duppeas Ape wallo at Sart eic mark per hour. spent
uding
Lower Proteduilty After Paro tarking
theeker Protuct
Probutaikty by After Plas turning
SFT
0.7
Bitl cating cilene
0.5
sang da swil
0.4
Probability Change
0.2
pifirii Thi vill Eat'
nour raking sounn
0.0
lrnk 2 the 2-
-0.2
-0.4
RL
-0.5
mndtcc trigtty INEE (BE))
0.7
SFT
cilli 11.0r2 2.01 Sarrg
-0.9
AL
20
40
60
80
100
Percentage (%)
a
(b)
Figure 2: Visualization of LLM distribution changes during fine-tuning. (a) Heatmap visualization comparing responses generated by fine-tuned and base models, where darker background colors indicate larger probability changes. (b) Distribution of token probability shifts across five benchmarks.
2 PRELIMINARIES
Supervised Fine- Line-Tuning (SFT) is a standard approach for adapting pre-trained language models to specific downstream tasks. Given dataset D {(xr; y))), where æ;; is an input prompt and y; is the corresponding target response generated by the behavior policy TH, the objective is to train the policy my maximize the conditional probability of generating the target response y, given æy. This is typically achieved by minimizing the negative log-likelihood over the dataset:
LSFT (0) = E [( log ne (y)[))]
Reinforcement Learning (RL) is typically applied after SFT to further align LLMs with complex behaviors that are challenging to specify through static datasets. To optimize the LLM policy, Group Relative Policy Optimization (GRPO) (Shao et al., 2024) offers a memory-efficient variant of Proximal Policy Optimization (PPO) (Schulman et al., 2017). A key characteristic is that GRPO typically operates without a learned value function. Instead, for a given prompt æ, it often generates a group of G responses (y1,.. ... yG} } using the current policy. The rewards (R(z, (1),,,)),... 91),..., R(æ, yG)) for these responses are then used to compute relative ad advantage for each response:
A R(x, yk) mean { R(x, æ y 1/4 = 1,2,....
(2)
std (1., yk) k 1,2..... G}
Then, GRPO maximizes a clipped surrogate objective function to ensure stable updates. The objective function for GRPO can then be expressed as:
G
vvl min (TE.(() -e,, (1), An).
TaRPO (0)
(3)
G
ykl
k=
where denotes the clipping range, Tew represents the policy before the update, and (0)
Toe =lr is the importance sampling ratio for each token Ik., in response yı of length lyal
Te e (e)))
3

---

Published as a conference paper at ICLR 2026
3 ANALYSIS OF SFT AND RL IN LLM REASONING
In this section, we provide a comprehensive analysis of the roles of Supervised Fine- Tuning (SFT) and Reinforcement Learning (RL) in LLM reasoning. We first examine their differential effects token distributions (Sec. 3. and learning dynamics (Sec. 3.1.2), then investigate the integration mechanisms through entropy perspective (Sec. 3.2).
3.1 SFT AND RL EFFECTS ON LLLSs: SLEDGEHAMMER VS. SCALPEL.
3.1.1 EFFECTS ON TOKEN DISTRIBUTIONS
To understand the differential impact of SFT and RL on reasoning. visualize token-level probability changes for identical prompts, measured before and after fine-tuning the same base model. As illustrated in Figure 2(a), the results reveal fundamental difference: SFT substantially reshapes the probability distribution across the entire response sequence, whereas RL primarily adjusts the probabilities of small subset of tokens, leaving most numerical content and formal proof statements largely unchanged. We further quantify these distribution shifts across five benchmarks, comprising approximately 600,000 token probability samples, as shown Figure 2(b). The results demonstrate that SFT induces more pronounced changes in policy distributions compared to RL. Specifically, token probability shifts from RL are concentrated near zero, whereas SFT leads to changes significantly larger magnitude. The details of visualization and further results for other models are provided in Appendices B.1 and I.
From theoretical perspective, this behavior can be understood through the gradient of the SFT objective function:
lvll
VoLSTT (
(To (r y<t) - 1 b Vo log To (vl v.))))))))))))).))))))))))
where V denotes the LLM vocabulary, and 1,=y, is an indicator function. This formulation reveals that SFT systematically sharpens the model distribution by increasing probabilities for target tokens while decreasing probabilities for all other tokens in the vocabulary, leading to more deterministic outputs. The detailed experiment setting and derivation of Eq. (4) provided in Appendix B.
3.1.2 VISUALIZATION OF LEARNING DYNAMICS
Beyond token-level probability analysis, we investigate the training paradigms through the lens of learning dynam- ics. Since directly characterizing the LLM feature space computationally intractable, we introduce a visualiza- tion approach that characterizes the evolution of learning dynamics by using the Kullback-Leibler (KL) divergence in policy distribution between group of existing refer- ence models πR and our fine-tuned model My. Specifically, we introduce three reference models: Powe2.5-Matth- DeepSeek-RR R1, and QwQ-32B. Since it is still challenging to give an exact KL divergence computation over large- scale distributions, we use cross-entropy for its estimation:
SFT
RL
SFT-RLL
SAFT
54.3
68.1
493
owa Dicunce
d H ( no) DKI. (TRllT))) + H (TR), (5)
Owwen2.5AMath-T0 Distance
Start FT
X Finish FT
where dn( denotes the distance between the fine-tuned model no and the reference model иn- H (TTn mo) is the cross-entropy, and (TT) is the entropy of the reference model, which is constant for given reference model. Therefore, (dR,, dR dR,)) T serves as coordinate repre- senting the policy Ty with the 3-dimensional reference space. As illustrated in Figure 3, all fine-tuning methods improve performance while diverging from the distribution the base model (Qwen2.5-4 100.5---athBBB- 7B). Notably, SFT induces larger distributional shifts from the initial policy than RL, achieving +5.0 points performance improvement. This result
Figure 3: Learning dynamics visualiza- tion of different fine-tuning paradigms in probability space. The (number) denotes the final performance of each method.
4

---

Published as a conference paper at ICLR 2026
Table I:
Performance comparison of SFT and RL integration strategies across multiple benchmarks.
Bold and underlined indicate the best and second-best performance, respectively.
Model
AIME24
AMC
MATH150
Minerva
Olympiad
Avg.
11.4
32.6
48.8
8.7
15.8
23.5
SFT
21.2
53.2
83.0
37.1
42.2
47.3
21.2
59.3
83.6
36.4
46.6
49.4
RL-+SFT
10.5
40.4
73.6
32.0
30.7
37.4
RL RL-ISTTKI.
13.1
45.2
70.2
26.5
36.3
38.3
SPT-RL
24.5
39.3
86.4
39.3
53.1
$2.5
further validates our findings in Sec. 3.1.1, confirming that SFT drives broader changes, whereas RL fine-tuning tends to remain close to the initial policy. Furthermore, during the SFT-RL pipeline, the optimization direction in the second-stage RL opposes that of SFT. This result suggests that SFT may overfit to demonstrations, and the subsequent RI. attempts to correct this deviation, thereby increasing the learning tax the two-stage paradigm. By contrast, our single-stage method, SRFT, follows more straightforward training trajectory in probability space, enabling more efficient optimization than sequential fine-tune min-ttning approaches. More technical details for this visualization and analysis provided in Appendix C.
3.2
INTEGRATION OF SFT AND RL: FROM TWO-STAGE o TO SINGLE-STAGE R
3.2.
SEQUENTIAL INTEGRATION ANALYSIS
In this section, we examine the in- tegration of SFT and RL through lens of entropy dynamics to clarify their complementary roles LLM fine-tuning We begin with analyzing two sequential inte- gration approaches: SFT-RLL and RL-SFT, shown in Figure 4. As demonstrated in Table 1 and Figure 4(a), applying SFT after RL consistently yields suboptimal per- formance across all benchmarks. To mitigate the policy shifts in- duced by RL- we introduce KI. divergence constraint (SFT detailed in Appendix G). However, even with this constraint, the performance improvements remained limited, suggesting fundamental incompatibility in this paradigm. In contrast, existing methods successfully achieve substantial performance gains through RL when applied after the base model SFT, as evidenced in Table 1. This different result reveals that the sequence of fine-tuning paradigms critically affects the final model performance, motivating our entropy-based analysis to understand the underlying mechanisms. To understand this difference, we analyze the training dynamics of SFT and RL from entropy perspective. As illustrated in Figure 4(b), policies after RL exhibit significantly lower entropy, approaching deterministic outputs. However, the distribution shift introduced by subsequent SFT causes rapid increase in entropy. corresponding to the sharp performance drop at step 20 in Figure 4(a), followed by gradual decline. Moreover, models after RL. demonstrate limited capacity for additional learning through SFT, as evidenced by the entropy plateau occurring after approximately 90 training steps in Figure 4(b). In contrast, base models undergoing SFT exhibit a brief initial entropy increase followed by a sustained decrease, ultimately yielding performance improvements. This distinct entropy trajectory suggests that while RL effectively enhances LLM performance, it simultnneuusly reduces the model's plasticity (its capacity to adapt through subsequent training). These findings establish entropy crucial indicator for effective SET and RL integration. More discussion about the role of entropy in LLM training be found in Appendix E.
80
Emmooy
82-
RL-GIT
01
1750
60
(a) Reasoning performance.
(b) Entropy dynamics.
Figure Comparison between SFT and RI.-SFT.
5

---

Published as a conference paper at ICLR 2026
3.2.2 SINGLE-STACE INTEGRATION ANALYSIS
Building upon the preceding analysis, we show that the SFT-RL paradigm is better suited for LLM reasoning than RL-SFT. Beyond these sequential approaches, we further investigate single-stage approach that directly unifies both paradigms (SFT+RL), with combined ob- jective CSFT+RL LSFT + LRL. We conduct preliminary experiments comparing pure RL, sequential SFT-RL with varying SFT steps, and the single-stage SFT+RL, as illustrated in Figure 5. Our empirical results reveal that the single-stage SFT+RL method achieves superior training efficiency compared with both the sequential SFT-RL and pure RL. We attribute this behavior to principal factors: First, the SFT dataset derived from other models' responses may not consistently represent optimal solutions, even when curated from high-quality demon- strations, therehy risking overfitting to suboptimal policies. Second, pure RI. is typically data-inefficient because it cannot exploit existing demonstrations that would guide exploration toward near-optimal solutions. In contrast, the single-stage SFT+RL method effectively leverages demonstrations through unified optimization. This approach enables direct policy optimization toward the target objective while preserving the knowledge distillation benefits of supervised learning from demonstrations.
50
Accuracy
BFT 190 steps
Pure RIL
SFT+FL
RL from SFT150
8FT360
360
100
200
400
Step
Figure 5: Comparison across pure RL, sequential SFT-RL, and single-stage SFT+RL integration approaches.
METHOD
In this section, we present the Supervised and Reinforcement Fine-tuning (SRFT), which integrates the advantages of SFT and RL in single-stage approach. Building upon the RL framework described in Sec. 2, SRFT incorporates flexible guidance from demonstrations, enabling to harness the strengths of both fine-tuning paradigms. The core innovation of SRFT lies in its single-stage learning mechanism: goarse-grained behavior policy approximation through SFT and fine- tre-grained policy improvement through RL, both applied to demonstration data and self-rollout data.
4.1 LEARNING FROM DEMONSTRATIONS
Given a dataset containing demonstrations Deemo (e.g., reasoning responses generated by expert LLMs), SRFT employs a dual suan-ee strategy to harness this valuable resource. First, we leverage SFT to perform coarse- coarse-grained approximation of the behavior policy underlying the expert's responses. The behavior policy T a)) captures the underlying generation patterns that produced these high-quality responses, which we seek to approximate through supervised learning: LSpr ((m.y)) log To ( [=)] Second, we adopt off- off-policy RL approach similar to LUFFY (Yan et al., 2025) to perform fine one-grained learning of the behavior policy. Specifically, we directly augment the LLM's rollout data with demonstrations, creating a heterogeneous training group:
Gang. ( y:) ).))l
GGmall U (10j )=
(6)
where Groll. denotes the on-policy rollout group, Gaemo. denotes the off- demonstration group. The advantage estimation for the entire group is given by:
Å r(z, yv.) mean (r (r(x yk ] 1, ... (aag.
std(r((x, yk ) = 1, 2. Gaug. }
Since responses generated by experts typically yield higher rewards, this augmentation mechanism effectively increases the advantage estimates for the entire group, shown in Eq. (2), thereby promoting optimistic exploration of the LLM policy (Ciosek et al., 2019).
To address the distribution mismatch between behavior policies Ty of demonstrations and the training policy Ty. we implement two mitigation strategies: For SFT on demonstrations, our entropy analysis in Sec. 3.2 demonstrates that entropy serves erucial indicator for the effective

---

Published as a conference paper at ICLR 2026
integration of SFT and RL. Building this insight, introduce adaptive weighting mechanism that dynamically adjusts based on the current policy entropy. Specifically, we employ WSFT = 0.5 x (oop_gra((ep((<((rr)
as the SFT weight, where stop_grad(-) denotes the gradient- stopping operation. This entropy-aware mechanism ensures that when the policy exhibits high entropy (indicating uncertainty), the SFT loss exerts diminished influence updates, thereby mitigating performance degradation caused by distribution mismatch between policies:
Calem (0) = WSFT B) [- log To (y|2))]
(8)
For off-policy RL training, we introduce an importance sampling ratio similar GRPO (Shao et al., 2024) and PPO (Schulman et al., 2017) to account for distribution mismatch:
RL
(0)
EE (2..)) Domoo Doeoo
min (T)..()) As.
Ax]]
(9)
where (0) #o (yn. (cc))))))
na (ux (xx Following established practices in recent work (Yan al., 2025; Ma et al., 2025), we set the behavior policy á avoid tokenization complexities when aligning the behavior and training policies. This approach enables straightforward integration of off-the-shelf
datasets without requiring recomputation of behavior policy probabilities.
4.2 LEARNING FROM SELF-EXPLORATION
In addition to leveraging the demonstration data, SRFT enables the LLM policy to learn from its own exploration rollouts. Although traditional RL methods learn from both positive and negative samples generated during rollouts, we observe that under on on-policy RL with binary rewards {1, (1, the standard RL objective function can be naturally decomposed into two distinct components:
caelf-rollout BEDD D.geWDyyaaaa [R x. y) log (1y [x]]
RR
Eg D.y y ~o (m)) log mo (y [x]] D ~ere (-1wa) [lo (0 |=))],
(10)
Positive Sample ①
Negative Sample
where 'D denotes the RL training dataset, and y² and y represent the correct and incorrect responses, respectively. A critical insight emerges from this decomposition: the positive-sample objective ! is structurally analogous to supervised fine-tuning,
maximizes the likelihood of correct responses. However, these positive samples are generated on-policy by the current training policy rather than sourced from SFT datasets. distinguishing our approach from supervised learning paradigms. The reward component ( implements likelihood minimization, systematically decreasing the probability mass assigned to incorrect responses. This structural correspondence implies that learning from positive samples constitutes a coarse-grained optimization strategy that necessitates careful balance. Morcover, in contrast to learning from demonstrations, self-cxploration induces rapid entropy reduction as the model converges toward increasingly deterministic outputs, potentially compromising exploration capabilities. To mitigate this phenomenon and preserve training stability, inspired by our analysis in Sec. 3.1.1, we introduce specifically for the positive sample objective. This mechanism parallels our formulation in Eq. (8) but serves the purpose of maintaining the policy's exploration. The complete self-rollout objective is formulated as:
antropy-adaptivg weighting mechanism
WRI. = 0.1 * stop_grad(cxpp)
collft CRL
WRL. PawD.yt~<o(t log na (y [a)] + R.e --oo [log To
(x)) (11)
4.3 INTEGRATING DEMONSTRATIONS WITH SINGLE-STAGE APPROACH
ROLLOUTS IN A
By leveraging both demonstrations and self-exploration rollouts, SRFT effectively balances the coarse- grained adjustments of SFT with the fine-grained refinements of RL throughout the single-stage fine-tuning process. The total loss function combines all three components:
S (0)
demmo SPTT
(0) + RL (0) + LR
caetf-rollout (0).
(12)
7

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Training Datasets. We employ OpenRR-Matth466--8122 (Yan et al., 2025) as the training dataset for SRFT, accompanied by high-quality reasoning responses generated by DeepSeek-RI (Guo et al., 2025). The dataset undergoes filtering through Math- Verify to exclude instances with unverifiable answers or responses exceeding 8, 192 tokens length. This dataset serves multiple purposes in our framework: providing prompts for policy rollouts, ground-truth answers for reward computation, and high-quality demonstrations for SRFT. Additional dataset details are provided in Appendix H.
Evaluation. We conduct a comprehensive evaluation on widely-adopted mathematical reasoning benchmarks, including AIME24 (Li et al., 2024), AMC (Li et al., 2024), Minerva (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), and MATH500 (Hendrycks et al., 2021). For datasets with limited sample sizes (AIME24 and AMC), we report avg@ 32; for the remaining datasets, we adopt pass@I 1 as the evaluation criterion. We further assess the model's generalization ability three out-of-distribution benchmarks: ARC-C (Clark et al., 2018), GPQA-Diamond (Rein et al., 2024) (denoted GPQA-D), and MMLU-Pro (Wang et al., 2024).
Baseline Methods. We benchmark SRFT against the following baselines using Qwen2.5-Math as the base model. SFT methods: (1) SFT training on the same dataset; (2) SFT training with KL divergence constraints (SFTx1.) ) on the same dataset. RL methods: (3) RLGRPO (Shao et al., 2024), a simplified PPO variant trained on the same dataset; (4) Sinple-RL-Zero (Zeng et al., 2025), applying GRPO to 24k mathematical samples from GSM8K and MATH; (5) OpenReasonee Zerro (Hu et al., 2025a), a PPO-based approach trained on 129k multi source samples; (6) PRIME-Zere (Cui et al.. 2025a), conducting policy rollouts on 150k NuminaMath queries with implicit process rewards and final labels. SFT and RL methods: (7) SFT- sequential training with SFT on the same dataset followed by GRPO; (8) SFT+RL, simultancous training with SFT and RL on the same dataset; (9) ReLIFT (Ma et al., 2025), an approach that interleaves RL with online Fine- Sine-Tuning on the hardest questions; (10) LUFFY (Yan et al., 2025), mixed policy GRPO approach using the same dataset; (Wu et al., 2025), dynamically integrating structured external knowledge within the GRPO framework, trained 5.5k samples from the MATH dataset.
Implementation Details, Building on recent work (Yan et al., 2025; Cui et al., 2025a), we adopt Swen2.5-Math-7B (Yang et al., 2024) the base model. In SRFT, generate eight rollout trajectories per prompt with a maximum sequence length of 8, 192 tokens. All experiments are conducted for 500 training steps. The source code is available here, and the comprehensive experimental details are provided in Appendix G.
5.2 EXPERIMENTAL RESULTS
Reasoning Benchmark Performance. Our main results are shown in Table 2, where compare SRFT with zero-RL baselines, as well as direct SFT, SFT-RL, and SFT+RL methods. Across five challenging competition-level reasoning benchmarks, SRFT achieves an average score of 59.1, significantly outperforming existing RL methods by margin of +9.0 points over the best baseline. This clearly demonstrates the advantages of integrating demonstrations with self-exploration LLM reasoning. We also observe that SRFT achieves a +4.8 points improvement over SFT methods, indicating that the self celf-exploration component effectively refines the policy distribution learned from demonstrations. Compared to the SFT-RL and SFT+RL methods, SRFT achieves +3.4 and +7.2 points improvement, respectively. These results demonstrate that our single-stage design and entropy- aware weighting mechanism effectively balance the benefits of demonstrations and self-exploration. Additional results are provided in Appendix A.I.
cot-of-Distributtinn Generalization. Regarding out-of-distribution performance, the results in Table 2 show that SRFT also achieves an average score 62.5 and exceeds the best baseline by +4.7 points. These results highlight that SRFT effectively combines demonstrations with self-exploration to mitigate the potential overfitting associated with SFT, thereby improving generalization.
Training Dynamics. Figure 6 presents the training dynamics of SRFT, including training rewards, response lengths, and training entropy. The results demonstrate that SRFT achieves more rapid improvement than RL, with both approaches exhibiting monotonically increasing training rewards.
8

---

Published as a conference paper at ICLR 2026
Table 2: Overall performance. Bold and underlined indicate the best and second-best performance.
D--Ditribution Performance
Out-of-Distrhutton Performance
AIME24
AMC
MATH500
Minerva
Olympind
Arg.
ARC-C
GPOA-D
MMLU-Pr
Avg.
Qwan2.5-htth-78
32.6
48.8
13.8
18.2
169
-rrrh25-Math-TB-Baeee 7B-Instruct
48.3
81.2
33.1
29.8
70.3
24.5
34,1
43.0
Supervised Fine-Iuning
SPTxt
13.0
45.2
70.2
26.5
36.3
38.2
33.3
22.2
30
26.6
Reinforcement Learning
RLearo
75.6
SimpleR1 (Zong 2025)
27.0
54.9
76.0
25.0
34.7
43.5
30.2
23.2
34.5
29.3
OpenRennoner-Zer (llu al., 2025a)
16.5
52.1
82.4
33.1
47.1
46.2
66.2
29.8
58.7
$1.6
PRIME-Zaro" (Cul 2015aa
54.0
81.4
39.0
40.3
73.3
32.7
41.4
Out-Zero' (Zang et all, 2025)
33.4
61.2
78.0
34.6
43.4
50.1
70.1
23.7
41.7
45.2
SFT and
SPT-RL
32.5
67.1
84.2
54.6
54.5
76.4
37.9
49.6
54.6
SPT+RI.
26.6
69.9
83.7
35.8
51.4
.2.3
74.2
34.1
44.5
50.9
LUFFY"
2025)
52.2
TAPO" (Wo et al, 2025)
33.3
77.5
83.4
38.2
46.2
55.7
81.6
37.9
49.6
56.4
ReLIFT (Ma et al. 2025)
28.2
64.8
85.0
37.1
54.9
54.0
74.9
40.9
51.9
SRFT (ours)
35.3
24,3
89.8
39.7
58.3
853
46.4
55.9
62.5
This methol's performance laken from the corresponding puper.
0.6
Outsomn Rewerds
0.
Resiponnel Langgh
c4.
Wir
Entrpyy
2000
1500
SRTT
1000
200
6000
500
200
300
450
500
100
200
300
500
Step
Step
Figure 6: Training dynamics during SRFT and RL training, including training rewards (left), response lengths (middle), and training entropy (right).
This pattern indicates that SRFT benefits from the sample efficiency provided by learning from demonstrations. Regarding response length, when confronted with challenging training data, RL exhibits a tendency toward generating more concise responses, whereas SRFT shows a progressive lengthening of responses, indicating the development of more comprehensive and detailed reasoning processes. From an entropy perspective, SRFT maintains relatively stable entropy throughout training, contrasting with RL's rapid entropy collapse. This stability indicates that the policy maintains exploration during training, validating the effectiveness of the entropy-aware weighting mechanism. The dynamics of the sequence mean for weights WSFT and WRL are visualized in Appendix A.2.
Ablation Study. We conduct a comprehensive ablation study to evaluate the impact of the entropy- aware weighting mechanisms and the necessity of each training component. To validate the effective- ness of our adaptive weighting mechanism, we first conduct an ablation study on WSFT and WRL- We substitute these entropy-aware dynamic weights with fixed constants. specifically ÜSFT 0.1 and WRI = 1.0. As detailed in Table 3, employing fixed SFT weight causes a substantial performance decline to 55.1, while a fixed RI. weight reduces accuracy 56.2. These outcomes corroborate our analysis in Sec. 3, confirming that using entropy dynamic indicator significantly outper- forms static weighting schemes by effectively balancing exploration and exploitation. Second, we investigate the contribution of specific loss components by selectively removing the SFT loss demonstrations (w/o demogrr) and the off- off-policy RL loss on demonstrations (w/o demoRL). Note that since SRFT is fundamentally built upon the RL framework, the self-rollout component (ollof-rollont) consistently retained. We observe that eliminating the demonstration SFT yields the most severe degradation, underscoring the critical role of grainne supervised guidance. Similarly, removing

---

Published as a conference paper at ICLR 2026
Table 3:
Ablation study weighting mechanisms and loss components. denotes using fixed
weights, and w/o denotes removing the specific loss component.
Method
AIME24
AMC
MATH500
Minerva
Olympiad
Avg.
W/ WSFT
30.1
65.8
87.0
36.8
55.8
55.1
w/ WRL
32.6
67.2
87.5
37.4
56.6
56.2
w/o demoSrT
29.5
65.2
85.6
35.9
53.1
53,9
w/o demoRL
30.8
68.0
86.4
36.2
54.5
55.2
SRFT
35.3
74.3
89.8
39.7
58.3
59.5
the demonstration RL component lowers performance to 55.2. Collectively, these results demonstrate that all components essential for SRFT, jointly contributing to the final reasoning capability.
RELATED WORK
Reinforcement Learning for LLM Reasoning. Recent RL approaches (Shao et al., 2024; Guo et al., 2025; Yu et al., 2025; Liu et al., 2025c) have demonstrated substantial improvements mathematical reasoning. However, the mechanisms through which RL enhances reasoning remain incompletely understood (Yue et al., 2025a; Wang et al., 2025a; Liu et al., 2025d). In this work, we introduce stagl method, SRFT, that integrates SFT with RL to maintain stable entropy throughout training while delivering sustained performance improvements.
Integrating Supervised Fine-Tuning and Reinforcement Learning. The integration between SFT and RL represents a critical frontier in recent LLM research. While SFT establishes robust policy foundations, RL enables optimization beyond potentially suboptimal SFT trajectories (Cai et al., 2025; Chen et al., 2025a). Recent approaches integrate external supervision into RL frame- works to enhance sample efficiency: UFT (Liu et al., 2025a) merges SFT and RL using partial solution hints; LUFFY (Yan et al., 2025) augments on-policy learning with off-policy traces from stronger models; ReLIFT (Ma et al., 2025) interleaves RL with SFT on high-quality demonstra- tions; TAPO (Wu et al., 2025) incorporates external "thought patterns" for balanced exploration; SASR (Chen et al., 2025b) provides a theoretical framework unifying SFT and RL with adaptive blending; and CHORD (Zhang et al., 2025a) proposes adjust the weights of SFT and RL based on the training dynamics. Single-stage integration also mitigates catastrophic forgetting encountered in sequential SFT- transitions (Chen al., 2025b; Liu al., 2025b). In this work, first analyze the different roles of SFT and RL in fostering LLM reasoning capabilities. Motivated by this analysis, we introduce SRFT, single-stage method that dynamically integrates SFT and RL through an catropy--warr weighting mechanism. More detailed background is provided in Appendix F.
7 CONCLUSION
In this work, we investigate the integration of SFT and RL for LLM reasoning. Through compre- hensive analysis, we reveal that SFT performs coarse-grained global adjustments while RL conducts fine-grained selective optimization, with entropy serving as crucial indicator of training dynamics. Building on these insights, we propose SRFT, a single-stage approach that unifies both paradigms through entropy- weighting mechanisms. Extensive experiments demonstrate the effectiveness of SRFT, which achieves 59. 19 average accuracy and outperforms zero-RL baselines by 9.0% on rea- soning tasks and 10.9% OOD benchmarks. Future work could explore extending SRFT to agentic RL scenarios (Zhang et al., 2025b) and investigating its performance with imperfect demonstrations.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
OpenAl. Introducing OpenAl o3 and 04-mini, 2025. URL -ing-o3-and-o4-me
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-R1: Incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(808 (0600081):33336333 2025.
Anthropic. Introducing Claude 4, 2025. URL https:tts/// http://wwwwnnthrpp.ccoeeoeeeeeee
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning, 2025.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. SFT or RL? an early investigation into training R1-like reasoning large vision-language models.
arXiv preprint arXiv:2504, 1468, 2025a.
Thomas Schmied, Jörg Bornschein, Jordi Grau-Moya, Markus Wulfmeier, and Razvan Pascanu.
LLMs are greedy agents: Effects of RL fine-tuning on decision-making abilities. arXiv preprint arXiv:2504. 16078. 2025.
Hongyi James Cai, Junlin Wang, Xiaoyin Chen, and Bhuwan Dhingra. How much backtracking is enough? exploring the interplay of SFT and RL in enhancing LLM reasoning. arXiv preprint arXiv:2505. 242273,,, 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu Yu Cheng, and Yue Zhang. Learning to reason under off- off-po guidance. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025.
Jinyang Wu, Chonghua Liao, Mingkuan Feng, Shuai Zhang, Zhenggi Wen, Pengpeng, Shao, Huazhe Xu, and Jianhua Tao. Thought-augmented policy optimization: Bridging external guidance and internal capabilities. arXiv preprint urXiv:2505.1569) 2025.
Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. UFT:
Unifying supervised and reinforcement finettuning. In The Thirty-Nint Annual Conference on Neural Information Processing Systems, 2025a.
Yihao Liu, Shuocheng Li, Lang Cao, Yuhang Xie, Mengyu Zhou, Haoyu Dong, Xiaojun Ma, Shi Han, and Dongmei Zhang. SuperRL: Reinforcement learning with supervision to boost LLM reasoning. arXiv preprint ar axXiv:250.00109 01096, 2025b.
Jack Chen, Fazhong Liu, Naruto Liu, Yuhan Luo, Erqu Qin, Harry Zheng, Tian Dong, Haojin Zhu, Yan Meng, and Xiao Wang. Step-wise adaptive integration of supervised fine-tuning and reinforcement learning for task-specific LLMs. arXiv preprint arXiv:2505.1 2025b.
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy RL meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508. / /408, 2025a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvementt arXiv preprint 2rXiv:2409.12122 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint axXiv:2402.03300, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXivv 2017.
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2026
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al. Learning what reinforcement learning can't 't: Interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.075277 2025.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-38577 2022.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyí Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties ( (Volume I: Long Papers), 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03882 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprint wrXivv1803.054550 2018.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. GPQA: graduate-Ievel google-prood q&a benchmark. In First Conference on Language Modeling, 2024.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. MMLU-Pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eighl Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun MA, and Junxian He. SimpleRL- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. In Second Conference on Language Modeling, 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open- BpeenReessonr--Zeee An open source approach to scaling up reinforcement learning on the base model. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025a.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint 25Xiv:2502.01456, 2025a.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. In The Thirty-Ninte Annual Conference on Neural Information Processing Systems, 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding RI R1-zero-like training: A critical perspective. In Second Conference on Language Modeling, 2025c.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In The Thirty-Nine Ninth Annual Conference on Neural Information Processing Systems, 2025a.
12

---

Published as a conference paper at ICLR 2026
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with training example. In The Thirty-Nini Annual Conference Neural Information Processing Systems, 2025a.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language models. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025d.
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, al. The landscape of agentic reinforcement learning for LLMs: A survey. arXiv preprint arXiv:2509.02547 2025b.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint wr.iv:2407.217888 2024.
Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look progress in language model reasoning: Pitfalls and paths to reproducibilityy In Second Conference on Language Modeling, 2025.
Jian Hu, Mingjie Liu, Ximing Lu, Fang Wu, Zaid Harchaoui, Shizhe Diao, Yejin Choi, Pavlo Molchanov, Jun Yang. Jan Kautz, et al. BroRL: Scaling reinforcement learning via broadened exploration. arXiv preprint arXiv:2510.01 2025b.
Devvrtt Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling reinforcement learning compute for LLMs. arXiv preprint arXiv:2510.13786 2025.
Qwen Team. QwQ-32B: Embracing the power of reinforcement learning, March 2025. URL https://qwenlnm gi github. htt/blog//wq-32/
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RI's razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.0422 2025.
Hangzhan Jin, Sicheng Lv, Sifan Wu, and Mohammad Hamdaqa. RL is neither panacea nor mirage: Understanding supervised vs. reinforcement learning fine-tuning for LLMs. arXiv preprint arXiv:2508. 16546, 2025a.
Hangzhan Jin, Sitao Luan, Sicheng Lyu, Guillaume Rabusseau, Reihanch Rabbany, Doina Pre- cup, and Mohammad Hamdaqa. RL fine-tuning heals OOD forgetting in SFT. arXiv preprint arXiv:2509. 2025b.
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation language models: Learning from self- generated mistakes. In The twelfth international conference on learning representations, 2024.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint wwXiv:2505.22611 2025b.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14758, 2025.
Jiawei Wang, Jiacai Liu, Yuqian Fu, Yingru Li, Xintao Wang, Yuan Lin, Yu Yue, Lin Zhang, Yang Wang, and Ke Wang. Harnessing uncertainty: Entropy-modulated policy gradients for long- morioori LLM agents. arXiv preprint wwX.:22509000265.. 2025b.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. In The Thirry- Annual Conference on Neural Information Processing Systems, 2025c.
13

---

Published as a conference paper at ICLR 2026
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, al. Veritool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055 2025.
Kaixuan Xu, Jiajun Chai, Sicheng Li, Yuqian Fu, Yuanheng Zhu, and Dongbin Zhao. DipLLM: Fine-tuning LLM for strategic decision-maken in diplomacy. In Forty-second International Conference Machine Learning, 2025.
Jiajun Chai, Guojun Yin, Zekun Xu, Chuhuai Yue, Yi Jia, Siyu Xia, Xiaohan Wang. Jiwen Jiang, Xiaoguang Li, Chengqi Dong, et al. RLFactory: A plug-and-play reinforcement learning post- training framework for LLM multi-turn tool-use. arXiv preprint arXiv:2509.0699 2025a.
Yuqian Fu, Yuanheng Zhu, Jiajun Chai, Guojun Yin, Wei Lin, Qichao Zhang, and Dongbin Zhao. RLAE: Reinforcement ltariing-asssste ensemble for LLMs. In Proceedings of the 2025 Confer- ence on Empirical Methods in Natural Language Processing, pages 13463-13477, 2025.
Jiajun Chai, Sicheng Li, Yuqian Fu, Dongbin Zhao, and Yuanheng Zhu. Empowering L.M agents with zero-shot optimal decision-making through q-learning. In The Thirteenth International Conference on Learning Representations, 2025b.
Songjun Tu, Qichao Zhang, Jingho Sun, Yuqian Fu, Linjing Li, Xiangyuan Lan, Dongmei Jiang, Yaowei Wang, and Donghin Zhao. Perception-coosistenny multimodal large language models reasoning via caption-regularized policy optimization. arXiv preprint arXiv:2509.21844 2025.
Yu Yue, Yufeng Yuan, Qiying Yu. Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et VAPO: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05//18 2025b.
Hugging Face Open R1: A fully open reproduction of DeepSeek-RI, R1, January 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Zichen Liu, Changyu Chen, Xinyi Wan, Chao Du, Wee Sun Lee, and Min Lin. OAT: A research- friendly framework for LLM online alignment. /stps://githhe com/ hwtm/sallssg/aaaaa 2024.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi. Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath, 2024.
Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, et al. 2 olmo furious. In Second Conference on Language Modeling, 2025.
14

---

Published as a conference paper at ICLR 2026
Appendix
Additional Experimental Results and Analysis
16
A.1 Performance Comparison on Different Models and Benchmarks
16
A.2 Dynamics of the Entropy-aware Weights
17
A.3 Analysis of Annealing Demonstration Weights
17
A.4 Dynamics of SFT-Initialized RL and SRFT
18
B Token Distribution Analysis
19
B.1 Experimental Setting for Visualization
19
B.2 SFT Gradient Derivation
19
Details of Visualization of Learning Dynamics
20
C.I Theoretical Definition
20
C.2 Experimental Setup
C.3 Learning Dynamics Visualization with Different Epochs SFT
22
Difference between SRFT and On-policy Distillation
22
Discussion about Entropy in LLM Fine-Tuning for Reasoning
23
F
Background
24
F1 Reinforcement Learning
24
F.2 Reinforcement Learning for LLM Reasoning
24
F3 Integrating Supervised Fine-Tuning and Reinforcement Learning
................................
24
G Experimental Details
25
H Dataset Details
27
H.I Training Dataset
27
H.2 Evaluation Benchmarks
27
I Token Probability Visualization of Other Models
29
Token Probability Visualization of SRFT
30
Disclosure on the Use of LLMs
30
15

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
A.1 PERFORMANCE COMPARISON ON DIFFERENT MODELS AND BENCHMARKS
We further evaluate the performance of SRFT on Llama-3.1 1.18--311-BB (Grattafiori et al., 2024), Qwen2.5- Math- I.5B and Qwen2.5-7B 57--ntruutt models, as shown in Table A1. We adopt the same training settings as in the main experiment, including training data and hyperparameter configurations. The results show that our method achieves superior performance compared to baseline approaches across most benchmarks. This confirms the effectiveness and scalability of our approach across different model series and sizes.
Table AI: Performance comparison on mathematical reasoning benchmarks on Qwen2.5-Math-1.55
Base and Qwen2.5-78-Instmuct models. Bold indicates the best performance.
Model
AIME24
AMC23
MATH500
Minerva
Olympiad
Avg.
Qwen2.5-Math-1558
26.4
28.0
9.6
21.2
18.5
SFT
11.7
37.8
70.6
26.8
31.3
35.6
GRPO
11.8
40.2
61.8
26.8
32.0
34.5
LUFFY
14.3
43.3
74.2
26.5
39.9
39.6
SRFT
15.5
44.9
74.6
28.7
39.8
40.7
Qwen2.5-7B-Instee
11.7
43.8
71.8
30.9
40.4
39.7
SFT
36.0
68.6
21.3
31,1
33.0
GRPO
14.1
43.5
74.0
33.8
37.6
40.6
LUFFY
17.7
50.9
82.0
31.3
47.4
45.9
SRFT
16.6
51.0
82.2
34.6
46.9
46.3
Llama-3.1-8B
0.3
4.2
13.8
4.4
3.8
5.2
SFT
5.4
20.2
5.3
GRPO
9.4
23.4
17.6
6.1
11.4
LUFFY
1.9
13.5
39.0
15.1
9.6
15.8
SRFT
1,9
14.3
40.1
15.3
9.5
16.2
Besides, we further compare the performance of SRFT with other baselines the AIME25 dataset et al., 2024), shown in Table A2. We observe that SRFT achieves superior performance compared to baselines, outperforming SFT by + +1.3 points and GRPO by +7.1 points. This validates the effectiveness of our method in simultaneously leveraging the advantages of SFT and RL within single-stage framework for reasoning tasks.
Table A2: Performance Comparison on AIME25.
Model
AIME25
Qwen2.5-Math-77
4.9
Dwen2.5-Matt-7B--nsse
10.1
SFT
20.3
GRPO
14,5
SFT-
21.2
SFT+RL
15.6
SRFT
21.6
16

---

Published as a conference paper at ICLR 2026
Table A3: Performance comparison under the evaluation setting proposed by Hochlehnert al. (2025). We report the mean accuracy and standard deviation across tasks.
Model
AIME24
AMC
MATH-500
Minerva
Olympiad
Avg.
Qwen2.5-Math-77
20.7 3.8
5.7
64.3
17.3
0.5
37.5
RLaRRO
22.5 + 2.6
6.4
78.6 0.3
32.8 = 1.3
42.3 0.9
48.1
SRFT (Ours)
28.3
71.8 3.9
6..0
34.4
54.7
55.5
Following the evaluation protocols proposed in recent work Hochlehnert et al. (2025), we present additional results in Table A3. The results demonstrate that SRFT maintains performance edge, outperforming the base model by average of 18.0 points and RLGRPO by 7,4 points. These findings corroborate the robustness of our method, confirming that the reported gains are consistent across varying conditions and not merely artifacts of specific evaluation setting.
A.2 DYNAMICS OF THE ENTROPY-AWARE WEIGHTS
In order to better understand the dynamics of entropy-aware weights, we visualize the temporal evolution of the sequence mean for WSFT and WRL throughout the training process, illustrated in Figure A1. It is important to clarify that WSFT and WRL serve distinct mechanisms: USFT balances the SFT and off-policy RL objectives specifically for demonstration data, whereas WRL balances the positive and negative sample objectives for rollout data. Consequently, their relative magnitudes do not represent the global loss contribution ratio between demonstration learning and self-exploration, which are weighted equally in the total objective (Eq. (12)). Regarding their temporal behaviors, observe that WSFT gradually increases to 0.40 as training progresses, whereas WRL fluctuates around 0.13. These observations demonstrate that our method can effectively adjust the internal balance of objectives in response to training dynamics, thereby ensuring consistent exploration throughout the optimization process. Besides, the baseline coefficients of 0.5 and 0.1 in the weight formulations hyperparameters that can be fine-tuned to further improve performance.
0.40
0.16
Weight Value
0.38
Weight Value
0.15
0.38
0.14
0.13
0.32
WSFT
Wiat
0.30
100
200
300
400
500
100
200
300
400
500
Step
Step
Figure A1: Dynamics of the sequence mean for WSFr and WRL during SRFT training.
A.3 ANALYSIS OF ANNEALING DEMONSTRATION WEIGHTS
To investigate the influence of SFT on the RL process SRFT, we designed an experiment where the weight of the demonstration data annealed over time. We modify the total objective function by introducing a time-dependent coefficient a, formulated as:
(A1) where a decays linearly from 1 to 0 over the initial 150 and 300 training steps, respectively. The results summarized in Table A4, indicating that annealing the demonstration weights at either 150 or 300 seeps-leads to performance degradation compared to the standard SRFT approach. We attribute this to the high quality of the available demonstrations; the underlying behavior policy is
LSRET (0) = a (Edero (0) + Ademo (0)) + C /self-rollour (0),
17

---

Published as a conference paper at ICLR 2026
Table A4: Performance comparison of annealing the demonstration learning component different
step counts.
Annealing Steps
AIME24
AMC
MATHI-50
Minerva
Olympiad
Avg.
150
31.4
69.5
86.2
36.1
53.8
55.4
300
33.8
74.6
88.4
38.2
56.4
58.3
w/o annealing
35.3
74.3
89.8
39.7
58.3
59.5
already near-optimal (DeepSeck-R1), maintaining the supervised signal throughout the RL process proves beneficial for stability and guidance. However, we hypothesize that over significantly extended training horizons, persistent reliance demonstrations might eventually constrain exploration and impede RL performance. Investigating the dynamics of SFT over long-term RL training (Liu et al., 2025d; Hu et al., 2025b; Khatri et al., 2025) remains promising avenue for future research.
A.4 DYNAMICS OF SFT-INITIALIZED RL AND SRFT
To investigate the impact of warm-start initialization, we further analyze the length and reward curves of the sequential SFT approach, as shown Figure A2. We observe that when RI. is initialized with SFT, the response length maintains approximately 4k tokens, exceeding the ~3k length of SRFT. However, despite the increased length, the performance SFT-RL exhibits premature convergence. In contrast, SRFT sustains continuous performance improvement driven by its unified, single-stage learning from both demonstration and rollout data. This comparison underscores the superiority of SRFT in effectively leveraging demonstrations guide exploration without suffering from the stagnation observed in sequential paradigms.
0.8
5000
0.7
Response Length
Outcome Rewards
M ww
4000
0.8
3000
0.5
2000
SFT-RL
RL
0.3
SRFT
100
200
300
400
500
100
200
300
400
500
Step
Step
Figure A2: Response lengths and training rewards of pure RL, SFT-RL, and SRFT.
18

---

Published as a conference paper at ICLR 2026
B TOKEN DISTRIBUTION ANALYSIS
In this section, we provide the detailed experimental setting and mathematical derivation for the gradient of the Supervised Fine-Tuning (SFT) loss function, which referenced in Sec. 3.1.1.
B.1 EXPERIMENTAL SETTING FOR VISUALIZATION
In Sec, 3.1.1, we visualize token distribution changes during SFT and RL training. We use Qwen2.5- Math-7B as the base model, with training data from the main experiment and visualization prompts from the five evaluation benchmarks.
Methodology. The response sequences in Figure 2(a) are generated independently by the SFT and RL models from the same prompt. We exclude the base model's outputs from generation, as they are frequently incorrect or too concise to support effective analysis. This approach allows us observe how fine-tuning redistributes probability mass along the model's ' own generation trajectory. For each token y/ a sequence y = (y1,-- 1/T). we compute two conditional probabilities: Pine- tuneet (y/ (+t)) from the fine-tuned model (SFT or RL) and Phase (Ve y<e) from the base model. Although the sequence is generated by the fine-tuned model, probabilities are queried from both models using the identical context.
Visualization Scheme. To quantify the distributional impact of fine-tuning, we calculate the proba- bility shift: Ap Ptine-tuned (Ye y<t) Ptase (3)))<<), where You is the context sequence up to token yt. A diverging color map visually encodes these dynamics:
Red (Amplification, Ap 0): Indicates that the fine-tuning process has increased the token's 's probability relative to the base model.
Blue (Suppression. 4p < 0): Indicates that the fine-tuning process has decreased the token's 's probability relative to the base model.
Grey/Colorless (Retention, Ap Rs 0): Indicates negligible impact, suggesting the fine-tuning largely preserves the base distribution for this token.
This visualization framework facilitates a dual-level analysis: it reveals global trends in probability mass migration at macro level, while simultaneously highlighting specific tokens that undergo significant distributional shifts at a micro level.
B.2 SFT GRADIENT DERIVATION
Problem Setup. Given demonstration dataset D {( y:) where æ; is an input prompt and y; (Wi,1 .... W1.2;) is the corresponding target sequence, where T, = ly: is the length of the target sequence. The SFT objective function (the same as Eq. (1)):
LSFT E( log To [=)]
(A2)
Since the sequence probability is factorized as:
Te (
mo (o II 1=1
(A3)
The SFT loss becomes:
(SF (0) = (a
log no (we yee)
(A4)
Gradient Derivation, To derive the gradient (o),, we need to compute:
Vo R (z) Dam
vo log mo (,)))
(A5)
Now, let's consider the key insight: each time step t, the model produces probability distribution over the entire vocabulary V. The gradient of the log probability for the target token y, can be expressed in terms of the model's prediction probabilities for all vocabulary tokens.
19

---

Published as a conference paper at ICLR 2026
For any token V at position t, we have:
Vo log To( yeu)
VOTTo yet)).
(A6)
To(ulæ, y=t)
Since EvEy To(ukz wer) 1, we have the constraint:
Σ
Vema(t = 0.
(A7)
BEV
Using the chain rule and the fact that the softmax normalization affects all vocabulary tokens, the where 1 v= is the indicator function that equals when ys and otherwise.
gradient can be written as:
V log To (v yc1)
E
(o(v(lz,,<))) Ve (v)
(A8)
"EV
Substituting this back into the SFT gradient, we obtain:
VOLSFT = BLa.oin
Σ
Σ VEV
(Te (ula yct) - 1 1u=9,) Ve a log To (v) (v)),
(A9)
where is the LLM vocabulary, and 1,=y. is an indicator function that equals when token matches the target token YE and 0 otherwise. This formulation reveals the fundamental mechanism of SFT: at each time step, the gradient encourages the model increase the probability of the target token (1amy,
1) while decreasing the probabilities of all other tokens in the vocabulary (1a-y, = 0). The magnitude of the decrease for each non-target token is proportional its current probability aa(p y<r).
This analysis confirms our empirical observations that SFT produces broad, coarse- boarse-grained changes to the model's probability distributions across the entire vocabulary, systematically sharpening the distribution toward the target tokens in the training data.
C DETAILS OF VISUALIZATION OF LEARNING DYNAMICS
To characterize model dynamics during training, as introduced in Sec. 3.1.2, we formulate as a mapping from input prompts to output probability distributions over the vocabulary. Under this view, two models are considered equivalent if they produce identical output distributions for all possible prompts. This perspective allows represent each model a point in high-dimensional function space, where each point corresponds to unique conditional probability distribution.
C.I THEORETICAL DEFINITION
The Model Probability Space Formalization. To visualize the learning dynamics, we conceptualize each model checkpoint as point in infinit-dimmensional probability space. A model's position is defined by its conditional probability distribution, (-- over the entire vocabulary for any given prompt æ. To measure the distance dissimilarity between fine-tuned model my and reference model x NR, we use the Kullback-Leible (KL) divergence. The KL divergence ( quantifies how much the distribution тe diverges from the reference distribution πR. However, computing the exact KL divergence is computationally intractable for large-scale language models. Therefore, we cross-entropy as a practical and effective proxy:
( (
me)
H (TRR To) H ((TR) = H(RR nT) - c,
(A10)
where (TTR) denotes the entropy, a constant term c for each reference model. Thus, we can use the cross-entropy to measure the distance between the fine-tuned model and the reference model.
Measure Learning Dynamics via Reference Models. To make this figh-dimensional space tractable for analysis, project the model's learning trajectory onto lower-dimensional space defined by
20

---

Published as a conference paper at ICLR 2026
set of diverse. perffrrnnn reference models, R {ITRI :a:..... RR }. As mentioned in the main text, our reference set includes Qwen2.5-Math-7B DeepSeek-R1, and QwQ-32B. For any given fine-tuned model checkpoint TT. its position in this reference- based space is represented by vector of cross-entropy values:
dR( = (dR, (Te), dR (Te), ..., dRR, (TT))"
(A11)
where each component of this vector, dR. (TT) tuned model from the corresponding reference model RR. By computing this vector at various stages of training, we can trace the evolution of т as trajectory through this constructed space. This process allows us to visualize and compare the optimization paths taken by different fine-tuning methods, such as SFT and RL, revealing their distinct impacts on the model's underlying probability distribution. For visualization in 3D plot (as in Figure 3), we can select the three most informative reference models as axes or use dimensionality reduction techniques.
(TTR; To). measures the divergence of the fine-
C.2 EXPERIMENTAL SETUP
To instantiate the above definition, we construct dataset comprising 1,024 response sequences generated by each of the reference models under identical prompts. These prompts are drawn from a diverse mixture of mathematical reasoning benchmarks (including AIME24, Minerva, Olympiad, AMC, and MATH500) to ensure broad coverage across problem domains and difficulty levels.
For each response, we compute the distance between the probabilities assigned by model अæ the reference tokens, and then aggregate overall responses to obtain the final distance dn, (To) for each R, These distances collectively define the model's position in the projected subspace. We select three reference models to construct the projection basis: (1) DeepSeek-R -1 (Guo et al., 2025), representing state-of-the-arr reasoning performance; (2) QwQ-320 (Team, 2025), serving as high-performing but structurally distinct baseline; and (3) Qwen-2.5 1.een-255-Math-77 (Yang et al., 2024), which acts as the base model prior to fine-tuming tunituning. Together, they span a spectrum from foundational to advanced capabilities, forming semantically meaningful coordinate system.
We train the LLM model on the OpenRI OpenRI-Matie Mathh 220k dataset, maintaining consistent training settings across all experiments. We construct evaluation datasets using 1,024 prompts sampled from five evaluation benchmark tasks. For every model checkpoint during training, we compute its distance to reference models based the probability assignments over reference responses. This procedure yields a trajectory in model space that traces the model's evolution throughout training. By comparing these trajectories across training paradigms (e.g.. SFT and RL), we reveal distinct optimization dynamics and convergence behaviors. Notably, we observe that during the SFT-RL training pipeline. the optimization direction in the RL phase opposes that of the SFT phase, imposing an additional learning tax on sequential LLM training. This phenomenon aligns with findings from recent studies (Shenfeld et al., 2025; Jín al., 2025a), providing insights into optimizing LLM training efficiency and the integration of SFT and RL strategies.
The dineesiiienii distance framework we proposed provides both theoretical grounding and practical interpretability for analyzing LLM training dynamics. It enables direct comparison of different model variants and reveals how specific training strategies influence progression through the reasoning capability landscape.
21

---

Published as a conference paper at ICLR 2026
C.3 LEARNING DYNAMICS VISUALIZATION WITH DIFFERENT EPOCHS SFT
In Sec. 3.1.2, we hypothesize that "SFT may overfit
SPFT
to demonstrations, and the subsequent RL attempts to correct this deviation, thereby increasing the learn- ing tax of the two-stage paradigm." " To provide granular analysis of this phenomenon, we investigate the impact of SFT training duration in Figure A3. We observe that this learning tax is sensitive to the number of SFT epochs: extended supervised train- ing (c.g., or epochs) tends to confine the model within the demonstration distribution, thereby in- creasing the overfitting issue. Conversely, early stopping (e.g.. epoch) followed by RL can mit- igate this distributional shift. These findings under- score that identifying the optimal SFT epoch is complex and often underexplored hyperparameter tuning challenge (Chu et al., 2025; Jin et al., 2025b). Our method SRFT addresses this dilemma by inte- grating SFT and RL into single stage with adaptive weighting. enabling for direct policy optimization without the need for manual stage- saie-swit
SFTLL epoct
SFT12 epochs)-RL
SFCO epochsi-RL
Reeooeoee Distance
44.7
6R.1
DwO 228 Distentee
54.5
Owen2255 M-att25- Distance
Start
X
Finch FT
Figure A3: Visualization of learning dynamics across different SFT epochs.
D DIFFERENCE BETWEEN SRFT AND ON-POLICY DISTILLATION
In this section, we clarify that a direct comparison between SRFT and on-policy distillation methods such as Generalized Knowledge Distillation (GKD) (Agarwal et al., 2024) is not straightforward due to fundamental theoretical and technical divergences.
Theoretical Differences. Fundamentally, SRFT and GKD operate under different paradigms:
RL vs. KD Frameworks: SRFT is grounded in the Reinforcement Learning (RL. framework. Its core innovation lies in the single-stage unification of SFT and RL achieve direct policy optimization via reward maximization. In contrast, GKD operates within the Knowledge Distillation (KD) framework. rather, it aims to mitigate the "distribution mismatch between output sequences seen during training and those generated by the student during inference" by minimizing the divergence between student and teacher distributions.
noted by Agarwal al. (2024), GKD is not an RL algorithm;
The optimization objective of GKD is to align the student's s policy with that of the teacher, which theoretically upper-bounds the student's performance by the teacher's s capabilities. Conversely, SRFT utilizes RL to explore the solution space. By leveraging reward signals, SRFT can potentially discover better solution paths and optimize the policy surpass the performance of the behavior policy.
Performance Upper Bounds:
Technical Differences. Implementing on-policy distillation with large-scale reasoning models (e.g.. DeepSeck-RI) presents specific technical challenges that SRFT avoids:
Vocabulary Mismatch: Standard GKD requires the teacher and student models to share the same tokenizer vocabulary to compute the token-level KI. divergence. In our setting, the teacher (DeepSeek-R1) and the student (Qwen-2.5-Math-77 Applying GKD would necessitate complex sequence and vocabulary alignment techniques. SRFT, relying on offline demonstration data, bypassing the need for the teacher's tokenizer.
utilize different tokenizers.
Initialization Requirements: GKD typically necessitates initialization from an SFT model that can already "generate sequences of adequate quality" for the teacher to provide meaningful feedback. In contrast, our experiments demonstrate that SRFT can effectively optimize the policy directly from the base model.
22

---

Published as a conference paper at ICLR 2026
E DISCUSSION ABOUT ENTROPY IN LLM FINE-TUNING FOR REASONING
Entropy is a crucial metric in the LLM reinforcement fine-tuning (Cui et al., 2025b; Cheng et al., 2025; Wang al., 2025b). is used measure the uncertainty of the LLM policy's ' predictions. The definition of policy entropy as follows:
H(mo)
no(ula, yet) log (o(aa yet),
(A12)
VEV
where my the LLM policy, the LLM vocabulary, a: is the input prompt, yer the sequence of tokens in the response before the current token, and / is the length of the response. A higher entropy indicates a more uniform distribution of probabilities across the vocabulary, while lower entropy indicates more concentrated distribution.
Eaare Gradient Clipping. In this work, the entropy of the LLM policy to adjust the weights of the SFT and RL components. Be- sides, our investigation into the entropy characters of tokens modified by fine-tuning reveals that re- inforcement learning predominantly targets tokens with high entropy distributions, a finding that aligns with recent work on selective optimization in lan- guage models ( (Wang et al., 2025c). To empirically validate this observation, we design controlled exper- iments implementing gradient truncation for high- probability tokens during RL training. As demon- strated in Figure A4, the model'' 's performance re- mains comparable to the original RL algorithm even when gradients are truncated for low-entropy tokens, providing strong empirical support for our hypoth- esis. This evidence confirms that RL operates with remarkable selectivity, precisely adjusting tokens with uncertain distributions while leaving confident predictions largely unchanged. In contrast, SFT applies broad modifications across the entire token space, fundamentally altering the model's distributional characters less discriminative manner.
Accuracy
Prob_clip
Prob_clip
Prob Prob_clip 0.9
Base Policy
100
200
300
400
500
Step
Figure A4: Performance of RL with gradient clipping for low-entropy tokens.
23

---

Published as a conference paper at ICLR 2026
F BACKGROUND
F.1 REINFORCEMENT LEARNING
In RL training, the LLM's token generation process is modeled as a Markov Decision Process (MDP) (Puterman, 2014). We define state 8 at step as the concatenation of the input prompt z and all tokens generated so far yer. This state serves as input to the policy model Ta (-(sss)) Specifically, the policy processes 8t (0,y)t))
-2, Y1, "22....ut--.. where x; denotes the i-th token of the input æ and y, represents the token generated by T at step j. An action d, corresponds to generates
the selection of the next output token yr- The LLM, acting as a policy Te geneeatt).ennerrte a trajectory
(a sequence of tokens) in response to the prompt æ. A reward function R(z, y) = Ei=1 r(æ, ye) provides scalar score for the entire trajectory y given prompt z, typically derived from human evaluations or automated metrics. In the context of RL, the behavior policy (g) (x) refers to the model that generated the responses in the replay buffer. This policy is crucial for RI., particularly for off-policy learning, as it enables proper importance sampling corrections to account for the distribution shift between the data-generating model and the current training model. The MDP formulation in LLMs presents several notable characteristics:
Sequential state representation: At each step t, the state 8 E S consists of the concatenation of the input prompt and all actions (tokens) generated so far yer. This state serves input to the policy model (-(1(8)).
Sparse and delayed rewards: Rewards R(x, y) are typically sparse, provided only upon completion of a sequence y. This dependency on the final output's overall quality complicates credit assignment across the generation process.
F.2 REINFORCEMENT LEARNING FOR LLM REASONING
The pursuit of complex reasoning capabilities in LLMs has witnessed remarkable progress, with RL emerging as a pivotal methodology for enhancing reasoning abilities (Jiang et al., 2025; Xu al., 2025) and tool-use capabilities (Chai al., 2025a; Fu et al., 2025; Chai al., 2025b; Tu et al., 2025) beyond the limitations of SFT alone. Recent approaches such as GRPO (Shao et al., 2024; Guo al., 2025), DAPO (Yu al., 2025), GR.GRPO (Liu et al., 2025c), and VAPO (Yue et al., 2025b) have demonstrated substantial improvements in mathematical reasoning and complex problem- solving tasks. However, the precise mechanisms through which RL enhances reasoning capabilities remain incompletely understood. Several empirical investigations suggest that reinforcement learning primarily serves to elicit, refine, or improve the sampling of pre-existing reasoning abilities rather than instilling entirely novel fundamental reasoning skills from scratch. For instance, Yue et al. (2025a) question whether current reinforcement learning via verifier feedback (RLVF) genuinely expands the reasoning boundary (pass@k or primarily improves the sampling efficiency of already known solutions (pass@ I). Similarly, Wang et al. (2025a) highlight that base models already possess substantial reasoning capabilitiess that reinforcement learning can effectively unlock redirect. Nevertheless, ProRL (Liu et al., 2025d) demonstrates that RL-trained models can achieve improved success rates tasks where base models completely fail, suggesting that sustained and stable reinforcement learning training can indeed expand the reasoning capability boundaries of LLMs. In this work, we design single-stage method that combines SFT and RL, maintaining stable entropy during training and achieving continuous performance improvement.
F.3 INTEGRATING SUPERVISED SONE-TUNNG AND REINFORCEMENT LEARNING
The synergistic interaction between SFT and RL represents critical area of investigation in modern LLM development. SFT on high-quality reasoning chains can establish a robust initial policy foundation, which RL can subsequently optimize. Cai et al. (2025) explore the necessary extent of exploration following SFT, finding that RL continues to provide substantial benefits by enabling models to deviate from potentially suboptimal SFT trajectories. Recent research suggests that SFT may equip models with structured reasoning templates that RL subsequently validates and improves (Chen et al., 2025a). Nevertheless, determining the optimal strategy for combining these complementary paradigms remains an active area of debate. To enhance sample efficiency and provide structured guidance for RL exploration, researchers have investigated various approaches
24

---

Published as a conference paper at ICLR 2026
for integrating external supervision into the reinforcement learning framework. UFT (Liu et al., 2025a) proposes a novel paradigm that merges SFT and RL into a single process, using informative supervision signals like hints from partial solutions to guide exploration and accelerate convergence. Addressing the limitations of on-policy learning, LUFFY (Yan et al., 2025) augments RLVR by incorporating off off-policy reasoning traces from stronger models, dynamically balancing imitation with on-policy exploration to improve capabilities, ReLIFT (Ma et al., 2025) addresses the limitations of pure RL by interleaving reinforcement learning with supervised fine-tuning on high-quality demonstrations collected during training, enabling models to acquire new knowledge beyond their original capabiliiies. TAPO (Wu al., 2025) enhances RL by incorporating external high-level guidance in the form of "thought patterns" abstracted from prior samples, adaptively integrating these to balance model-internal exploration with external strategy exploitation. SASR (Chen et al., 2025b) offers hybrid framework that theoretically unifies SFT and RI, using SFT for initial warm-up and then adaptively blending it with an online RL method based on training dynamics to maintain core reasoning while exploring diverse paths, using high-quality SFT demonstrations as key external data source. CHORD (Zhang et al., 2025a) proposes a dynamic weighting scheme that dynamically adjusts the weights of the two components based on the training dynamics, achieving superior performance on both SFT and RL tasks on the instruct model. Furthermore, the single-stage integration of SFT and RL helps mitigate the catastrophic forgetting problem that previous methods encountered when transitioning from SFT to RL (Chen et al., 2025b: Liu et al., 2025b). These approaches collectively underscore an emerging trend toward more sophisticated integrations of supervised signals within reinforcement learning frameworks to improve reasoning alignment and overall performance.
G EXPERIMENTAL DETAILS
Training. For SFT, we follow the SFT configuration of OpenR I-Qwen-7B (Face, 2025), performing full tin-ttuing on DeepScek-RI generated reasoning traces and prompts. The training hyperpa- rameters include a batch size of 128, a learning rate of 5 x 10-6 a linear learning rate schedule with 10% warmup, and training for 3 epochs. For SFT with KL regularization, we use identical settings while adding a KL divergence regularization between the current policy and the base model Soven2.5-Math- with weight À = 0.2. The SFTKL loss is:
SFF (0) = F log [y]==]] + (0, Obes)),
(A 13)
where (K, (raee) is the KI. divergence between the current policy and the base model. For RL, we train for 500 steps with 8 rollouts per prompt. The learning rate is fixed at 1 x 10-6. For our method, SRFT, we set the entropy loss coefficient to 0.001 and remove the KL. loss term by setting A = O. To further maintain entropy stability, we adopt the policy shaping mechanism proposed by Yan et al. (2025) and set the shaping value to y 0.1. Additionally, we omit the clipping operation, the standard clipping mechanism becomes imbalanced and potentially unstable when setting the behavior policy n. = 1. Since the maximum sequence length for Qwen2.5-Matth -att2hh-- 7 is 4,096, which is insufficient for our tasks, we increase the RoPE theta from 10,000 40,000 and expand the window size to 16,384. For all experiments, we use verl (Sheng et al., 2024) as the implementation framework. We tune hyperparameters for all baselines to ensure fair and optimal performance comparisons. All experiments are conducted on 8x8 A A100 GPUs.
Evaluation. All evaluations are conducted using VLLM (Kwon et al., 2023) with the sampling temperature set to 0.6 and a maximum generation length of 8,192 tokens. Given the substantial computational costs associated with evaluating multiple RL baselines, we do not perform an ex- haustive, model-specific hyperparameter grid-search during evaluation as suggested by Hochlehnert et al. (2025). Instead, to ensure fair and rigorous comparison, we maintain the same evaluation protocol across all methods, utilizing unified prompt template, an identical reward function, and consistent inference parameters. Regarding metrics, for datasets with limited sample sizes (AIME24 and AMC), we report the @vg@32 metric to reduce variance; for the remaining three datasets, we adopt pass@1 I as the primary evaluation criterion. We verify the correctness of generated solutions using Math- Math-Verify and OAT-Grader (Liu et al., 2024). For baseline comparisons, we independently validate the results of the base model, SFT-related baselines, GRPO (Shao et al., 2024), LUFFY (Yan
25

---

Published as a conference paper at ICLR 2026
et al., 2025), and ReLIFT (Ma al., 2025), while results for TAPO (Wu et al., 2025) and other zero-shot reinforcement learning models are taken from the TAPO and LUFFY papers.
Reward Design. To evaluate the impact of our method, adopt simple reward function follows. All training experiments employ the same reward function.
1,
y is correct
R(x, y)
(A14)
0, otherwise
Chat Template. Following Yan al. (2025); Ma al. (2025), for all training paradigms (SFT, RL, SRFT), we employ unified system prompt that encourages systematic reasoning, as shown in Figure A5, We also experimented with alternative templates, shown in Figure A6.
Chat Template (Ours)
Your task follow systematic, thorough reasoning process before providing final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution.
In the Thought section, present your reasoning using the f<rmat:"<think<ln [thoughts] </think<<nn" Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas.
After "</think>In" in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration the Thought section.
If applicable, include the Answer in \boxed for closed-form results like multiple choices or mathematical solutions.
Question: (question)
Answer: [answer)
Figure A5: Chat template for all training paradigms (SFT, RI,, SRFT).
Chat Template (Qwen)
Please reason step by step, and put your final answer within Wboxed| ).
Question: (question)
Answer: (answer)
Figure A6: Chat template for Qwen-2.5-Math.
Template Ablation. minimize template influence, we evaluated the base Qwen-7B-Man model with different templates. Results are shown in Table A5, which indicates that template design effectively guides the model's reasoning process while maintaining consistency across different mathematical domains.
Table AS: Template ablation results on mathematical reasoning benchmarks
Template
AIME24
AMC
MATH500
Minerva
Olympiad
Average
No Template
0.302
0.132
0.596
0.424
0.134
0.318
Owen2.5-Math-7B
0.144
0.088
0.446
0.303
0.111
0.218
SRFT
0.296
0.165
0.648
0.448
0.141
0.340
26

---

Published as a conference paper at ICLR 2026
Table A6: Benchmarks used in this study. "_" indicates the split is not officially provided.
Dataset
#Train
#Test
Task Type
Domain
License
Source
Training Dataset
OPENRI-MATH-222 220,000
Math reasoning
Mathematies
Apachc 2.0
[Link]
Mathematical Reasoning Benchmarks
AIME24
30
Math competition
Mathematics
MIT
[Link]
AMC
83
Math competition
Mathematics
Apache 2.0
[[ink]
MATH500
500
Mathematical reasoning
Mathematics
[Link]
MINERVA
272
STEM reasoning
STEM
Apache 2.0
[Link]
OLYMPIAD
674
Math competition
Mathematics
Apache 2.0
[Link]
Out-of-Distribution (OOD) Benchmarks
ARC-C
1,172
Science reasoning
General science
CC-BY-SA-4.0
[Link]
GPQA-D
198
Scientific reasoning
Bio, Phys, Chem
CC-BY-4.0
[Link]
MMLU-PRO
12,032
Multi-task understanding
Multddsscplinaay
MIT
[Link]
H DATASET DETAILS
H.1 TRAINING DATASET
MaanRR-Math-4kk---11 (Yan et al., 2025) is the training dataset in this work, which constitutes subset of OpenR 1-Mttt--2Mttt--20 220k4 (Face, 2025) comprising 46,000 mathematical problems sourced from NuminaMath 1.5 (LI et al., 2024), accompanied by high-quality reasoning responses generated by DeepSeek-R1 (Guo al., 2025). Each problem is associated with two to four reasoning traces generated by the DeepSeek-R1 model. The traces have been verified using tools like Math Verify and Llama-3.3-7000 3.3-70B-Instruch ensuring least one correct reasoning path per problem. This dataset chal- lenges models to understand and replicate complex reasoning processes across various mathematical domains, including algebra, geometry, number theory, and calculus.
H.2 EVALUATION BENCHMARKS
To evaluate the models above, we use eight benchmarks categorized into mathematical reasoning benchmarks and out-of-distribution (OOD) benchmarks as described below. To mitigate potential information leakage, we randomly shuffle the option orders for all multiple-choice questions.
H.2.1 MATHEMATICAL REASONING BENCHMARKS
AIME24 is a benchmark dataset based on problems from the 2024 American Invitational Mathematics Examination, a prestigious high school mathematics competition in the United States. The AIME24 benchmark tests a model's ability to solve challenging mathematics problems by generating step-by- step solutions and providing the correct answers. This dataset contains problems from the American Invitational Mathematics Examination (AIME) 2024, organized in JSONL format, where each line represents complete problem. Concepts typically covered include topics elementary algebra, geometry, trigonometry, well as number theory, probability, and combinatorics. The examination consists of 15 problems with integer answers between 0 and 999, requiring advanced mathematical reasoning and problem-solving skills.
AMC is a validation dataset containing problems from the American Mathematics Competitions, specifically AMC12 from 2022 and 2023. All 83 problems come from AMC12 2022 and AMC12 2023, and have been extracted from the AOPS wiki page. The AMC 10 is a 25- 25-question multiple- choice competition designed for students in grades 10 and below. The content covers mathematical
wtt http:///rra...toeee.eee////////e "https://huggiggtf http:///ggggggggeeeoccaaaatteeep
27

---

Published as a conference paper at ICLR 2026
topics such elementary algebra, basic geometry, area and volume formulas, elementary number theory, and elementary probability. This dataset serves as an internal validation set and focuses on competition-level mathematical problems comparable in difficulty to AMC12 and AIME exams.
MATH500 is a carcfully curated subset of mathematical problems designed for robust evaluation. MATH500 is subset of 500 randomly sampled questions from Hendrycks' 2021 MATH dataset, created by OpenAl in late 2024 as a consequence their appropriation of 90% of the original 5,000 MATH questions for training data for reinforcement learning on Ol-series models. The dataset maintains the diversity and complexity of the original MATH benchmark while providing a clean evaluation that avoids potential data contamination issues.
Minerva is reasoning benchmark that encompasses wide range of STEM domains and difficulty levels. The dataset is designed evaluate models' capabilities advanced STEM problem-solving including "solid-state chemistry", "information and entropy", "differential equations", and "special relativity".. includes problems requiring multi-step reasoning, symbolic manipulation, and deep mathematical understanding across various STEM fields. Minerva contains 272 problems in total, 191 of which have numeric solutions and have symbolic solutions.
Olympiad refers to mathematical olympiad-level problems that represent some of the most chal- lenging mathematical reasoning tasks. Unlike existing Olympiad-related benchmarks, datasets in this category focus exclusively on mathematics and comprise vast collections of competition-leved problems. These problems are meticulously categorized into 33+ sub-domains and span across 10+ distinet difficulty levels. These problems require exceptional mathematical insight, creativity, and advanced problem-solving techniques typically seen in international mathematical competitions. We specifically utilize the OE_TO_MATIIS
COMP subset for our evaluation.
H.2.2 OUT-OF-DISTRIBUTION (OOD) BENCHMARKS
ARC-C (AI2 Reasoning Challenge-Challlngge is dataset of grade-school level science questions that require commonsense reasoning and knowledge application. The dataset consists of multiple- choice questions that are designed to be easy for humans but challenging for Al systems, testing the model's s ability to apply scientific knowledge and reasoning in everyday contexts beyond pure mathematical domains.
GPQA-D Graduate-Lever Google- Proof Q&A-Diamond) is a challenging benchmark designed to evaluate advanced reasoning capabilities in scientific domains. GPQA consists of 448 multiple-choice questions designed to evaluate the capabilities of LLMs and scalable oversight mechanisms. This dataset provides ghoogle-proof questions in biology. physics, and chemistry, designed to test deep domain expertise and reasoning under challenging conditions. The diamond subset contains 198 hard problems. The questions require graduate-level knowledge and are specifically designed to be difficult to answer, even with an internet search.
MMLU-Pro
(Massive Multitask Language Understanding Srarssaaddinn-oofessionalll an enhanced version of the original MMLU benchmark designed to be more challenging and robust. The MMLU-Pro dataset an enhanced version of the Massive Multitask Language Understanding benchmark. It's designed to be more robust and challenging, aiming to rigorously evaluate language understanding capabilities. MMLU is a comprehensive benchmark that covers 57 subjects across fields like mathematics, history, law, and medicine. It assesses not only factual knowledge but also the model's capacity to apply this knowledge in context-specifis scenarios. MMLU-P U-Pro increases the difficulty and reduces potential shortcuts while maintaining broad coverage across academic disciplines.
28

---

Published as a conference paper at ICLR 2026
1 TOKEN PROBABILITY VISUALIZATION OF OTHER MODELS
To verify the generalizability of our findings regarding the distinct impacts of SFT and RL on token distributions, and to address potential concerns regarding model-specific biases in the Qwen series, we extend our visualization analysis to non-Qwen architectures. Specifically, we conduct identical token probability analyses on OLMo (Walsh et al., 2025) and the Llama-3.1-4 3.BBBB series (Grattafiori et al., 2024) (both Base and Instruct variants).
As illustrated in Figure A7 for OLMo, and Figures A8 and A9 for Llama-3.1, the results exhibit pat- terns highly consistent with our primary observations Qwen2.5-Math-78 (Sec. 3.1.1). Across these diverse architectures, SFT consistently induces broad, global shifts in probability mass, effectively reshaping the policy's output distribution. In contrast, RL demonstrates a more selective optimization strategy, primarily adjusting probabilities for specific tokens while maintaining a distribution closer to the initial policy.
Question: You helpnal All Asstrtant that prow dee reli-rusoeed desiledd reppraas. You tirat than reseoring process niemal monolquee and then provide the user with the a sower Respond the folloing formal sik-... -lthinko <irgware -rarrsers elusen tudlt 2:33 cur, dusing 1ong ahwo they anive ther doul 1 Please reason step step, and your fina answer within (ooeec))
Lower Probability After Funi--uning
Similar Probability
Higher Probabillty After Fine tuning
SFT
Reiw Surves
1907s srrend west they onder wed Niom
se tei li ins cee litaaingg our from 1S snevts slevee nut coun lmvens divinc lncish
Eial sunee you
thw bed. That should mabi biuie tw alie hovw and
to
woutee bul mouse al tounl Beceesn I4 mnnose
1R Norr houn) un Seat thee hore ser mavnng form 2:30 me, sallin S000 beny Shant Ii He] The seer vesgrtheent Mill doiti cren 12th Secoens dirded neterl UN snttrell
war Ver just vale 2130
be
agril
thy
lapo seords
300 claed boous sederr houe miveee
wan
M000 the 7004
ites ii's
wi
his abette Ind Bull wis whein Yomker
sife we
akd from
The cordard
vee the will
soowe MMB lito Vaanre steal oer M00 mecra until smeul IB ooeenie aM aveat fex coreen 1000 moonc ino BI Jon vanr moces) tou vedeiine bean
seerd meuuee Usmel
21 to (yee IN3 Leee founs) yeer
baeand mey.
RL
aoeree ane basd 11000 Nobhe's dtaanicc this MMak
L moneeity 2004 Holk r-b Bric not MonG
12 yeet
(oats 19901 INC 1/ae
Figure A7: Token probability distribution visualization for OLMo-2-
Question Cusotion At 2:30 pun. during long divve Buttii asio for pose its "Avo we there yar? Hor nother respords." "We will be there /200 secuncs." " Oxberss mother comre al the attemoon will the artve their deetrratin? Anover Lat' thine step thpp
Lower Probability After Fiieettuiing
Simitar Probability
Higher Probability After Fine-tuning
SFT
Hey Lnve. the Mi mon live elll work L1
be 'oEA I Nesl
poy. (at 58's 6
C.U Besi eeE our
Jui
2010
teee tov Lat's
angle
RL
1/B 281 Nes 380 monidy sil wes 25 2013 1000 mmund Mael Day 4420 ane
uppl wide Thank be hog)
Figure A8: Token probability distribution visualization for Llama-3.1-88
29

---

Published as a conference paper at ICLR 2026
iist tiek about the resseing prcceso insma mondligue prov de the uset with siees Pacporel in (ollwiing not with otfiiio
catewe chrnperseient Jeb-deast header siheaandend header -d: At 2:50 p.m durpo long diie Botel wass pa pavents "Ane yet"
thy
Please
deep by des, and your fins anooer withn (acooet))-aee dect, Mi-dddaa 1_ header, Mtbaadidanddees header, ltl-
Lower Probabiiity After Fine tuniig
Similar Probability
Higher Probability After Fine tuning
SFT
Sillo ja Deip, Jet jestein miai Low fall bes Seriyy we and tee howe 2. di Biil ku Lakel
2, LoW oou whi Uitiss will but
reed to Bond I. 203
and credes fou tie
2010 the Jecvll bene DeNis, an
th enenn toi
diadel (Sed
wit
Eel the Took dlenn deikel tiem,, (MM 138
7,200 (n9t /oooa hoan
6.00 1.20 7,300
60) 2.300 100 vnudes Ouyy the mndes
1X0
sicuus
4:30
Rioo
hour
Figure A9: Token probability distribution visualization for Llama-3.1 2.1-8B-Instrue
J TOKEN PROBABILITY VISUALIZATION OF SRFT
We visualize the token probability distribution of SRFT after training, as shown in Figure A10. We observe that the token probability changes are moderate, achieving a balanced point between SFT and RL that enhances the model's reasoning capabilities while preserving its base abilities.
Question: Every momiin Ava goes 939 Allometer long walt stope at coffee stop antrwwrdss When walks at constant apeed per and Aind the number of minutes the walk takes hor, induuing the S6 minutes opent the coffee stop.
takeo her
inclucing 955 minutes sownl the cothee shop. Suppoon Ays walks Se- y21s kicomieee hour
Lower Probabilitt After Fline-tuting
Simitar Probability
Higher Probability After Fine-tuning
SRFT
other Lane
sa
1/ikl LS D. 2' Aocumties houe
reert
utar hour]
raloo ven Iill autou.. Ion (8h0n saalin Tec 3
the ssing xX InC (XX 3 hoam WMR 1. rnliuce BS0g thil tna cost
the cofte Phg
giaz
1900 be boat (. 5 den Dis maninn i' cae a centar
muk
Afer
Uhw 10 24
1 A col E8
Su Inc C. IB 2.. Vaa4-
the 111 the fox equrtee 1. EBC X 4 the 1.8 Stl ea uin
Ie02 vee 1
Lift hor ruli
Figure A10: Token probability distribution visualization for SRFT.
K DISCLOSURE ON THE USE OF LLMS
We utilized large language models to enhance the grammar and clarity of this paper. The model's role was strictly limited to language refinement and did not extend to generating core content, such as research ideas, experimental results, or technical analyses. All authors have reviewed and assume full responsibility for the final content.
30


Reviews:
[{"review_id": "35FMmBS0PP", "reviewer": "Reviewer_bDfv", "summary": "This paper proposes SRFT (Supervised Reinforcement Fine-Tuning), a single-stage approach that unifies SFT and RL for LLM reasoning. The key idea is to balance demonstration distillation (SFT) and online exploration (RL) in one loop, using policy entropy as a dynamic indicator to weight the two objectives and avoid either overfitting to demos or premature entropy collapse. Conceptually, SFT makes coarse, global shifts in the policy, while RL makes fine, selective adjustments; combining them directly improves efficiency and stability. Empirically (Qwen-2.5-Math-7B), SRFT achieves improvement across five math-reasoning benchmarks over baseline methods like RL-only and SFT+RL. Training curves show faster reward growth, longer solutions, and stable entropy versus RL-only.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposal to unify SFT and RL into a single stage is interesting and enables finer control over the trade-off between the two.\n- The paper offers a comprehensive analysis of learning dynamics and the respective effects of SFT and RL for language models, improving our understanding of both paradigms.\n- The manuscript is clearly written and easy to follow.", "weaknesses": "- The experiments are not yet fully convincing; more evidence is needed to demonstrate the effectiveness of the proposed SRFT method. In particular, the SFT data use DeepSeek-R1 responses (Line 360), which likely exceed the quality of the Qwen2.5-7B policy’s rollouts. Figure A2 suggests the model learns primarily from SFT—the SFT loss dominates—implying that distillation, rather than RL, drives most gains. For a fair comparison, both RL and SRFT should start from the same fine-tuned initialization, or the SFT data should be generated offline by the policy model itself to remove teacher-distillation effects.\n\n- Prior work (e.g., OpenAI-o1, DeepSeek-R1) has demonstrated the importance of RL scaling for boosting LLM capabilities. Always incorporating SFT during training may hinder RL scalability and waste compute; the paper should consider when SRFT scales favorably versus when pure RL is preferable or how SRFT affects the scalability of RL.\n\n- In the method part, the paper proposes various strategies, like entropy-guided reweighting and mixing different losses for SRFT. It would be better to provide detailed ablations to demonstrate the effectiveness of these designs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TfcR6tw6RK", "reviewer": "Reviewer_svi3", "summary": "This paper conducts an analysis on the learning dynamics (as evidenced by induced token probability distributions, entropy-based analysis and training dynamics) of SFT and RL approaches for fine-tuning LLMs for improved reasoning. The paper shows that SFT induces coarse-grained and large changes in the base policy distribution of the LM whereas RL induces finer-grained and localized policy changes. Using these observations, the paper introduces a joint single-stage method called SRFT that couples three different loss objectives including self-rollouts and standard SFT + RL. Results showcase that the proposed method improves results across several benchmarks like AIME'24, AMC'23 and even OOD benchmarks like GPQA-D and MMLU-Pro.", "questions": "All my questions are also raised in the weakness section so that its easy for the authors to answer all concerns / queries jointly.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to follow.\n- The analyses conducted are presented in a compact form making it easily digestible.\n- The motivation to introduce the SRFT method is well-grounded based on the empirical analyses conducted in the paper.", "weaknesses": "- Currently in line 160, there is a claim that states that the fig 2a results reveal a fundamental difference between SFT and RL regarding their reshaped probability distributions. Could there be some details provided regarding how exactly the heat-map is computed. Currently, it is unclear and a lot of details are omitted.\n    - Is the token output sequence generated by the base model or the SFT/RLd model? Further, are the log-probs simply computed token-wise with the base and the SFT/RLd model? The appendix mentions teacher forcing, but it is unclear what that means in this context, is the base model the teacher whose rollouts are used?\n    - This experiment is also only done with the Qwen model series, since recent works have shown that Qwen models might have some unintended biases (https://arxiv.org/abs/2506.10947, https://arxiv.org/pdf/2507.10532), it would be great to also show the result on non-Qwen models.\n    - It would be interesting to further establish this trend by plotting KL-divergence / JSD plots wrt to the base and SFT/RL models.\n    - Further, this heat-map is for one particular problem instance. It would be good to show some more examples, even though fig 2b is a more a quantitative treatment of the same.\n- The numbers shown in tab 1 don’t seem fully accurate for Qwen-2.5-Math-7B, AMC and Minerva. Reference the Sober reasoning paper (https://arxiv.org/pdf/2504.07086). What hyper parameters were used for evaluation? Would the same takeaways hold If the \"optimal\" evaluation hyper parameters were used as suggested in the sober reasoning paper?\n- In lines 202-203, it is stated: “This result suggests that SFT may overfit to demonstrations, and the subsequent RL attempts to correct this deviation, thereby increasing the learning tax of the two-stage paradigm.” Is this result globally true or does this depend on the number of epochs of SFT training (currently the paper seems to be doing 3 epochs). It would be interesting to ensure that this result globally holds true regardless of the number of optimization steps (since this is important for final performance as noted in OpenThoughts (https://arxiv.org/abs/2506.04178))\n- Is the SRFT method better than simple on-policy distillation (as shown in methods like GKD: https://arxiv.org/abs/2306.13649). For a fair comparison in the current setting, the teacher used for on-policy distillation should be DeepSeek-R1.\n- All experiments are only conducted with Qwen models. As mentioned earlier in the review, prior works have shown biases and data contamination in Qwen models with regards to the benchmarks used for testing. Would the results of SRFT also hold for non-Qwen models?\n- Why do the performance numbers between tab 1 and tab 2 differ for qwen2.5-math-7B, shouldn’t they be the same numbers?\n- There should be an ablation with different weighting factors for each of the loss terms in eq 12. For example, it might be plausible that only the self-rollout RL loss term might be the most beneficial (similar to an on-policy distillation loss), hence a leave-one-out ablation experiment (where each time the training is done only with two loss terms in the mix is used) should be conducted for more conclusive results.\n- Do the results also improve on AIME’25? This is an important test of generalisation as shown in Sober reasoning paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sxn8Frc4Fl", "reviewer": "Reviewer_5rC4", "summary": "The paper introduces SRFT, a single-stage fine-tuning paradigm that combines supervised fine-tuning (SFT) and Reinforcement Learning (RL) through an entropy-aware weighting mechanism. Instead of applying SFT and RL sequentially, SRFT jointly optimizes both objectives. The approach introduces a combined Loss function that integrates SFT, off-policy RL and on-policy RL components. The authors analyse the resulting learning dynamic and compare singe-stage versus two-stage training. The experiments are based on Qwen2.5-Math-7B, with ablations on two additional Qwen variants and show that SRFT outperforms standard SFT->RL pipelines.", "questions": "- **Q1:** Training stability\n   - (a) Reinforcement learning on smaller models (1.5B/7B) is often brittle and sensitive to hyperparameters. Could the authors comment on the training stability of SRFT compared to standard RL (e.g., GRPO)?\n   - (b) Introducing additional loss components can sometimes destabilize optimization. How does SRFT’s stability compare to pure SFT training in practice?\n- **Q2:** It is mentioned that math-verify is used for dataset filtering. Is it also used for the answer matching in the evaluation? If not, could the authors clarify which evaluation framework or procedure is used?\n- **Q3:** Generalization across models\n   - (a) Does SRFT also improve performance when starting from an RL tuned model (like Qwen2.5-Math-7B-Instruct)?\n   - (b) Were other model architectures tested (e.g. Gemma, LLaMa) to asses the whether SRFT benefits generalize beyond the Qwen family?\n- **Q4:** How does the computational cost of SRFT compare to the standard SFT->RL pipeline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Evaluation methodology is thoroughly reported and appears to follow best practices (e.g. multi-seed evaluation on high-variance benchmarks, ablation across prompt variants) which increases confidence in the robustness of the results.\n- Evaluation on both in-distribution and out-of-distribution benchmarks is good.\n- The authors include a thorough analysis of learning dynamics.\n- Multiple ablations are presented, including on entropy weighting factors and training on different Qwen-based model variants\n- Figures and Tables are well made\n- Motivation is clear", "weaknesses": "- The method is only tested on the Qwen model family.\n- The claims in section 3.1 seem to lack quantitative validation. (It is the result of one training run per category on Qwen2.5-Math-7B)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KD7b2rCelc", "reviewer": "Reviewer_KnF6", "summary": "This paper proposes SRFT, a single-stage framework that unifies SFT and RL via an entropy-aware weighting scheme: SFT provides coarse global shifts while RL delivers fine selective refinements, coordinated by policy entropy; SRFT applies both SFT and RL on demonstrations, decomposes the RL loss on rollouts into positive/negative parts; on Qwen2.5-Math-7B across iid and ood benchmarks, SRFT improves over zero-RL baselines by 9.0% and 10.9% and maintains more stable policy entropy during training.", "questions": "1. Justification and robustness of entropy coefficients. What motivated $w_{\\text{SFT}}=0.5$ and $w_{\\text{RL}}=0.1$ beyond heuristic tuning? Please include sensitivity analyses spanning orders of magnitude, or consider replacing fixed constants with a learned/monotone mapping from entropy to weights.\n2. Balancing the three objectives. Since $L_{\\text{SRFT}} = L_{\\text{SFT}}^{\\text{demo}} + L_{\\text{RL}}^{\\text{demo}} + L_{\\text{RL}}^{\\text{self-rollout}}$, have you tried introducing explicit coefficients (e.g., $\\alpha$,$\\beta$) and ablating their ratios to clarify relative contributions?\n3. Interpreting the response-length gap in Figure 6. RL-only produces notably shorter outputs. Is this primarily because RL-only lacks a demonstration warm start while SRFT leverages expert traces (e.g., DeepSeek-R1)? If RL-only were initialized on the same demonstrations, would the reward and length curves approach SRFT?\n4. Compute accounting and scalability. Please report compute and memory for SRFT vs. RL-only under the stated setup (500 steps, 8 rollouts/prompt): total and per-step FLOPs, peak memory, effective batch sizes, sequence lengths, and tokens processed. This would clarify fairness and inform scalability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Grounded in an entropy-based view of SFT–RL integration, the paper frames SFT as a coarse, global shift and RL as a fine, selective refinement, then unifies them in a single-stage training scheme that balances both with entropy-aware weighting. \n2. Methodologically clean and well-motivated: on demonstrations, SRFT applies SFT and RL jointly with an entropy-decayed SFT weight; on rollouts, it decomposes the RL loss into positive/negative components; the three parts are combined into a single, explicit loss. \n3. The evaluation is comprehensive: based on Qwen2.5-Math-7B, SRFT is tested on five competition-level math benchmarks and three OOD sets, using standard metrics (avg@32 / pass@1). Table 2 shows that SRFT achieves the best in-distribution and OOD averages. \n4. Training dynamics further support the design: SRFT maintains more stable policy entropy and smoother reward/length curves than RL-only training, consistent with the paper’s motivation.", "weaknesses": "1. Under-justified entropy weights. The entropy-aware coefficients are hard-coded to $w_{\\text{SFT}}=0.5$ and $w_{\\text{RL}}=0.1$. The paper notes they “can be fine-tuned” but offers no rationale or analysis for these choices. \n2. Unclear compute parity in training-dynamics comparisons. SRFT optimizes three losses—demo-SFT, demo-RL, and self-rollout RL—over two data streams, trained for 500 steps with 8 rollouts per prompt. Figure 6 compares reward, response length, and entropy to RL-only, but it is not stated whether tokens, FLOPs/GPU-hours, or memory were matched, making it hard to attribute gains to the algorithm rather than extra compute.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)