On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
Authors: Yongliang Wu
Yizhou Zhou
Zhou Ziheng
Yingzhe Peng
Xinyu Ye
Xinting Hu
Wenbo Zhu
Lu Qi
Ming-Hsuan Yang
Xu Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Supervised Fine-Tuning
Large Language Model
Reinforcement Learning
Primary area: reinforcement learning

Abstract: In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (DFT), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, \model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT.

Published as a conference paper at ICLR 2026
ON THE GENERALIZATION OF SFT: A REINFORCE- MENT LEARNING PERSPECTIVE WITH REWARD REC- TIFICATION
Yongliang Wu²* Yizhou Zhou2*t
Zhou Ziheng* Yingzhe Peng"
Xinyu Yea
Xinting Hu Wenbo Zhu Lu Qi
Ming-Hsuan Yang*
Xu Yang"
"Southeast University 'Independent Researcher University of California, Los Angeles "Shanghai Jiao Tong University "Nanyang Technological University
'University of California, Berkeley "Wuhan University
UUniversity of California, Merced vongliang0222 22mmmgg iommi.ccm, zyz0205@hotmail.cm xuyang palenepppalleeeee seu. edhu on
ABSTRACT
this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), address- ing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization ca- pabilities of model compared RL. To rectify this, we propose Dynamic Fine- Tuning (DFT), stabilizing gradient updates for each token by dynamically rescal- ing the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-mode tasks, demonstrating improved generalization. Additionally, DFT achieves competitive results in offline RI. settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state SFT. The source code will be available at https: / //g// hub. com/ yongliame mongliang-- Langgiiaggwu/DF..
INTRODUCTION
Supervised Fine-Tuning (SFT), which adapts models expert demonstrations, has become the standard post-training paradigm for Large Language Models (LLMs) (Zhang et al., 2025c; 2024a; 2025b:a; Wang al., 2025b). It enables efficient task adaptation and capability enhancement (Chung et al., 2024; Zhang et al., 2024c: Sanh et al., 2022: Ouyang et al., 2022; Chen et al., 2024; 2025a; Fang et al., 2025), and is popular for its ease of implementation and rapid acquisition of expert- like behaviors (Wei et al., 2022; Zhou et al., 2023). Despite these advantages. SFT often shows limited generalization compared to reinforcement learning (RI.) (Chu et al., 2024; Ouyang et al., 2022: Christiano et al., 2017; Bai et al., 2022; Huan et al., 2025; Swamy et al., 2025). RL lever- ages explicit reward or verification signals to explore diverse strategies and thus generalizes better. However, RL requires substantial computation, careful hyperparameter tuning, and explicit reward signals-condition often impractical in real-world settings (Schulman et al., 2017; Ouyang et al., 2022: Sheng et al., 2025; Strubell et al., 2019; Liu & Yin, 2024; Winsta, 2025). Moreover, RL can struggle to recover expert-like behaviors that SFT captures efficiently (Mandlekar et al., 2022; Chen al., 2025c).
To exploit the complementary strengths of both approaches, many hybrid methods combine SFT with RL (Ouyang al., 2022; Sheng al., 2025; Rafailov et al., 2023; Liu al., 2025; Qiu al., 2025). Yet a key question remains: can SFT itself be fundamentally improved? This is crucial, as SFT remains the only viable option when datasets contain only positive demonstrations, with no negative samples or reward model available.
"Equal Contribution. 'Project Leader. 'Corresponding Author.

---

Published as a conference paper at ICLR 2026
In this work, we address this gap with mathematical analysis of the connection between SFT and RL. We show that the gradient update in SFT can be interpreted as a form of policy gradient with specific, implicitly defined reward under certain assumptions. Crucially, this reward is (i) sparse, and (ii) inversely proportional to the model's probability of expert actions (see equation 6). As a result, when the model assigns low probability to expert actions, the gradient becomes excessively large, yielding an ill-posed reward structure and unstable optimization (Pascanu et al., 2013; Yang al., 2019).
Building on this insight, we propose Dynamic Fine-Tuning (DFT), a principled fix. Our method rescales the SFT objective at each token by its probability, canceling the distortion introduced by pierrse-robbabiiity weighting. This reframing turns the SFT gradient from a potentially unstable and biased estimator into a more stable, more uniformly weighted update rule that behaves closer to an RI.-stlee.
Empirically, DFT delivers substantial improvements. On the Qwen-2.5- 2.5t series (Qwen Team et al., 2024b) fine-tuned with NuminaMath-CoT (LI et al., 2024), DFT yields gains several times larger than standard SFT. More importantly, unlike SFT, which often degrades on challenging bench- marks such as OlympiadBench (He et al., 2024), AIME 2024 (American Institute of Mathemat- ics, 2024), and AMC 2023 (Mathematical Association of America, 2023), our method consistently improves performance and generalization. These improvements hold across models, scales, and data sizes (Table 1, Figure 1), and extend to code generation and multimodal reasoning (Tables 3, 4) (Zhao al., 2025d; Luo et al., 2025; Li et al., 2025a;b) (Zhu et al., 2024b;c;a; 2020).
We further test DFT in off-policy RL settings (Table 2). where dense rewards are available (Levine al., 2020). Our method not only outperforms offline RL approaches such as DPO (Rafailov al., 2023) and RAFT (Dong et al., 2023; Ahn et al., 2024), but also achieves competitive or superior performance to online methods like GRPO and PPO on math tasks with Dwen2.5-Math--1.5. (He et al.. 2025; Tan et al., 2026; Zhao et al., 2025b;a). Unlike these RL methods, DFT requires neither a reference model nor large batch sizes, making a simpler and more resource-efficient alternative.
To understand its effect, we analyze token probability distributions after training (Figure 2). While traditional SFT uniformly pushes probabilities toward the training set, DFT selectively increases some while reducing others. In particular, the proportion of less strongly fitted tokens rises, suggest- ing improved regularization. We provide further discussion in Appendix A.3.
The contributions of this work are theoretical and practical. On the theoretical side, we mathemati- cally establish L.MM SFT as special RI. in policy gradient space, pinpoint the underlying reasons for the limited generalization of SFT, and derive method to improve it. On the experimental side, we show that such a simple solution, just one line of code, can enhance the performance and gener- alization capabilities of SFT across various tasks and models.
2 RELATED WORK
The trade-off between supervised fine-tuning (SFT) and reinforcement learning (RL) is central to the alignment of large language models (Song et al., 2024; Chai et al.,, 2024; Song et al., 2025c;a; Xu et al., 2025; Song et al., 2025b; Zhu et al., 2025b;caa). SFT is widely adopted due to its simplicity and efficiency in imitating expert demonstrations (Chung al., 2024; Zhou al., 2023; Wei et al., 2022), analogous to behavioral cloning in robotics (Sammut, 2011: Mandlekar et al., 2022). However, the literature consistently highlights its limitations, particularly the tendency to overfit and generalize poorly compared to RL, which leverages reward signals to discover more robust policies (Ouyang al., 2022; Christiano al.. 2017: Bai et al., 2022; Swamy et al., 2025; Zhang et al., 2025d). A recent systematic comparison by Chu et al. (2024) across textual and visual domains confirms this distinction, concisely summarized as "SFT memorizes while RL generalizes." " They further show that SFT remains indispensable as initialization step, stabilizing output formatting prior to effective RL training. Nonetheless, RL faces significant practical hurdles, including computational expense, sensitivity to hyperparameters, and the requirement of an explicit reward function, all of which constrain its applicability (Schulman et al., 2017; Strubell et al., 2019; Sheng et al., 2025).
To combine the strengths of both paradigms, much recent work has pursued hybrid approaches. The most common strategy involves SFT pretraining followed by RL-based refinement with a learned reward model, as popularized by InstructGPT (Ouyang al., 2022). More recent methods interleave
2

---

Published as a conference paper at ICLR 2026
SFT and RL updates to improve stability and performance (Sheng al., 2025; Liu et al., 2025; Qiu et al., 2025). Other approaches, such as Direct Preference Optimization (DPO) (Rafailov et al,, 2023), bypass reward modeling entirely by directly optimizing policies on preference data, thereby unifying imitation and reinforcement signals within a single loss function. Chen et al. (2025b) introduce Negative-aware Fine-Tuning (NFT), which models incorrect generations via implicit negative policy, enabling sell-inppovement without explicit feedback. While powerful, these meth- ods rely on reward signals, preference pairs, or negative samples. They enrich the training pipeline but do not fundamentally improve SFT in its native setting, where only positive demonstrations are available. Our work instead focuses on enhancing SFT itself without requiring external feedback.
A complementary line of theoretical research seeks to unify SFT and RL under a common formal- ism. Du et al. (2025) reinterpret RI.F as a reward- rewghe-weighted weighted variant of SFT, preserving reliance on an explicit reward. Wang et al. (2025a) show that SFT can be cast as RI. with an implicit reward, proposing adjustments such as smaller learning rates to manage the vanishing KI. constraint, Abdol- maleki et al. (2025) analyze learning from both positive and negative feedback, studying how their balance affects convergence. Qin & Springenberg (2025) view SFT as lower bound of RL and introduce importance weighting based on the generating policy. While these works establish connections between SFT and RL through weighting, they do not provide a precise mathematical equivalence between the SFT gradient and the offline policy gradient. Some methods approximate this connection in practice by reweighting training losses. For instance, MixCE (Zhang et al., 2023) combines the forward and reverse KL divergences to form a unified objective, while GOLD (Pang & He, 2021) adopts offline RL with demonstrations, introducing reliance on an unknown demon- stration distribution Tú and restrictive 1/N assumption. Kantharaju & Sankar (2022) also provide clear and insightful exposition of GOLD's motivation and mechanics from an alternative perspec- tive, offering useful intuition for understanding its underlying design. Zhao al. (2025c) provide a promising method to combine RL and SFT during training. In contrast, our work offers more for- mal perspective on this connection, highlighting the role of the prvorse-pobbability weighting term in shaping the difference between SFT and RL-like updates. This perspective motivates simple adjustment: multiplying the loss by the model's 's token probability to neutralize the weighting.
Interestingly, method modifies the standard cross-entropy (CE) loss in way that inverts the weighting philosophy of the widely used Focal Loss (Lin et al., 2017). Specifically, our modified CE takes the form pogg(p), whereas focal loss is defined as -(1 p)" log(p.. Focal Loss delib- erately downweights well-classified samples to emphasize underrepresented or hard cases, whereas deliberately downweight poorly classified samples to encourage generalization. This inversion reflects a fundamental shift in the LLM era: while underfitting was once a central challenge, over- fitting and memorization now dominate, demanding a rethinking of objective design.
3 METHOD
3.1 PRELIMINARIES
Supervised Fine- muniTTnnng. Let = {(x, y²)) denote a corpus of expert demonstrations, where y" is the complete reference response to the query x. SFT minimizes the sentence-level cross-entropy:
(S) E (x [- log xo
(y* [=]]
(1)
Its gradient is:
(oLsFr(0) Ex
Vo log To (b x)).
(2)
Reinforcement Learning. Let y denote response sampled from the policy Ie(: I) for query x.
Given a reward function r(z, y) E IR, the policy objective is
(()) EanD, y> [r]
(3)
Its policy gradient at the sentence level is
(() EywDe g0)) Vo log Toly x)) r(x, v)]-
3.2 UNIFY SET AND RL GRADIENT EXPRESSION
Rewriting SFT Gradient as Policy Gradient via Importance Sampling. The SFT gradient
in equation 2 is taken under the fixed demonstration distribution. We convert it on-policy
3

---

Published as a conference paper at ICLR 2026
expectation by inserting importance weight that compares the expert (Dirac Delta) distribution with the model distribution.
1/y 4²] Th ( x)
E )=
[- Velog (b =)] ED Bynmml
Vo log To (o x)]
(5)
resample reweight
Define the auxiliary variables (importance sampling weight) as w(y x)) To ax) r(z, v) 1 - y"].
Reorganizing equation 5 and rewriting it using the above auxiliary variables, obtain the form
VoLsrT (0) = E. x><x((-)))))
[w[y
x)) Vo log (y r(
(6)
This form the SFT gradient closely resembles the policy gradient in Equation equation 4. Under this formulation, conventional SFT can be interpreted as on-policy gradient method, where the reward is a sparse indicator function matching the expert trajectory, but biased by an importance weighting term 1/TO- We emphasize that this RL-sttle characterization serves solely theoretical lens: both the analysis and subsequent modifications are developed within the RL framework, while the final method remains fully implementable in standard SFT form for computational efficiency. Detailed derivations are provided in Appendix A.2.
Due to the inherently sparse reward signal in the SFT setting, we identify the importance weight 1/To as a key contributor to SFT's generalization limitations compared to RL. When the model as- signs low probability to the expert response, the resulting weight becomes excessively large, intro- ducing an ill-posed reward landscape. This leads to disproportionately large gradients and training instability. The issue is compounded by the fact that the reward function r(z, y) = [ = y is non- zero only for exact matches to the expert outputm causing optimization to overfit rare exact-match samples and weakening the model's s ability to generalize beyond the training data.
3.3 PROPOSED METHOD
Reward Rectification via Dynamic Reweighting. To neutralize the skewed reward issue identi- fied when viewing SFT under the RI. RI. objective, we dynamically reweight the reward by multiplying by corrective inverse ratio given by the policy probability 1/w. The resulting "dynamically fine- tuned" gradient is then
Vo -E D., yor (-lx) [sg( w(y x)) Vo log To x)) (2x,())))
where sg(-) denotes the stop gradient operator, ensuring that gradients do not flow through the reward scaling term w. To facilitate transition later equations, we directly write 1/w be тe (vy xx) instead of (yy .)) because the indicator function in equation 5 or equation 6 would leave all cases where y # y² is Now since the gradient does not flow, the corrected SFT loss also becomes simple reweighted loss, called Dynamic Fine-tuning (DFT).
PDF (0) Ee )~D [ sg (mo(y" (x)) log To(y" |m)]
However, in practice, computing importance weights over the entire trajectory can induce numerical instability. common treatment of this issue to simply apply importance sampling at the token level, was adopted in PPO (Schulman et al., 2017). This leads to the final DFT loss version:
(DrT(() E (a.ss
sg (To(vt v))) log (o(i )
D
Note that the reward of this corrected SET (in RL form), ie., DFT, now becomes uni- formly for all expert trajectory. This is akin to contemporary verification based reward approach
4

---

Published as a conference paper at ICLR 2026
RLVR (DeepSeek-Al et al., 2025) that assigns uniform reward to all correct samples. Consequently, it avoids over weerconneentrtion on specific low-rrobability reference tokens, leading to more stable up- dates and improved generalization without introducing any additional sampling reward models.
4 EXPERIMENTS
We design four groups of experiments to comprehensively evaluate DFT. We first study the standard SFT setting on mathematical reasoning tasks to establish its core advantage over SFT (Section 4.1). )). We then extend to an offline RL setting, comparing DFT with representative offline and online RL methods (Section 4.2). To test cross-domain robustness, we further examine DFT on code generation benchmarks (Section 4.3) and its applicability to multi-modal reasoning math datasets (Section 4.4).
4.1 MAIN EXPERIMENT MATHEMATICAL REASONING TASK
To examine whether DFT can outperform vanilla SFT across tasks, architectures, and scales, we use mathematical reasoning as a representative testbed.
Implementation details. To efficiently manage computational resources, We andomly sample 100,000 instances from the the NuminaMath-Con dataset (IJI et al., 2024) for training. We con- duct experiments using multiple models, including Qwen2.5-Math-1.. 1.5B, Qwen2.5-Math--7 (Qwen Team et al., 2024a), LLaMA-3.2-3B. LLaMA-3.1-88 1-BB (Dubey et al., 2024), and DeepSeekMath- 7B (Shao et al., 2024). Our implementation builds upon the verl framework (Sheng et al., 2025), using recommended SFT prpeerpprammters... Specifically, we employ the Adam optimizer with learning rates of 5 10 for all models except the LLaMA-3.1-88 1----BB-8-BBBB for which adopt lower learning rate of 2 x 10 We set the mini-batch size to 256 and the maximum input length to 2048 tokens. The learning rate follows cosine decay schedule with a warm-up ratio of 0. We evaluate on benchmarks including Math500 (Hendrycks et al., 2021), Minerva Math (Lewkowycz al., 2022). Olympiad Bench (He et al., 2024), AIME 2024 (American Institute of Mathematics, 2024), and AMC 2023 (Mathematical Association of America, 2023) through the official Qwen2.5- Math evaluation pipeline (Qwen Team al., 2024a). Each model uses the default chat template and Chain-of- Thought (CoT) prompting to stimulate step-by-ster reasoning. All reported results repre- sent average accuracy across 16 decoding runs, evaluated with a temperature of 1.0 and maximum generation length of 4096 tokens.
DFT consistently yields average performance improvements over base models compared standard SFT across all benchmarks. Table I shows that, for S.een..5-Math-1.55 DFT achieves an average gain of of5.66.6 66 points over the base model, which is over 5.9x larger than the +2.09 point improve- ment from SFT. This pattern generalizes across other model families and sizes: LLaMA-3.2-3B benefits from a +3.46 point gain with DFT, exceeding the SFT gain (+2.05) by approximately 1.4x; LLaMA-3.1 3.1-BBB11-BBBB achieves +10.02 from DFT, surpassing SFT's +5.33 by 1.88x; DeepSeekMath-7B sees +15.51 point improvement via DFT, which is 1.58x larger than SFT's +7.18, and Qwen2.5- Math-7B reaches +15.90 point gain, nearly 3.8x higher than the SFT improvement of +2.37.
DFT demonstrates generalization and robustness, especially on challenging benchmarks where stan- dard SFT yields minimal or even negative impact. For instance, on Olympiad Bench, SFT degrades performance for Swen2.5-Math--155
dropping accuracy from 15,88 to 12.63, while DFT boosts 27.08, +11.20 point improvement over base model. On AIME24, SFT reduces accuracy for Qwen2.5-Math-77 by 4.20 points (from 6.68 to 2.48), whereas DFT improves performance to 8.56, achieving +1.88 point gain over the base model despite the difficulty of the benchmark. similar trend is observed on AMC23. SFT reduces the performance of 2w-ah.5-Math-11 18.75, while DFT raises 38.1 13, a +18.75 point gain over base. For Qwen2.5-Math-775 SFT yields only marginal improvement (+1.86), whereas DFT achieves +17.04 point gain. These results underscore that DFT not only scales more effectively across models of varying capacities, but also exhibits better resilience on difficult reasoning tasks where traditional SFT struggles.
from 19.38 to
DFT exhibits better learning efficiency and faster convergence characteristics. Figure reveals clear differences in learning dynamics between DFT and standard SFT Qwen2.5-Mate Ma5 across all math reasoning benchmarks. Compared to SFT, our method demonstrates three distinct advantages: (1) Faster convergence, achieving peak performance within the first 1 120 training steps
5

---

Published as a conference paper at ICLR 2026
Table I: Average @ 16 accuracy of five state-of-the-ar large language models mathematical rea- soning benchmarks. The best performance of each model across benchmarks is bold.
MathS00
Minerva Math
Olympiad Bench
AIME24
AMC23
Avg.
LLaMA-3.2-3B
1.63
1,36
1.01
0.41
1,56
1.19
LLaMA-3.2-33 w/SFT
8.65
2.38
2.06
0.00
3.13
3.24
LLaMA-3.2-3B w/DFT
12.79
2.84
2.90
0.83
3,91
4.65
LLaMA-3.1-8B
1.86
0.98
0.94
0.21
1.01
1.00
LLaMA-3.1-8B w/SFT
16.85
5.78
3.88
0.00
5.16
6.33
LLaMA-3.1-8B w/DFT
27.44
8.26
6.94
0.41
12.03
11.02
DeepSeekMath-7B
6.15
2.15
1.74
0.21
2.97
2.64
DeepSeckMath-78 W/SFT
26.83
7.26
6.33
0.41
8.28
9.82
DeepSeekMath-P7 w/DFT
41.46
16.79
15.00
1.24
16.25
18.15
Owen2.5-Math-1.35
31.66
15.88
4.16
19.38
15.92
Qwen2.5-Math-1155 w/SFT
43.76
13.04
12.63
1.87
18.75
18.01
Qwen2.5-Muth-1.5B w/DFT
27.08
6.87
38.13
31.58
Qwcn2.5-Math-7B
40.12
14.39
17.12
6.68
27.96
21.25
Qwen2.5-Math-78 w/SFT
53.96
16.66
18.93
2.48
26.09
23.62
Qwcn2.5-Muth-7B w/DFT
68.20
30.16
33.83
8.56
45.00
37.15
on most benchmarks; (2) Better early-stage performance, with DFT already outperforming best final accuracy of SFT within the first 10-20 steps; and (3) Higher sample efficiency, consistently requiring fewer updates to reach relatively optimal results. This accelerated convergence shows that the dynamic reweighting mechanism in DFT leads more informative gradient updates, guiding the model toward high-quality solutions early training. It also suggests that DFT helps avoid the optimization plateaus noise-prone regions often encountered in standard SFT, thereby enabling more efficient acquisition of complex mathematical reasoning patterns.
We also report the results of parameter-efficient fine-tuning (PEFT) training setting (Hu et al., 2022) and training on the OpenR 1 -Math dataset (Hugging Face, 2025) with better quality in Ap- pendix A.6 and Appendix A.5, respectively. Comparison and Discussion with the concurrent method iw-SFT (Qin & Springenberg, 2025) is provided in Appendix A.4.
4.2 EXPLORATORY EXPERIMENT OFFLINE RL SETTING
Equation shows that SFT suffers from reward sparsity, since in a constructed dataset each query x has only single reference answer y² From the perspective of RL, RFT/RAFT (Dong et al., 2023; Ahn et al., 2024) can be viewed as alleviating the sparse reward issue by effeetively increasing reward density, thereby enhancing model performance. Motivated by this observation, conduct an exploratory study applying DFT in an offline RL setting, where the reward sparsity problem is inherently less severe compared standard SFT, further validate the effectiveness.
Implementation details. We sample responses for 100,000 math questions using a temperature of 1.0 and generate four responses per question from the base model itself. Correct responses are iden- tified using math verify and retained as training data, resulting in approximately 140,000 examples. For DPO training, we construct 100,000 positive-negatee sponses. We compare DFT with representative offline RL methods, including DPO (Rafailov et al., 2023) and RFT (Dong et al., 2023; Ahn et al., 2024), as well as online RL methods PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024). For RFT and DFT, the training setup follows the con- figuration in Section 4. For DPO, we use the ms ms-swift (Zhao et al., 2024) with a learning rate of 1 x 1 batch size of 128, and warmup ratio of 0.05. For PPO and GRPO, training is performed using the verl (Sheng et al., 2025) with a learning rate x 10 batch size of 256, and warmup ratio of 0.1. We set the number of response 4 for GRPO.
preference pairs from the generated re-

---

Published as a conference paper at ICLR 2026
Mara 500
Minerva Math
Oly mppiaad Bench
20-
169
1.5
259
100
256
30
160
250
Skap
Averuge
20
109
1.5
24
28
100
254
250
300
24
189
1.5
286
200
46
Skep
Figure 1: Accuracy progression for Qwen2.5 -012255--hh---1 across mathematical benchmarks, illustrat- ing faster convergence and better performance achieved by DFT relative to SFT.
Table 2:
Evaluation results on mathematical reasoning benchmarks an offline reinforcement learn- ing setting using reward signals from rejection sampling. The best performance in bold.
Setting
Math500
Minerva Math
Olympiad Bench
AIME24
AMC23
Avg.
Qwen2.5-Math-1.55 w/DFT
64.89
20.94
27.08
6.87
38.13
31.58
Qwen2.5-Math-1.5B w/DPO
Offline
46.89
11.53
22.86
4.58
30.16
23.20
Qwen2.5-Muth11.53 w/RFT
Offline
48.23
14.19
22.29
4.37
30.78
23.97
Qwen2.5 1ree225-Math-1.35 1.55 w/PPO
Online
56.10
15.41
26.33
7.50
37.97
28.66
Qwen2.5-Math-1150 Ma5h w/GRP
Online
62.86
18.93
28.62
8.34
41.25
32.00
Qwen2.5 Maten255-Mtth11.. M.5h-1158 w/DFT
Offline
64.71
25.16
30.93
7.93
48.44
35.43
DFT demonstrates competitive performance in the offline RL setting, outperforming both offline and online RL baselines. Table shows DFT achieves an average score of 35.43, exceeding the best offline method RFT by +11.46 points, and even outperforming the strongest online RL algorithm GRPO by +3.43 points. Specially, on Math500, DFT scores 64.71, slightly ahead GRPO (62.86) and better than PPO (5).10) and RFT (48.23). The gains also notable more challenging benchmarks: on AMC23, DFT achieves 48.44, a +7.19 point margin over GRPO and +17.66 point gain over RFT. Similarly, Minerva Math, DFT reaches 26.16, outperforming GRPO by +6.23 points, PPO by +9.75, and all offline baseline methods.
These results highlight the strength of DFT as simple yet effective fine-tuning strategy. Despite its lack of iterative reward modeling or environment interaction, provides a stronger learning signal than both offline methods like DPO/RFT and online policy optimization algorithms like PPO/GRPO in certain scale train set. This suggests that DFT can serve as a more efficient and scalable alternative traditional RL pipelines, particularly domains where preference supervision available but reward modeling or online response sampling is expensive or impractical.
4.3 EXPLORATORY EXPERIMENT CODE GENERATION TASK
Implementation details. We adopt UltraFeedback (Cui et al., 2024) as the training dataset. From this corpus, sample 10,000 prompts and, for each prompt, select the response with the highest average score to perform supervised fine-tuning (SFT) (Du et al., 2025). Model performance is assessed on three widely used code generation benchmarks: HumanEval (Chen et al., 2021), )), Hu-

---

Published as a conference paper at ICLR 2026
Performance of various models code generation benchmarks. The best performance for each benchmark is highlighted in bold.
Table 3:
HumanEval
MultiPL-E
IIE
HEE+
Python
C++
Java
PIIP
TS
C#
Bash
JS
Avg.
Qwen2.5-3B
43.3
36.0
43.29
40.99
37.34
37.89
47.17
43.04
40.05
Qwen2.5-3B W/SFT
41.5
34.8
42.07
42.24
37.97
37.27
43.40
41.77
20.25
47.83
39.10
Qwen2.5-3B w/DFT
45.7
39.0
45.73
44.72
41.77
45.34
42.14
43.04
27.85
44.10
41.84
@wen2.5-Coder-3B
52.4
42.7
51.83
53.42
46.20
47.20
54.09
$5.06
25.32
54.04
48.39
Qwen2.5-Coder338 WSFT
51.8
43.9
51.22
51.55
48.10
54.66
59.12
51.27
34.18
54.04
50.52
Qwen2.5-Coder-38 w/FT
56.7
50.0
57.32
54.66
51.27
58.39
58.49
60.76
31.01
53.42
53.16
Qwen2.5-Codee-7B
62.2
53.0
63.41
63.98
53.16
59.01
62.89
59,49
39.24
60.87
57.76
Qwen2.5-Code=-7B w/SFT
54.9
48.8
54.88
64,60
51.27
62.11
68.55
60.76
33.54
65.22
57.62
Qwen2.5-Coder-7B w/DT
67.7
59.8
67.68
67.70
54.43
60.87
70.44
65.19
48.73
63.35
62.30
Table 4: Performance comparison across different multi-modal reasoning benchmarks. performance on each benchmark is highlighted in bold.
The best
Math Verse
Math MathVision
WeMath
Vision Only
Vision Intensive
Vision Dominant
Overall
Qwen2.5-VL-3h
28.81
30.96
31.60
33.83
21.25
4.10
DWwn2.55VLL-3B w/SFT
30.96
33.63
32.74
35.66
21.02
23.33
Qwen2.5-VL-3B w/DFT
32.49
35.91
33,50
37.54
22.30
23.71
manEval+ (Liu al., 2023), and MultiPL-E (Cassano et al., 2023). Training is conducted for one epoch with a learning rate of 5 x 10-5, warm-up up ratio of 0.05, and a batch size of 16.
Table 3 shows DFT achieves improvements in most cases compared to both base models and SFT. For Qwen2.5-3B, DFT raises HumanEval from 43.3 to 45.7 and HumanEval+ from 36.0 to 39.0, with the MultiPL-E average also increasing from 40.05 (base) and 39.10 (SFT) 41.84. Simi- lar trends are observed for Qwen2.5-Coder-3B where DFT improves HumanEval to 56.7 and Hu- manEval+ to 50.0, outperforming both base and SFT. For Qwen2.5-Coder-77 DFT reaches 67.7 on HumanEval, 59.8 on HumanEval+, and 62.3 average MultiPL-E, surpassing SFT by +12.8, +11.0, and +4.7 points respectively. The overall trend demonstrates that DFT generally provides stronger performance across different models and languages.
4.4 EXPLORATORY EXPERIMENT MULTI- REASONING
Implementation details. We use the WeThink dataset (Yang et al., 2025) for training. The model is fine-tuned using LLaMA-Factory (Zheng et al., 2024) and evaluated with VLMEvalKit (Duan al., 2024). We train the model for epoch with a learning rate of 5e-5. To comprehensively sess reasoning capabilities, we adopt suite of multi-modd modal reasoning benchmarks including Math- Verse (Zhang et al., 2024b), Math Vision (Wang et al., 2024), and WeMath (Qiao et al., 2024) for evaluation.
DFT achieves consistent improvements over base models and SFT across all multi-modal reason- ing benchmarks. Table 4 shows, on Math Vatheerse, DFT boosts Qwen2.5-Vi 5-LBB from 33.83 to 37.54 average accuracy, outperforming the SFT gain of only +1.83 by +3.71 points. Consistent improve- ments are observed across all major vision-related subcategories. On MathVision, DFT improves performance from 21.25 (base) to 22.30, exceeding SFT which fails to provide gains (21.02). On WeMath, SFT already yields +19.23 point gain, but DFT pushes performance slightly further to 23.71, maintaining superiority over both base and SFT. These results indicate that DFT not only strengthens text-only reasoning but also extends effectively multi-modal domains.
8

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of weighting strategies on mathematical reasoning benchmarks.
Math500
Minerva Math
Olympiad Bench
AIME24
AMC23
Avg.
Swen2.5-Math-1e.
31.66
8.51
15.88
4.16
19.38
15.92
Sentence- Weighting
31.26
8.05
16.47
3.12
19.84
15.75
Geometric-Men Weighting
42.87
12.34
13.03
1.23
16.56
17.21
Token-Level Weighting
64.89
20.94
27.08
6.87
38.13
31.58
4.5
LIMITATIONS OF DFT: A CASE STUDY ON FACTUAL KNOWLEDGE
While DFT consistently outperforms SFT reasoning-heae heavy tasks, it may not always be the better choice, particularly in factual knowledge domains. We conduct an exploratory experiment on the Natural Questions dataset (Kwiatkowski al., 2019), which consists of real-user, open-domain factual queries grounded in Wikipedia articles.
In this setting, we find that SFT improves performance from 31.24% 36.62%, while DFT un- expectedly reduces it to 14%. This result reveals an important limitation of DFT: because it reweights samples based on the model's own confidence, tends reinforce the model's existing beliefs. When the model lacks sufficient factual knowledge, such reinforcement may hinder effective learning instead of facilitating it.
This case suggests that DFT is most effective when the task aligns well with the model's prior competence, such as logical reasoning or structured prediction. In contrast, when the objective is to absorb new factual information, especially in domains beyond the model's current capabilities, SFT remains a more reliable and stable fine-tuning strategy.
4.6 AN EMPIRICAL COMPARISON WITH SENTENCE-LEVEL WEIGHTING
Our framework applies confidence-based weighting at the token level. While this design was pri- marily motivated by numerical stability, we also compared it against two sentence-level variants to better understand their behavior.
The first variant uses the full sequence probability to scale the loss. However, these values are extremely small in practice, making the loss nearly uninformative and producing a highly skewed weight distribution that is difficult to tune. To address this, we also evaluated geometric-ment mean variant inspired by GSPO (Zheng al., 2025), which rescales sentence probabilities avoid nu- merical collapse. Although this version is more stable, it still provides weak training signal and offers limited performance gains.
As shown in Table 5, both sentence-level strategies lead to minimal changes over the base model, while our token-level formulation delivers substantial and consistent improvements, raising average accuracy from 15.92 to 31.58. These results demonstrate that token-level weighting provides more reliable optimization signal and significantly stronger empirical performance.
4.7 ANALYSIS OF PROBABILITIES
To understand how the model trained by DFT is different from SFT and other RL methods, look into the token probability distribution of the model's output over the training set in Figure 2. SFT tends to uniformly increase token probabilities, shifting the entire distribution towards higher confidence, but mainly targeting the lower and lowest probability tokens. The highest probability token portion barely increases. stark contrast, DFT exhibits polarizing effect: significantly boosts the probabilities of subset of tokens while actively suppressing the probabilities of oth- ers. This leads to a bimodal distribution, with more tokens occupying both the highest and lowest probability bins. Other RL methods such DPO, GPPO and PPO show the same trend DFT. although the scale is much milder than it. We look into the words that belong to the lowest proba- bility bin. and find that they are generally the conjunctive words or punctuations such 'the", 'let', etc. These results suggest that for robust learning, models should not attempt to fit all tokens with uniform confidence. It may be beneficial to deprioritize fitting tokens that serve grammatical

---

Published as a conference paper at ICLR 2026
Probubility Distribution
100
Metted
Base
DPO
30
PPO
DFT
Percentag of Tokons (%)
0.3
0.1
80 80-0m00l
co01-00000
8a8)-0at
a0-411
81.02
42-42
03.44
04-83
0.5-0
0d641
0,7 0.8
0.8-419
63-10
Prediction Probahility
Figure 2: Token probability distributions on the training set before training and after fine-tuning with DFT, SFT, and various RL methods. A logarithmic scale used on the y-axis for clarity.
functions rather than carrying primary semantic content. This concept is analogous human peda- gogy, where students are taught to focus on substantive concepts rather than perfecting the usage of common connective words. Further analysis be found in Appendix A.3.
CONCLUSION
In this work, we revisit the well-known generalization gap between SFT and RL. We offer the- oretical perspective showing that the standard SFT gradient can be interpreted as policy gradient with ill-posed, implicitly defined reward inversely related to model confidence. This formulation helps explain the instability and limited generalization observed in SFT training. Motivated by this analysis, introduce DFT, simple yet effective method that dynamically reweights the SFT loss using the token probability. This one-line change improves gradient stability and leads to better gen- eralization. Our empirical results show that DFT consistently improves over standard SFT across a range of models and challenging mathematical reasoning tasks. Beyond supervised settings, we adapt DFT to offline RL scenarios and find that it outperforms several established online and offline RL baselines, suggesting broader applicability. Overall, this work contributes both refined under- standing of SFT's limitations and lightweight, practical method that helps bridge the gap more complex RL-based approaches.
Limitations. While our experiments demonstrate the effectiveness of DFT on mathematical rea- soning benchmarks and code generation tasks. the evaluation scope remains limited. We have not yet assessed its performance on broader task categories or with larger-scale LLM, which we leave for future exploration. Moreover, DFT can not offer universal benefits across all scenarios. In do- mains that primarily involve the acquisition of factual knowledge, conventional SFT still remains the most efficient approach. DFT may also not be an ideal choice for hard examples or domains under- represented in the training data, since it assigns low initial probabilities to such samples, reducing their learning weight. Our aim is not to assert that DFT universally outperforms SFT, but rather to offer perspective on objective design by analyzing the distinction between RL and SFT. Be- sides, an important future direction is to explore non- non-uniform or quality-aware reward assignments for demonstrations.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
Supported by Jiangsu Province Carbon Peak Carbon Neutrality Science and Technology Innovation Special Fund Project (Grant No. BT2025029), National Natural Science Foundation of China (Grant No. 62576091), and Big Data Computing Center of Southeast University.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our study does not involve human subjects, per- sonally identifiable information, proprietary data. All datasets used, including NuminaMath, OpenRI-Mathh UltraFeedback, and WeThink, are publicly available and documented in the ap- pendix. The proposed method is a simple training strategy that modifies gradient computation for improved generalization. It does not introduce any new capabilities that could cause harm, nor does enable misuse beyond the standard capabilities of existing large language models. We are not aware of any potential risks related to bias, fairness, security that arise specifically from the method proposed. Nonetheless, we acknowledge that like any fine-tuning strategy, DFT may inherit biases present in the underlying data model, and future research may explore safeguards for these scenarios. No conflicts of interest, legal compliance issues, or sponsorship-related influences are present in this work.
REPRODUCIBILITY STATEMENT
We have taken multiple steps ensure the reproducibility of our work. All datasets used in our experiments are publicly available and properly cited the main text and appendix. Training con- figurations, including model architectures. hyperparameters, optimizers, and evaluation settings, deseribed in detail in Section 4 and Appendi A.5-A.6. Theoretical claims, including the equiva- lence between SFT and policy gradient, are formally derived in Appendix A.2. Experimental results include multiple model scales, tasks, and training settings to validate robustness. A complete imple- mentation of our method is included in the supplementary material, along with scripts for reproduc- ing all reported results. We will release the full source code and training logs upon publication to further support reproducibility.
REFERENCES
Abbas Abdolmaleki, Bilal Piot, Bobak Shahriari, Jost Tobias Springenberg, Tim Hertweck, Michael Bloesch, Rishabh Joshi, Thomas Lampe, Junhyuk Oh, Nicolas Heess, et al. Learning from nega- tive feedback, or positive feedback both. ICI.R, 2025.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In EACLW, pp. 225-237, 2024. 2, 6
American Institute of Mathematics. Aime 2024 competition mathematical problems, 2024. 2,
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dasgupta, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Hase, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 2022. 1,
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl- e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering. 49(7):3675-366) 2023.
Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and new benchmark. arXiv preprint arXiv:2410.0300 2024. 2
Hongli Chen, Pengcheng Fang, Yuxia Chen, Yingxuan Ren, Jing Hao, Fangfang Tang, Xiaohao Cai, Shanshan Shan, and Feng Liu. difi-mamba: Dual-stream w-laplacian enhanced mamba for high-fidelity mri reconstruction. arXiv preprint arXiv:2508.09/77 099 2025a.

---

Published as a conference paper at ICLR 2026
Huayu Chen, Kaiwen Zheng, Qinsheng Zhang, Ganqu Cui, Yin Cui, Haotian Ye, Tsung- Yi Lin, Ming- Yu Liu, Jum Zhu, and Haoxiang Wang. Bridging supervised learning and reinforcement learning in math reasoning. arXiv preprint arXtv:2505.18116 2025b. 3
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint (ttv:2007.03377 2021.
Yuxia Chen, Pengcheng Fang, Xiaoling Zhong, Jianhui Yu, Xiaoming Zhang, and Tianrui Li. Hi- resnet: Edge detail enhancement for high-resolution remote sensing segmentation. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 17:1 17:15024-15040, 2024.
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang. Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang, and Ji-Rong Wen. An empiri- cal study eliciting and improving r1-like reasoning models. arXiv preprint a5X45:250..04588 2025c.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS, volume 30, 2017. 1,2
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. In ICML., 2024. 1,2
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling guage models. Journal of Machine Learning Research, 25(70):1-53, 2024. 1,2
lan-
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: boosting language models with scaled ai feedback. In ICML, pp. 9722-9744, 2024. 7
Dee,Seek-AL, Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan. S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He. Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu WU, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025. 5, 20
12

---

Published as a conference paper at ICLR 2026
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023, 2023. 2,
Yilun Du al. Simplify rhhf reward-weighted sft: A variational method. arXiv:2502.1 10026, 2025. 3,7
arXiv preprint
Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. ACM MM, pp. 11198-11201, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint preprimi arXiv:2407.21788 2024. 5
Pengcheng Fang, Yuxia Chen, and Rui Guo. When and what: Diffusion-grounded videollm with entity aware segmentation for long video understanding. arXiv preprint arXiv:2508. 15641, 2025.
Yoav Freund. A more robust boosting algorithm. arXiv preprint arXiv:0905.2133 2009. 19
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. ACL, pp. 3828-3850, 2024. 2,
Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, and Bo Zhang. Tempflow-grpo: When timing matters for grpo in flow models. arXiv preprint arXiv:2508.04:: 04324, 2025. 2
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurlPS, 2021. 5
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general Ilm capabilities? understanding transferability of Ilm reasoning, arXiv preprint arXiv:2507.00432, 2025.
Hugging Face. Open rl: A fully open reproduction of deepseck-rl, Janbuar January 2025. 6, 20
Pavan Kantharaju and Aiswarya Sankar. An understanding of learning from demonstra- tions for neural text generation. In ICLR Blog Track, 2022. URL https:// io 2ttp////tttttte-i----Eaa-- bhggttakkkghhbbbioo0002022222eee
bclr-blog-tzack.
https://ielr-
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions the Association for Computational Lingnistics, 7:453-466, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020. 2
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman- Solo, et al. Solving quantitative reasoning problems with language models. NewrIPS, 35:3843-3857, 2022. 5
Jia LI. Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang. Ziju Shen. Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath, 2024. 2,5
13

---

Published as a conference paper at ICLR 2026
Ouxiang Li, Yuan Wang, Xinting Hu, Huijuan Huang. Rui Chen, Jiarong Ou, Xin Tao, Pengfei Wan, Xiaojuan Qi, and Fuli Feng. Easier painting than thinking: Can text-to-image models set the stage, but not direct the play? arXiv preprint arXiv:2509.03516, 2025a.
Ouxiang Li, Yuan Wang, Xinting Hu, Houcheng Jiang, Tao Liang, Yanbin Hao, Guojun Ma, and Fuli Feng. Speed: Scalable, precise, and efficient concept erasure for diffusion models. arXiv preprint arXiv:2503.07392, 2025b.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, pp. 2999-3007, 2017.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. NeurIPS, 04221588-21722 2023.
Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. Uft: Unifying supervised and reinforcement fine-tuning, arXiv preprint arXiv:2505. 16984, 2025. 1,3
Vivian Liu and Yiqiao Yin. Green ai: exploring carbon footprints, mitigation strategies, and trade off's in large language model training. Discover Artificial Intelligence, 4(49), 2024.
Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24960-24971 1, 2025. 2
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In CoRI. pp. 1678-1690, 2022. 1,
Mathematical Association of America. Ame 2023 competition problems, 2023. 2,5
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. NeurIPS, 35:27730-277 277444, 2022. 1.2
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. ICLR, 2021.
3
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. ICML, pp. 1310-1318. Pmlr, 2013. 2
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multi- modal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024. 8
Chongli Qin and Jost Tobias Springenberg. Supervised fine tuning curated data is reinforcement learning (and can be improved). arXiv preprint urXiv:2507.12856 2025. 3. 6. 19
Haibo Qiu, Xiaohan Lan, Fanfan Liu, Xiaohu Sun, Delian Ruan, Peng Shi, and Lin Ma. Metis- rise: RI incentivizes and sft enhances multimodal reasoning model learning. arXiv preprint arXiv:2506.1300 2025. 3,3
Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2.5: A party of foundation models. arXiv preprint arXiv:2412.15115 2024a.
Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2.5 technical report. arXiv preprint /5/1: 2024b.
Rafael Rafailov, Archit Sharma. Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In NewrlPS, volume 36, 2023. 2,
14

---

Published as a conference paper at ICLR 2026
Claude Sammut. Behavioral Cloning. 2011.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutton, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. In ICLR, 2022.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In ICLR, 2020. 19
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017. 1,2, 4,
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 6,
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. In ECCV, pp- 1279-1297, 2025. 1, 2, 5,
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221-18232, 2024. 2
Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang. Video-mmlu: A massive multi-discipline lecture understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6099-6113, 2025a. 2
Enxin Song, Wenhao Chai, Shusheng Yang, Ethan Armand, Xiaojun Shan, Haiyang Xu, Jianwen Xie, and Zhuowen Tu. Videonsa: Native sparse attention scales video understanding. arXiv preprint arXiv:2510.02295, 2025b. 2
Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-awant sparse memory for long video question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025c.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACL, pp. 06645-656 2019. 1,2
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint arXiv:2503.01067 2025. 1.2
Xiaofeng Tan, Wanjiang Weng, Haodong Lei, and Hongsong Wang. Easytune: Efficient step-aware fine-tuning for diffusion-based motion generation. arXiv preprini arXiv:2602.07967. 202779977226.... 2
Bo Wang, Qinyuan Cheng, Runyu Peng, Rong Bao, Peiji Li, Qipeng Guo, Linyang Li, Zhiyuan Zeng. Yunhua Zhou, and Xipeng Qiu. Implicit reward the bridge: A unified view of sft and dpo connections. arXiv preprint arXiv:2507.0001 2025a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In NeurlPS, 2024. 8
Zhenglin Wang, Jialong Wu, Yilong Lai, Congzhi Zhang, and Deyu Zhou. Seed: Accelerating reasoning tree construction via scheduled speculative decoding. In Proceedings of the 31st Inter- national Conference on Computational Linguistics, pp. 4930,4937, 2025b.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, and Quoe Le. Finetuned language models are zero-shot learners. In ICLR, 2022. 1,
15

---

Published as a conference paper at ICLR 2026
Jenis Winsta. The hidden costs of ai: A review of energy, e-waste, and inequality in model develop- ment. arXiv preprint arXiv:2507.06611, 2025.
Weili Xu, Enxin Song, Wenhao Chai, Xuexiang Wen, Tian Ye, and Gaoang Wang. Auro- ralong: Bringing mns back to efficient open-ended video understanding. arXiv preprint wwXiv:2507.02591, 2025. 2
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. arXiv preprint 1xXiv:5504.149453 2025. 20
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. In ICLR, 2019. 2
Jie Yang, Feipeng Ma, Zitian Wang, Dacheng Yin, Kang Rong, Fengyun Rao, and Ruimao Zhang- Wethink: Toward peneral-purpose vision-language reasoning via reinforcement learning, arXiv preprint arXiv:2506.07900 07905, 2025,
Congzhi Zhang. Linhai Zhang, and Deyu Zhou. Causal walk: Debiasing multi-hop fact verification with front-door adjustment. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 19533-19541, 2024a.
Congzhi Zhang. Jiawei Peng, Zhenglin Wang, Yilong Lai, Haowen Sun, Heng Chang, Fei Ma, and Weijiang Yu. Vrest: Enhancing reasoning large vision-language models through tree search and self-reward mechanism. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp. 3922-3941, 2025a.
Congzhi Zhang, Zhibin Wang, Yinchao Ma, Jiawei Peng, Yihan Wang, Qiang Zhou, Jun Song, and Bo Zheng. Rewatch-rl: Boosting complex video reasoning in large vision-language models through agentic data synthesis. arXiv preprint vrXiv:2509.23662 2025b.
Congzhi Zhang, Linhai Zhang, Jialong Wu, Yulan He, and Deyu Zhou. Causal prompting: Debiasing large language model prompting based on front-door adjustment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 25842-25850, 2025c.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised IIm reasoning incentivization. arXiv preprint arXiv:2504.058/2, 2025d. 2
Renrui Zhang. Dongzii Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal 1lm truly see the diagrams visual math problems? ECCV, pp. 169-186, 2024b. 8
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction tuning for large language models: A survey. arXiv preprint a308v: 10792, 2024c. 1
Shiyue Zhang, Shijie Wu, Ozan Irsoy, Steven Lu, Mohit Bansal, Mark Dredze, and David Rosen- berg. Mixce: Training autoregressive language models by mixing forward and reverse cross- entropies. In ACL, pp. 9027-9050, 2023. 3
Kesen Zhao, Jiaxin Shi, Beier Zhu, Junbao Zhou, Xiaolong Shen, Yuan Zhou, Qianru Sun, and Hanwang Zhang. Real-time motion-controllable autoregressive video diffusion. arXiv preprint arXiv:2510.08 2025a.
Kesen Zhao, Beier Zhu, Qianru Sun, and Hanwang Zhang. Unsupervised visual ofain--f--hought reasoning via preference optimization. arXiv preprint arXiv 18399, 2025b.
Xiangyu Zhao, Junming Lin, Tianhao Liang, Yifan Zhou, Wenhao Chai, Yuzhe Gu, Weiyun Wang, Kai Chen, Gen Luo, Wenwei Zhang, et al. Mm-helix: Boosting multimodal long-chain reflec- tive reasoning with holistic platform and adaptive hybrid policy optimization. arXiv preprint 100.0:25100085440 3025c.
16

---

Published as a conference paper at ICLR 2026
Xiangyu Zhao, Peiyuan Zhang, Kexian Tang, Xiaorong Zhu, Hao Li, Wenhao Chai, Zicheng Zhang, Renqiu Xia, Guangtao Zhai, Junchi Yan, et al. Envisioning beyond the pixels: Benchmarking massonnggiinfomae visual editing. arXiv preprint arXtv:2504.02826. 2025d. 2
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scalable lightweight infrastructure for fine-tuning. 2024.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yugiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507. 18071, 2025. 9
Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan Ye Yanhan, and Zheyan Luo. Llamafac- tory: Unified efficient fine-tuning of 100+ language models. In ACL, pp. 400 410, 2024. 8
Chunting Zhou, Pengfei Liu, Puxin Xu. Srinivasan Iyer, Jianfeng Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. In NeurIPS, volume 36, 2023. 1, 2
Xingyu Zhu, Xu Xu, and Zhongfu Ye. Robust adaptive heamforming via subspace for interference covariance matrix reconstruction. Signal Processing, 167:107289, 2020. 2
Xingyu Zhu, Shuo Wang. Jinda Lu, Yanbin Hao, Haifeng Liu, and Xiangnan He. Boosting few-shot learning via attentive feature regularization. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 7793-7801, 2024a. 2
Xingyu Zhu, Beier Zhu, Tan, Shuo Wang. Yanbin Hao, and Hanwang Zhang. Enhancing zero- shot vision models by label-free prompt distribution learning and bias correcting. Advances in Neural Information Processing Systems, 37:2001-2025, 2024b.
Xingyu Zhu, Beier Zhu, Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Selective vision- language subspace projection for few-shot clip. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3848-388 3857, 2024c. 2
Xingyu Zhu, Shuo Wang, Beier Zhu, Miaoge Li, Yunfan Li, Junfeng Fang, Zhicai Wang, Dongsheng Wang, and Hanwang Zhang. Dynamic multimodal prototype learning in vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2501-2511, 2025a. 2
Xingyu Zhu, Beier Zhu, Yunfan Li, Junfeng Fang, Shuo Wang, Kesen Zhao, and Hanwang Zhang. Hierarchical semantic alignment for image clustering. arXiv preprint arXiv:2512.00904, 2025b.
Xingyu Zhu, Beier Zhu, Shuo Wang, Kesen Zhao, and Hanwang Zhang. Enhancing clip robustness via mross-modality alignment, arXiv preprint arXiv:2510.2403: 2025c. 2
17

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1
USAGE OF LLM
We employ LLM primarily as writing assistants to refine and polish the manuscript. Their usage was limited to improving clarity, coherence, and presentation, while all conceptual and experimental contributions remain original.
A.2 DETAILED DERIVATION OF EQUATION (5)
We start from the SFT gradient Equation (2):
V (oLr = (( Vo log To [x]]-
(1)
For each query the expectation over expert demonstrations (x, v²) can be written explicitly summation over possible outputs y::
E (( Va lolooga (y |x)] EUNDD Σ y'll V log (o x)).
We insert the model distribution Toly x)), which allows to express the summation in terms of importance weights:
To .)) 1 y'll Σ y
E E D.
Vo log To m)]-
To x)
Here, the term ne(ulz) serves as an importance weight comparing the expert (Dirac delta) distribu-
1|v=y"]
tion with the model's distribution.
The summation over can now be rewritten as an expectation under the policy distribution ~ To(: æ):
1 y'll (- Vo log no (
EgwD, Byywx(((a))
(4)
Toly x)
Thus, obtain Equation (5):
[ y']] T x)
Vo log 2))
[(ay)>D Vo log (v [x]] EEND By
(5)
This derivation shows that the SFT gradient can be expressed as an on-policy policy gradient with importance sampling, where the expert demonsttaiion distribution is reweighted relative to the model distribution.
A.3 DISCUSSIONS AND INSIGHTS
Gradient Analysis DFT. We now analyze the gradient induced by the DFT surrogate loss. Recall the sequence-level definition:
LDFT(O) -g--sg(TToo)) (To"To"" (x) log (v²" xx)
(10)
where sg(-) denotes the stop-gradient operator. Since the stop-gradient blocks backpropagation, the detached probability sg(Te( æ)) is treated as constant during differentiation. Consequently, the gradient becomes
VoLDET sg((T((" xx))
VOTO(Y" x))
(11)
Te x))
=- (o Vore(y" xx).
(se(xo(u"12))
(12)
18

---

Published as a conference paper at ICLR 2026
Since [)) equals (y x:) in the forward pass, the prefactor is numerically equal to 1. Therefore,
VoLDFT = VVTTaV" [x)
(13)
This shows that DFT is mathematically equivalent to directly maximizing the model probability of the target token, rather than its log- pog-prbbability as in cross-entropy.
For cross-entropy, the loss is
(C) = log no (3/ x),
yielding gradient
V VOLCE
Vomoly" x:).
my (x)
Thus both CE and DFT share the same gradient direction but differ in scaling: CE amplifies updates for low-probability targets (factor 1/π), while DFT applies a uniform factor 1. As result, DFT avoids the instability caused by excessively large gradients on unlikely expert tokens, providing more conservative and stable updates.
From the reinforcement learning perspective, the reward of DFT becomes uniformly 1 across all ex- pert trajectories, equivalent to verification-sty. objective that treats all correct references equally. From the optimization perspective, DFT trades off aggressive fitting rare tokens for better stabil- ity and calibration. Practically, this explains why DFT often yields smoother training and stronger generalization, while maintaining alignment with the pre-training distribution.
Learning from Noisy Data. DFT offers a simple yet effective approach, prompting us to reflect on why it might actually work. One intuitive explanation lies in its ability to learn from noisy data (Fre- und, 2009). Sasaki & Yamashina (2020) propose an imitation learning algorithm for learning from noisy demonstrations, based on the core idea of avoiding the fitting of data that difficult model, as such data is likely to originate from suboptimal behaviors, i.e., noise. Their method introduces weighted behavioral cloning objective, where the weights are derived from a previously trained policy's 's confidence in each action. Similarly, the weighting mechanism in DFT shares the same intuition, but instead of relying on a fixed old policy model to compute confidence scores, it uses a single policy model to perform confidence-bartd based weighting on-the-ffy ffy during training.
A.4 COMPARISION WITH CONCURRENT WORK IW-SFF WFFT
We include a concurrent method, Importance- Weighted SFT (iw-SFT) (Qin & Springenberg, 2025), for comparison. All training settings follow those reported in the original paper, except that we set the number of training epochs to
As shown in Table 6, DFT achieves higher average accuracy than iw-SFT on most model fami- lies: LLaMA-3.2-3B (+2.39), LLaMA-3. 1-B (+4.15), DeepSeekMath-75 (+3.34), and Qwen2.55 Math-1.5B (+1.30). Although iW-SFT outperforms improvement is not consistent across datasets. In particular, for LLaMA-3.2-333 iw-SFT under- performs standard SFT Math500 (5.13 vs. 8.65) and AMC23 (2.03 3.13). Similarly, for LLLMMA331188B iw-SFT results in worse performance than SFT on Minerva Math (4.31 vs. 5.78) and AMC23 (7.34 vs. 8.28). In contrast, DFT consistently improves upon both the base model and SFT across nearly all datasets, including those where iw-SFT fails. These results underline better generalization ability DFT in diverse mathematical reasoning scenarios. Moreover, iw-SFT incurs additional computational overhead by requiring separate reference model to compute importance weights, whereas DFT dynamically derives its own weighting directly from the token probabilities of model, resulting in more efficient training procedure.
method on Qwen2.5-Math- (+2.45), this
We also compare against iW-SFT under the offline setting, as shown in Table 7. While iw-SFT performs competitively on certain datasets, achieving 60.80 on Math500 and 44.21 on AMC23, its overall average performance (31.86) remains below that of our method by +3.57 points. Moreover, iw-SFT shows only modest improvements compared to its standard SET counterpart, with aver- age score of 31.86 in the offline RL setting versus 30.28 with SFT (+1.58). In contrast, DFT achieves larger gain of +4.76 (from 30.67 to 35.43). These results indicate that iw-SFT provides limited benefits from reward supervision under offline constraints, whereas DFT is able to more effectively incorporate such signals, leading to better generalization and higher task performance.
19

---

Published as a conference paper at ICLR 2026
Table 6: Comparison with concurrent work iw-SFT math benchmarks. DFT outperforms the
iw-SFT in most settings across model families and benchmarks. The best performance is bold.
Math500
Minerva Math
Olympiad Bench
AIME24
AMC23
Avg.
LLaMA-3.2-3B w/w
5.13
2.63
1.51
0.00
2.03
2.26
I.I.MA-322-BB W/DFT
12.79
2.84
2.90
0.83
3.91
4.65
LLaMA-3.1-8B w/w--FFT
18.21
4.31
4.31
0.20
7.34
6.87
LLaMA-3.1-8B w/DFT
27.44
8.26
6.94
0.41
12.03
11.02
DeepSeekMath-7B w/w-SFT
35.32
8.75
11.11
0.61
18.28
14.81
DeepScekMath-7B wDFFT
41.46
16.79
15.00
1.24
16.25
18.15
Qwen2.5-Math-1.58 wiw-SFFT
59.38
17.08
26.82
8.13
40.00
30.28
Qwen2.5 5-aah.5-Matt-11.BB w/DFT
20.94
27.08
6.87
31.58
0wen2.5-Math-7B wf
70.28
25.70
34.46
16.46
51.09
39.60
Qwen2.5-Math-78 w/FFT
68.20
30.16
33.83
8.56
45.00
37.15
Table 7: Evaluation results five mathematical reasoning benchmarks in the offline reinforcement
learning setting using rejection-sampling baseld reward signals, compared against iw-SFT.
Setting
Math500
Minerva Math
Olympiad Bench
AIME24
AMC23
0wen2.5-Math-11BB wiw-SFT
SFT
59.38
17.08
26.82
8.13
40.00
30.28
Qwen2.5-Math-1 w/FT
62.50
22.94
7.31
Qwen2.5-Math-1.59 wiw-SFT
Offline
60.80
18.13
27.83
8.33
44.21
31.86
Qwen2.5-Math-1.5 w/DFT
Offine
64.71
25.16
30.93
7.93
48.44
35.43
A.5 EXPLORATORY EXPERIMENT OPENRI TRAINING DATASET
Inspired by DeepSeek-RI DeepSeek-Al et al. (2025), several studies have attempted to train open- source models reproduce its reasoning capabilities (Hugging Face, 2025). To this end, high- quality dataset, 1ppeRRi-Math-201 (Hugging Face, 2025), was constructed, where the prompts are drawn from NuminaMath 1.5 and the off-policy reasoning traces generated by DeepSeek-RI. LUFFY (Yan et al., 2025) further filtered out sequences longer than 8192 tokens as well as those verified incorrect by Math- Merh-yerify, resulting in about 45k prompts paired with off-policy reasoning traces. We adopt this dataset as the training corpus for SFT. All training details remain the same as previous experiments, except that the number of epochs is
3.
As shown in Table 8, training on OpenR M-athh2200 consistently improves performance, and the use of higher-quality annotations yields additional gains. SFT on this dataset increases the average accuracy of Qwen2.5-Math-1.
by + +13.24 points compared to the base model, while DFT provides further +9.03 gain, resulting total improvement of +22.27. These results suggest that DFT remains effective even when applied on top of high-quality training data, highlighting its potential as a general fine-tuning paradigm.
A.6 EXPLORATORY EXPERIMENT PEFT TRAINING SETTING
To investigate whether DFT remains effective under parameter-cfficient fine-tuning (PEFT) settings with limited compute, we apply DFT using LoRA adapters across two model families: LLaMA-3.2- 3B and Sove2.5-Matth 1.5B. All training configurations remain identical to previous full- palammerrtter experiments, except that LoRA is enabled with rank=8 and alpha=16.
As shown in Table DFT provides consistent improvements over both base and SFT baselines under LoRA-basee PEFT. For Qwen2.5-M 5-tthh 1.5B, DFT increases the average accuracy from 15.92 (base) and 16.87 (SFT) to 32.90. For I.LMMA3.2-3B. DFT achieves gain of +3.44 over SFT (from 1.19 to
20

---

Published as a conference paper at ICLR 2026
Table 8: Performance training the OpenR 1-eat---Mth-22000 dataset. mark is in bold.
The best score for each bench-
MathS00
Minerva Math
Olympiad Bench
AIME24
AMC23
Avg.
Owen2.5-Math-L.5B
31.66
8.51
15.88
4.16
19.38
15.92
Qwen2.5-Math-1.55 w/STT
61.60
20.29
24.27
4.16
35.47
29.16
Owcn2.5-Muth-L.SB w/DFT
71.76
27.00
33.48
9.79
48.91
38.19
Table 9: Performance on five mathematical reasoning benchmarks using LoRA for training. The best score each model across benchmarks highlighted in bold.
Math500
Minerva Math
Olympiad Bench
AIME24
AMC23
Avg.
LLaMA-3.2-3B
1.36
1.01
0.41
1.56
1.19
I.a.A-3.2--B w/SFT
4.88
1.56
1.68
0.00
2.66
2.56
LLaMA-3.2-3B w/DFT
11.13
3.87
0.00
2.97
4.63
Qwen2.5-Math,
8.51
15.88
4.16
19.38
15.92
Qwen2.5-Mathe Mathh w/SFT
41.47
10.85
11.56
1.45
17.03
16.87
Qwen2.5 Matn2.h-MMth---- w/FFT
22.58
28.45
5.84
40.78
32.90
Pefiemane
Perfeemeer Differet
KF1
XFT
26-4
10-4
76-5
1e5
32
64
255
Laarznng Rots
Figure 3: Ablation study of training byper-parameters, learning rates and batch size, for DFT and SFT Qwen2.5-Math-1a. M.5B model.
4.63). These results indicate that DFT can serve an effective fine-tuning strategy low-resource or compute-constrainee settings, where full model updates are not practical.
A.7 TRAINING
ABLATION
To assess the robustness and sensitivity of our approach (DFT) with respect to key training hyper- parameters, we conduct ablation study focused on learning rate and batch size, using the Qwen2.5-Math-11.55 M..h base model. This analysis aims to answer two central questions: (1) Is the performance gap between DFT and SFT due to suboptimal hyperparameter configuration SFT? (2) How sensitive are both methods to changes in learning rate and batch size?
We evaluate both DFT and SFT across four learning rates: 2e-4, le-4, 5e-5, and le-5. As shown in Figure (left), both methods exhibit certain degree of sensitivity to the learning rate. DFT con- sistently outperforms SFT under all configurations, suggesting that the performance gap cannot be attributed solely suboptimal hyperparameter choices in SFT. For both methods, intermediate learning rates (1e-4 and 5e-5) yield the best results, while both lower (le-5) and higher (2e-4) values lead to noticeable degradation.
We further assess the impact of batch size, sweeping values from 32 to 256. As shown in Figure (right), both DFT and SFT exhibit relatively stable performance across the full range of batch sizes.
21

---

Published as a conference paper at ICLR 2026
While minor fluctuations observed, there is consistent trend indicating that larger smaller batches significantly affect final accuracy. This suggests that batch size is not a dominant factor for either method in this setup, and that default values may suffice in practice.
22


Reviews:
[{"review_id": "zGvi7CAoga", "reviewer": "Reviewer_JU9o", "summary": "The paper observes that supervised fine-tuning (SFT) encodes a sparse reward that limits its generalization ability when compared to reinforcement learning (RL). Due to the sparse nature of the reward, using SFT may lead to optimization difficulty and/or poor generalization. The paper introduces a method called dynamic fine-tuning (DFT) that reweighs the SFT gradient to avoid the problem observed with SFT. The effectiveness of DFT over SFT is analyzed via experiments on several language models. Additional results are provided with other setups including coding and multimodal models.", "questions": "- Please check the ICLR Blog Post I shared above and see if that merits being cited in the paper. While the analysis is still useful my suggestion is to focus on empirical results as the main contribution in the paper\n\n- I would like to understand why there are no RL results in Table 1 (and others).\n\n- Are DFT-based models RL-able? \n\nOverall, this is a well written and simple to follow paper. I look forward to interacting with the authors, other reviewers and AC to get a deeper understanding of this work and make a decision.\n\nThe following paper may be of interest to the authors: \n\"Simplify RLHF as Reward-Weighted SFT: A Variational Method\" https://arxiv.org/abs/2502.11026\n\n(Note: I am sharing the above in the spirit of helping out a colleague in the research community. Citing the above is not necessary for me to recommend an acceptance.)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The analytical arguments are clear and presented in an easy-to-follow manner. The proposed fix via weighing the gradient follows naturally from the analytical argument.\n\n- Experimental results that show improvement over SFT models are fairly extensive. These experiments cover a variety of language model families (Llama, Qwen, DeepSeek). \n\n- The impelemtnation included in the appendix is easy to understand and use by a general ML practitioner.", "weaknesses": "-While the analysis is good, there may be other papers that have made observations about sparse rewards and done the derivation perhaps under a different guise. I appreciate the authors referring to GOLD (Pang & He, 2021) in the paper as I learned about this method as well. A follow up ICLR Blog Post does a derivation that looks similar to what is included in the paper (https://iclr-blog-track.github.io/2022/03/25/text-gen-via-lfd). So the derivation, while insightful, is perhaps not as valuable as the empirical results. \n\n- Table 1 reports SFT vs DFT results. I wonder why there are no results available with RL (SFT + RL, DFT + RL perhaps?). I am curious to understand whether DFT-based models can be effective when used to further train with RL. So the claim on DFT surpassing RL may need to be qualified as I do not see any RL methods in Table 1\n\n- Similar comment can be made for other setups like coding and multimodal models considered in the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjIvXHUTsz", "reviewer": "Reviewer_d8uu", "summary": "Proposes a modification to supervised finetuning (SFT) for language models, called Dynamic Fine-Tuning (DFT). Instead of minimizing the standard cross-entropy loss, DFT weights each gradient of an output by the probability of that output. This change is motivated by framing SFT as a reinforcement learning method. Empirical evaluations demonstrate that DFT outperforms SFT across several benchmarks and base models.", "questions": "--", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Relatively well-written and easy to follow.\n\n2. The idea behind DFT is intuitive and conceptually simple.\n\n3. Experiments demonstrate that DFT leads to substantial performance gains over SFT across several models and settings, covering mathematical reasoning, code generation, and multi-modal tasks.", "weaknesses": "1. The theoretical motivation behind DFT is imprecise and includes several unsubstantiated claims.\n\n    - The dependence of SFT on $1 / \\pi_\\theta (y | x)$ is fake, in the sense that it is obtained by multiplying and dividing by $\\pi_\\theta (y | x)$. In particular, it is not true that the gradient of SFT grows unboundedly due to the term $1 / \\pi_\\theta (y | x)$ in Equation (6), as this term cancels out with the expectation.\n\n    - The claim of SFT having sparse rewards does not make much sense to me. In SFT, you are guaranteed access to the output $y*$ with the optimal reward of 1 (according to the formulation in the paper). Such a claim about sparseness would be true if one would actually sample outputs from $\\pi_\\theta$ until landing on $y^*$, but this is not what SFT does.\n\n    - In several places (e.g., line 51 and lines 167 to 171), a connection is being made between large gradient norm, unstable training, and overfitting. Regardless of the fact that, as mentioned above, the gradient norm of SFT does not grow unboundedly when $\\pi_{\\theta} (y^{*}|x)$ is small, why would large gradient norms lead to overfitting? When making such claims you need to either ground them in the literature or theoretical evidence. Otherwise, it is best to refrain from making them or clearly hedge them as intuition to not mislead readers.\n\n    To be clear, if a method works well in practice, I do not believe that theoretical motivation is strictly necessary. There are plenty of heuristics that are useful in practice, yet lack theoretical backing. However, if one does choose to motivate their method theoretically, that motivation should be sound and clearly separate intuition from formal arguments, which can be backed by results in the literature or new results. Another downside of imprecise theoretical motivation is that it may hide the true reason for why DFT improves upon SFT.\n\n\n2. The paper lacks a discussion of potential limitations of DFT. For example, weighting outputs by their current probability should induce a tendency to prefer high probability outputs, thereby sharpening the model’s distribution. This suggests that the performance of DFT may highly depend on the initial model, to a greater extent than SFT. Does DFT indeed improve over SFT across the board, or when finetuning on domains further away than the model's current capabilities SFT works better? Are there any other potential limitations of DFT or do the authors always recommend using DFT over SFT?\n\n\n\nReview Summary and Recommendation\n---\n\nOverall, DFT seems to bring substantial benefits compared to SFT across several practically relevant settings. The fact that such an improvement can be achieved by a simple (in a good way) tweak to SFT is quite remarkable. I therefore tend toward recommending acceptance. However, I do strongly suggest either toning down some of the imprecise and unsubstantiated claims made regarding SFT (see weaknesses portion of the review) or grounding them in a formal analysis or empirical evaluation.\n\n\nAdditional (More Minor) Comments\n---\n\n1. In Equation (7) there is a minor issue with the definition of DFT: $L_{SFT}$ was defined as an expectation over the training data, while here it is treated as if it is the loss over a single example $(x, y^*)$.\n\n2. Typo in Equation (8): I believe there should be a minus sign there (i.e., the objective is to minimize the negative log probability and not the log probability).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1lDSHS4hE7", "reviewer": "Reviewer_8xgv", "summary": "This paper presents Dynamic Fine-Tuning (DFT), a method that improves the generalization of Supervised Fine-Tuning (SFT). The authors first provide a analysis by framing SFT from a reinforcement learning perspective, identifying that its implicit reward is inversely proportional to the model's confidence, which leads to unstable training. DFT rectifies this by reweighting the SFT loss with the token probability, a simple modification that is shown through extensive experiments to deliver performance gains across diverse models and tasks.", "questions": "- How does the DFT method influence the calibration of the model's output probabilities? \n- The decision to apply weighting at the token-level was justified by stability. Did the authors investigate applying a sequence-level probability as the weight, perhaps with safeguards like clipping, and how did its performance compare?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The theoretical motivation is clear, connecting the SFT gradient to policy gradients and pinpointing the problematic inverse-probability weighting as the root cause of poor generalization.\n- The proposed DFT is simple to implement, requiring only a minor modification to the standard SFT loss function, yet it yields substantial empirical improvements.\n- The evaluation is comprehensive, spanning multiple reasoning domains including mathematics, code generation, and multi-modal tasks.", "weaknesses": "- My main concerns lie in the potential conceptual limitations and unintended consequences of the core reweighting mechanism. The strategy of down-weighting low-probability tokens encourages the model to ignore what it finds hard. While this appears to prevent overfitting on noisy or rare patterns in the presented experiments, the paper does not adequately explore the boundary conditions under which this approach might fail. A more thorough discussion or ablation study on when this *hard example ignorance* becomes detrimental would strengthen the paper's claims.\n- The authors mention avoiding numerical instability as a reason for token-level weighting but lacks detail on how it handles token probabilities that are extremely close to zero, which could cause vanishing gradients for those specific tokens.\n- The implicit assumption that a uniform reward of 1 is optimal for all expert demonstrations might be an oversimplification, as it treats demonstrations of varying quality, complexity, or importance equally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)