Proximal Supervised Fine-Tuning
Authors: Wenhong Zhu
Ruobing Xie
Rui Wang
Xingwu Sun
Di Wang
Pengfei Liu
Venue: ICLR 2026 Poster (2026)
Keywords: SFT
generalization
language models
vision language models
Primary area: foundation or frontier models, including LLMs
TL;DR: PSFT is a trust-region–inspired fine-tuning objective that views SFT as a policy gradient method with constant advantages, constraining policy drift to stabilize training and improve generalization.

Abstract: Supervised fine-tuning (SFT) of foundation models often leads to poor generalization, where prior capabilities deteriorate after tuning on specific tasks. Inspired by trust-region policy optimization (TRPO) and proximal policy optimization (PPO) in reinforcement learning (RL), we propose Proximal SFT (PSFT), a fine-tuning objective that incorporates the benefits of trust-region, effectively constraining policy drift during SFT while maintaining competitive tuning. By viewing SFT as a special case of policy gradient methods with constant positive advantages, we derive PSFT that stabilizes optimization and leads to generalization, while leaving room for further optimization in subsequent post-training stages. Experiments across mathematical, human-value, and multimodal domains show that PSFT matches standard SFT in-domain, outperforms it in out-of-domain generalization, remains stable under prolonged training without causing entropy collapse, and provides a stronger foundation for the subsequent optimization.

Published as a conference paper at ICLR 2026
PROXIMAL SUPERVISED FINE-TUNING
Wenhong Zhu 1.2 Ruobing X Rui Wang1.2> Xingwu Sun 3,1 Di Wang' Pengfei Lin "Shanghai Jiao Tong University Shanghai Innovation Institute LLarge Language Model Department, Tencent University of Macau fzwhong714, wangrui12, jengfei)esjtu.eee. [xrbsnowing] 0163.
1,2
ABSTRACT
Supervised fine-tuning (SFT) of foundation models often leads to poor general- ization, where prior capabilities deteriorate after tuning on specific tasks. Inspired by trust-region policy optimization (TRPO) and proximal policy optimization (PPO) in reinforcement learning (RL), we propose Proximal SFT (PSFT), fine-tuning objective that incorporates the benefits of trust-region, effectively constraining policy drift during SFT while maintaining competitive tuning. By viewing SFT as special case of policy gradient methods with constant positive advantages, we derive PSFT that stabilizes optinization and leads to generaliza- tion, while leaving room for further optimization in subsequent post-training stages. Experiments across mathematical, human-value, and multimodal domains show that PSFT matches standard SFT in-domain, outperforms in out-of-domain generalization, remains stable under prolonged training without causing entropy collapse, and provides stronger foundation for the subsequent optimization.
INTRODUCTION
Recently, post-training has become crucial part of the overall training process. In particular, rein- forcement learning (RL) algorithms, such as Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Group Relative Policy Optimization (GRPO) (Shao et al., 2024), have demonstrated sig- nificant effectiveness when applied to language models (LMs) focused on reasoning tasks. As RL is scaled over time, foundation models gain the capacity to address complex problems through more profound and extended reasoning (OpenAI, 2024; Guo et al., 2025; Zhu et al., 2025), producing high-quality reasoning trajectories. Numerous community efforts have focused on leveraging this knowledge via supervised fine-tuning (SFT) (Guha al., 2025; Li al., 2025), distillation ap- proach valued for its efficiency and simplicity compared to RL.
However, SFT models are often criticized for poor generalization (Huan al., 2025). This limita- tion arises because SFT essentially performs behavior cloning, which can result in weak general- ization when the fine-tuning dataset is suboptimal distributionally misaligned with the pretrain- ing data (Chu et al., 2025), potentially causing large policy updates (Schulman et al., 2015). RL fine-tuning (RFT) provides a promising alternative, as numerous studies have shown that RL better preserves the generalization capabilities that SFT tends to erode (Chu et al., 2025; Huan et al., 2025).
Another concern is maintaining the ability to explore. practice, SFT is often used as a cold start to stabilize RFT training. The gains from RFT may largely come from refining capabilities already acquired during pre-training and SFT (Gandhi et al., 2025). However, excessive reliance on SIT can diminish a model's capacity for exploration (Xie et al., 2024), as it would cause the entropy collapse (Cui et al., 2025) and thereby constrain exploration during RL training ( (Yu et al., 2025).
Therefore, the current challenges lie in improving the generalization and exploration abilities of SFT models. To tackle these challenges, this paper proposes an improved fine-tuning strategy Pine-Tuning (PSFT) that avoids rote learning and achieves
Proximal Supervised liable performance in post-training. Specifically, we establish a theoretical connection between SFT and RL, and then introduce novel objective based clipped surrogate objective from PPO to SFT, which leverages the benefits of trust regions to constrain policy updates.
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Our experiments demonstrate that, compared to standard SFT, PSFT attains comparable perfor- mance on the target task while preserving the model's general capabilities. Moreover, PSFT miti- gates entropy collapse during training and yields superior target and generalization performance in subsequent RI. stages. We evaluate PSFT on mathematics, human value alignment, and multimodal tasks, highlighting its broad applicability in practice. Our contributions
follows:
We propose PSFT, novel SFT optimization method that utilizes clipped surrogate objec- tive to enforce trust-region-like constraints, preventing excessive policy updates. Compared to standard SFT, preserves the model's general capabilities while achieving comparable perfor- mance on target tasks.
PSFT effectively prevents entropy collapse and overfitting during SFT, thereby enabling sub- sequent RL stages to achieve more robust and superior results on both target and general tasks.
We extensively validate PSFT across various base models, tasks, settings, and evaluation met- rics, demonstrating its potential promising alternative to standard SFT.
2 PRELIMINARIES AND MOTIVATIONS
We formulate autoregressive language modeling as Markov Decision Process (MDP) (S, A, P. R). Here, S denotes the state space ( the prefix of the current decoding step), AA the action space (possible next tokens), and P(s') 8, the transition probability mo, and R(s, a) the reward function. Given a query x = (#1, Fm), the initial state is &p x. response = (y1,-.., Mn) is generated by sequentially selecting actions a₂ A. The policy defines the probability of y given æ as
To x)
II Ta(ac 8t): 8t = (I, Mcte, Ver (V), . ..-
2.1 SET AS THE SPECIAL POLICY UPDATE
Supervised Fine-tuning, The training objective of SFT is to minimize the cross-entropy loss be- tween the model's predicted token distribution and the ground truth tokens, which is defined
I SPT (0) (=,,,)>>> [log To(a [x)],
where az) pairs are sampled from an offline dataset D, with at representing the ground-truth token.
Policy Gradient. In contrast, policy gradient directly samples trajectories from the current policy Ty as it interacts with the environment. Using the policy gradient theorem (Sutton al., 2000), the objective is to maximize the expected return:
L PG (0) B(scaa)~=T logne(au |ss) A.].
where Å, is the estimated advantage function step which measures how much better an action a, is compared to the average action at state 8, under the current policy.
From this perspective, SFT can be seen special case of policy gradient where the sampling is from fixed offline dataset and the advantage is fixed Å, for the ground-truth actions, which corresponds to maximizing the likelihood of ground-truth tokens.
2.2 REUSIRREGION CONSTRAINTS RL
Trust Region Policy Optimization. TRPO (Schulman al., 2015) maximizes surrogate objective by leveraging trajectories collected under the old policy new introducing an importance sampling ratio (0) to reweight these samples for evaluating the new policy me.
Jo (at 9t) A. = E, [r.(0)A,]
(P) = 1e (1) ) TB (ar 81)
(4)
2

---

Published as a conference paper at ICLR 2026
where the superscript CPI refers to conservative policy iteration (Kakade & Langford, 2002). TRPO ensures that each policy update stays within trust region by enforcing a hard KL constraint, pre- venting the new policy from deviating too far from the previous one and thus maintaining stable and reliable learning.
Proximal Policy Optimization. However, directly maximizing L with hard KL constraint, as in TRPO, can be difficult to optimize in practice because of complex second-order optimization and sensitivity the constraint threshold (Schulman et al., 2017). To address this, PPO (Schulman al., 2017) modifies the surrogate objective by clipping the importance sampling ratio, formalized as:
C. (0) ) min (r=(0) Ac, clip(r.(0),1 -,,
e, -,,++ +))
(5)
This clipping mechanism effectively defines soft trust region rz()), which is restricted to remain within small neighborhood of (controlled by <) prevent destructive updates. Compared to TRPO, PPO achieves greater stability and efficiency.
3 PROXIMAL SUPERVISED FINE-TUNING
PSFT aims to improve performance on the target task while preserving general capabilities beyond the target domain such as scientific reasoning and instruction following and preventing entropy collapse (the over-concentration of the output distribution that reduces diversity and harms general- ization (Ziegler et al., 2019) (0),,, thereby enabling further optimization in the RL stage.
PSFT. We revisit the TRPO and PPO objectives in RI.. Both methods rely on importance sampling ratios (0) to reweight returns, while constraining these ratios to prevent the new policy from deviating excessively from the old policy Tew. Translating this idea to the supervised setting with the offline dataset D, where all actions are assumed
to be "correct" (i.e., A > O), we simplify the
advantage to A, = 1 (as in SFT), and define our PSFT loss as follows:
min (ux st))
TT (at (s)) clip( (a: 81))
e, 1 (.11+))))) +
7.PSPT (0) (e....))
(6)
To.. (a: st)
This objective regularizes supervised learning updates by limiting the ratio between the new and old policy probabilities. Intuitively, it discourages overconfident changes in token probabilities, thereby
optimizing target tasks while preserving existing general capabilities.
Comparison with PPO. Although PSFT adopts the soft trust region mechanism used in PPO, it is not a policy-gradient RL method. PPO optimizes expected return using on-policy trajectories and advantage estimates; by contrast, PSFT is a supervised offline objective (here simplify to A, 1), where the elipped ratio functions mainly as regularizer that limits overly large changes in token probabilities and reweights gradients.
Dynamic update of Tew Allowing the old policy Tew evolve rynamically-rather than keep- ing the initial model Teh fixed-is essential for gradual and stable learning from the dataset. In contrast, fixing the initial model confines optimization to the trust region centered on the reference model, which limits the knowledge that can be effectively leveraged from the offline dataset (see Section 5.3).
(st, a.) in Eq. 6 is sampled from the offline dataset D. In the initial steps, (f) may yield a biased expectation since Te., may not be aligned with the distribution of D. We let To... evolve during SFT, gradually learning from D, which could largely alleviate the potential bias issue. Besides, a warm-up SFT phase on D can be optionally introduced to better align the initial old policy Tha with the offline dataset, further improving PSFT's target-domain performance (see Section 4.1).
Warm-Up.
Gradient Analysis. Similar to PPO, the gradient confined the trust region for fine-tuning:
PSFP (0) (> (r) Igrust (rx(0) Vo log no (at st)],
where Tiu (r)( = { otherwise.
0 (1) > 1+
shows that if the offline dataset distribution deviates significantly from the model distribution (which is more likely to disturb the general ability), these tokens will have no gradient. This mech- anism is simple yet effective: it prevents large policy updates, mitigates entropy collapse, and pre- serves generalization, thereby enabling sustainable progress in the subsequent RL stage. Typically,
3

---

Published as a conference paper at ICLR 2026
the optimal value of e is set to 0.2 0.28 (a larger can lead to large gradients). The clipped token cloud given in Section 5.1, I, and further analysis is provided in Section 5.2.
4 EXPERIMENTS
In Section 4.1, we explore the training dynamics and then evaluate both in-domain and out-of- domain performances. In Section 4.2, verify the effectiveness of PSFT as the foundation of RL. In Section 4.3, we further conduct experiments in the alignment domain to assess the universal applicability of our approach. In Section 4.4, we further extend PSFT to vision language models.
4.1 MAIN EXPERIMENTS ON MATH REASONING IN THE SET STAGE
Setup. (I) Models and Datasets. We evaluate method based et al., 2025) and 1-8-n.ttt8Bt (Dubey et al., 2024). Our training data focuses on the math domain, aiming to improve general LLM capabilities through math reasoning. We employ the OpenRI-Maan - 1M22 long mhain-of-thoughe (CoT) dataset (Face, 2025). (2) Baseline. We con- sider the standard SFT method and SFT variant, denoted as SFT KL. which incorporates KL di- vergence constraints into the loss function. The KL regularization coefficient is set to 0.5, while the parameter PSFT fixed at 0.28, More details can be found in Appendix C.7.1.
Bwen2.5--B-Instteeee (Yang
4.1.1 TRAINING DYNAMICS ON SFT
We begin by examining the training dynamics of each method. Specifically, train each for around 10 epochs and plot the corresponding changes in entropy and performance. We report in-domain performance with AIME-24 avg and out-of-domain performance with GPQA avg@8.
0.8
SFT
0.8
SFT
SFT-KL
0.7
SFT-KL
0.6
PSFT
PSFT
0.5
(SFT(warm- 100)
20.5
PSFT(warm up 100)
0.4
0.4
PSFT/warm up 200)
0.3
0.3
0.2
0.1
0.1
0
200
400
600
800 1000 1200 1400 1600
200
400
600
800 1000 1200 1400 1600
Step
(a) Swen2.5-7B-Innsruu
(b) Llama3.1 1-BB Instructt
Figure 1: Training dynamics of entropy. Each epoch contains 178 steps.
Finding PSFT avoids entropy collapse. The entropy evolution is shown Figure 1. Compared to SFT and SFT-KL,, our PSFT produces smoother entropy curve. For SFT and SFT-KL, entropy exhibits a marked decline after each epoch, indicating potential overfitting. This suggests that PSFT is capable of sustaining long-term fine-graine ine-grained training without triggering entropy collapse. Notably, PSFT with a warm-up phase demonstrates the same stability, and the extent of warm-up plays a critical role in shaping the overall entropy level.
Che
1800 126
030 LEG0
000 BC 19K 100
1880
(a) In-domain performance.
(b) Out-of-domain performance.
Figure 2: Training dynamics of in-domainnouttof-doooe performance in SFT experiments, In each subfigure, Qwen2.5-7B-Inntrrul is shown on the left, and 1-8- on the right.
Finding 2: PSFT matches or surpasses in-domain performance of standard SFT at simi- lar entropy level. Figure 2a illustrates the evolution of in-domain evaluation performance. PSFT
4

---

Published as a conference paper at ICLR 2026
Table 1: Detailed results of the SFT stage. For AIME and AMC, the results are avg. @32. For the other in-domain benchmarks, the results are avg. @8. For GPQA, ARC-C, TruthfuQQA, and IFEval, the results are avg. @ For the remaining out-of-domain benchmarks, the results are pass. 1.
Bree2.5-7B-Inaae
Llama3. 1-8B-Instruct
Benchmark
Base
SFT
SFT-KI.
PSFT SaFTwam.no
PSFT
Base
SFT
SFT-KI.
PSFT "SFTTwam
PSFT
AIME-24
11.25
22.08
19.27
22.92
19.38
3.96
10.63
9.58
12.08
10.31
In-Donain
AIME-25
8.75
23.02
21.56
23.02
21.98
0.73
16.77
16.15
18.75
14.48
AMC
52.58
62.73
63.20
62.42
62.34
24.45
47.19
45.55
49,45
46.80
MATH-500
75.05
84.10
83.55
84.68
83.35
48.55
72,60
69.83
74.15
71.98
OlympidBench
39.72
52.35
52.70
52.30
51.50
17.09
41.43
39.19
42.07
39.61
Minerva
40.53
43.66
42.19
43.38
43.33
25.87
32.17
26.75
33.64
32.40
Avg.
37.98
47.99
47,08
48.17
46.98
20.11
36.80
34.51
38.36
35,93
GPQA
31.38
32.89
32.95
33.27
33.21
24.62
19.38
18.18
23.99
26.89
Out-of-Domain
ARC-C
91.54
92.22
91.86
92.09
92.29
80,96
87.73
87.02
89.72
89.11
TruthfulQA
55.14
MMLU-Pro
54.99
58.98
58.35
59.28
59.18
43,70
50.29
47.02
56,03
56.58
SuperGiPQA
27.59
29.02
27.69
28.37
28.10
18.16
21.95
19.81
25.26
25.85
HeadQA
73.41
74.65
74.07
75.24
75.82
68.20
73.52
71.95
77.71
77.90
IFEvaloos
73.94
54.42
55.44
55.07
78.10
33.45
33.08
Avg.
59.85
57.90
57.38
58.53
61.26
52.70
50.49
49.31
53,48
59.25
achieves performance on par with, and sometimes exceeding, SFT under comparable entropy lev- els. For example, at step 1300, PSFT reaches an AIME24 score of nearly 20 with an entropy of around 0.3. This entropy corresponds to that of SFT between epochs 300 and 520, yet PSFT deliv- ers a higher AIME score within this range. SFT with KL remains less effective than PSFT when considering both entropy dynamics and in-domain performance.
Finding 3: PSFT with warm-up surpasses standard SFT. Figure 2a shows the effect of adding a warm-up phase to PSFT. The in-domain performance improves steadily, with PSFT + warm-up consistently outperforming both vanilla PSFT and standard SFT baselines. Furthermore, longer warm-up periods yield even greater gains in in-domain performance.
Finding 4: PSFT avoids large fluctuations in policy updates. As observed in both in-domain and out-of-domain evaluations (GPQA, Figure 2b), PSFT generally maintains an upward performance trend. In contrast, SFT and SFT-KL exhibit pronounced fluctuations in out-of-domain performance. Therefore, PSFT preserves the better generalization to out-of-domain tasks.
4.1.2 DETAILED EVALUATIONS
Setup. (!) Model setup: We select checkpoints by in-domain performance: for Qwen2.5-7B. Instruct, the SFT model at 700 steps, SFT-KL at 900 steps, and PSFT at 1300 steps. (2) Evaluation benchmark: See Appendix C.7.1. (3) Inference. The inference length set to 10,240 tokens (4,096 for IFEval), with top-p of 0.95 and temperature of 0.7.
Finding PSFT with warm-up consistently outperforms standard SFT in-domain tasks. Table presents the performance across various in-domain benchmarks, where all fine-tuned meth- ods achieve notable improvements over the Base model. PSFT with the warm-up phase achieves the best overall results, maintaining robustness across different in-domain benchmarks.
Finding 2: PSFT demonstrates strong generalization ability. Table also shows different perfor- mance on out- out-of-domain benchmarks. We find that injecting long CoT data into the model improves its general reasoning ability, consistent with the findings reported Zhou et al. (2025). In con- trast, PSFT exhibits a significantly better generalization ability compared to SFT. For example, in IFEval, which evaluates fnslluctionnfollooiing capability. SFT, SFT-KI,, and PSFT with warm-up significantly degrade the original results. In contrast, PSFT not only maintains strong in-domain performance but also significantly improves reasoning abilities on diverse out-of-domain tasks, with only minimal compromise in instruction compliance.
Finding 3: PSFT demonstrates robustness across models. For example, when evaluating GPQA and TruthfulQA tasks, SFT and SFT-KL exhibit varying behaviors across different base mod- els-one one sometimes compromises performance while the other may boost it. In contrast, PSFT consistently improves performance on both Qwen and Llama models.

---

Published as a conference paper at ICLR 2026
Detailed results of the RL stage. For AIME and AMC, the results are avg. @32. For the other in-domain benchmarks, the results are avg. @8. For GPQA, ARC-C, TruthfulQA, and IFEval, the results are avg. @ For the remaining out-of- benchmarks, the results are pass. 1.
Table 2:
Qwen2.5-77 5-77--nnstruet
Llama3. 1omma3.1-BB... -B--Instrcct
Benchmark
GRPO
PSFT
GRPO
PSFT
GRPO
AIME-24
22.08
27.40
19.38
28.13
10.63
14.06
10.31
14.27
In-Domsin
AIME-25
23.02
26.15
21.98
27.19
16.77
17:50
14.48
18,02
AMC
62.73
70.86
62.34
71.72
47.19
54.38
46.80
55.23
MATH-500
84.10
86.60
83.35
87.48
72.60
77.03
71.98
77.78
OlympidBench
52.35
58.09
51.50
$8,50
41,43
47.13
39.61
47.74
Minerva
43.66
45.27
43,33
46.83
32.17
33.09
32.40
37.55
Avg.
47.99
52,40
46,98
53.31
36.80
40.53
35.93
41.77
GPQA
32.89
39.39
33.21
43,43
19.38
31.06
26.89
36,23
Out-of-Domain
ARC-C
92.22
92.25
92.29
92.42
87.73
88.57
89.11
90.92
TruthfulQA
61,93
67.16
68.69
MMLU-PPo
58.98
62,61
59.18
63.65
50.29
54.39
56.58
60.92
SuperGPPA
29.02
33,80
28, 10
34.51
21.95
27.80
25.85
31.30
HeadQA
IFEvalyes
54.42
53.89
73.03
73.73
33.45
31,85
69.75
71.02
Avg.
57.90
59,90
61.26
64.06
50.49
53.19
59.25
61.96
4.2 EXPLORATION ON THE POTENTIAL OF MODELS IN THE RL STAGE
Practical LLMs often conduct RL after SFT in the post-training phase. The SFT models should serve as an appropriate starting point for the subsequent RL stage, avoiding both overfitting and underfitting. and thereby better stimulate the potential of RL. We evaluate the power of PSFT in the RL stage.
Setup. We adopt DAPO (Yu al., 2025) with clip-higher value of 0.28, stable variant of GPPO. The RL training uses the DAPO-MATH-I 17k dataset, with detailed training configurations provided in Appendix C.7.2. We select the checkpoints based the highest in-domain performance for evaluation.
4.2.1
TRAINING DYNAMICS ON RL
0.50
Mat send
3.6
-330
1.36
3-3
3-4
3.9
20
2
24
(a) Entropy evolution.
(b) Performance evolution.
Figure 3: Training dynamics of entropy and performance in RL experiments. In each subfigure, Owen2.5-7B-Insttree is shown the left, and 1-0aa3.--BBIInsteee on the right.
Finding 1: PSFT leaves much room for RL optimization. The RL entropy evolution is shown in Figure 3a. As low entropy limits the model's exploration, it is notable that using PSFT as cold start results in higher entropy throughout training, with a steeper increase.
Finding 2: PSFT shows slow start and rapid catch-up in RL. The in-domain performance evolu- tion is presented in Figure 3b. Thanks to its high entropy, which promotes exploration, using PSFT the cold start leads to relatively steep performance improvement and ultimately surpasses the performance achieved when using the standard SFT as the cold start.
4.2.2 DETAILED EVALUATION
Finding 3: PSFT achieves better in-domain performance after RL. Table 2 presents the results of RL experiments in-domain benchmarks. Overall, RL improves the performance of all models. comprehensive evaluation of six in-domain tasks shows that, although PSFT initially lags behind SFT, it holds significant potential that can be further unlocked through RI.

---

Published as a conference paper at ICLR 2026
Finding 4: PSFT still outperforms the standard SFT on out-of-domain tasks by large margin after RL. Table 2 also presents the results of RL experiments on out-of-domain benchmarks. is evident that if the model exhibits weak capabilities during the cold-start phase, the subsequent RL training is constrained by these limitations (e.g.. the results of IFEval). PSFT functions well on different base models. It further underscores the necessity of enhancing the current SFT algorithm.
4.3 FURTHER EXPERIMENTS HUMAN ALIGNMENT
In this section, we demonstrate the universality of PSFT by extending our experiments human alignment datasets, employing various base models and algorithms, such as DPO (Rafailov et al., 2023).
PSFT
SFT
0.8
Entropy
0.6
Reward
-3
PST DP0 Insected"
Kif (njpcted)
0.2
SIT+ORO
500
1000
1500
2000
2500
100
200
300
400
Stop
Step
(a) Entropy evolution during stage.
(b) Positive/Negative reward progression during
DPO training.
Figure 4: Training dynamics of human alignment experiments.
Our results show that PSFT effectively reduces the alignment tax (i.e., the generalization gap) while still leaving room for further optimization.
Setup. We adopt completely different setup demonstrate the universality of PSFT further. (!) Model and Dataset. We fine-tune the pre-trained Qwen3-4B-Base (Yang et al., 2025) model with the UltraFeedback dataset (Cui et al., 2023). (2) Training pipeline. We select the chosen region of the dataset and then perform SFT/PSFT. And for PSFT, we additionally train the double steps named PSFT proooo Then, use the DPO algorithm to enable the model to learn the contrastive reward signal. (3) Evaluation. See Appendix C.7.3.
Finding 1: PSFT reliably prevents entropy collapse and enables DPO for alignment. As shown in Figure 4a, the entropy loss of SFT still shows severe sawtooth shape, indicating potential over- fitting phenomenon, while PSFT overcomes this issue. Therefore, SFT leads to a severe alignment tax (Figure 5) and restricts further optimization. Table 3 presents the results of DPO experiments on alignment benchmarks. We observe that PSFTppolonn achieves similar effect to SFT while avoiding entropy collapse, and it performs even better during DPO training. PSFT provides strong starting point, with DPO training unlocking further gains.
Finding 2: PSFT training exploits both positive and negative samples well. As shown in Figure 4b, the cold-start point can influence the sub- sequent DPO phase. PSFT prlongg and PSFT yield similar rewards on positive samples; the main difference is that prolonged training reduces the occur- rence negative samples. contrast, SFT training results in positive sam- ples appearing frequently while nega- tive samples are rare, which limits the model's s ability to learn from the reward signal and instead biases it toward the numan-selected data. This is consistent reported by Xiao (2024) that when SFT model becomes overly biased toward certain outputs with near certainty, it can subsequently induce preference collapse the alignment process.
Table 3: Qwen3-4B-Bass DPO training on the alignment benchmarks.
AlpacaEval2
Arena-Hard WR (%)
MT-Bench
Method
LC (%)
WR (%)
1-turn
2-tumn
SFT
12.24
8.52
17.90
7.64
5.71
< DPO
16.96
13.40
26.50
7.91
6.00
PSFT
11.95
8.37
17.40
7.41
5.84
4 DPO
19.26
15.17
30.20
7.63
6.74
PSFT - DPO
11.79
7.49
11.40
6.83
4.86
23.29
20.13
36.40
8.51
6.95

---

Published as a conference paper at ICLR 2026
GSMBK
NMLLI
TratifuRQA
CMKLU
GPQA
38.5
76.0
75,855.4
36
35.4
75.5
$3.0
68.1
57.5
33-
68-
78.6
$7.3
74.5
32-
36.3
74.0
77.0
73.5
76.0
76.4
65
65.0
35.5
15.0
25.5
25.5
23.0
72.5
72.02.4
28-
29
SFT-DPO
Figure 5: Results on alignment tax (out-of-domain tasks).
Table 4: Detailed results of in- domain text-only evaluation.
Table 5: Detailed results of multi-modal evaluation.
Benchmark
Base
SFT
PSFT
Benchmark
Geo3K
Math Verse
MATHVista
MMMU
MMMU-Pro
AIME-24
8.54
Base
29.28
37.03
66.80
41.89
29.59
AIME-25
1.88
10.21
10.73
+ GRPO
49.42
42.44
71.10
40.00
28.61
MATH-500
AMC
65.65
34.77
71.70
44.69
44.84
70.58
SFT
33.78
38.83
68.00
40.78
23.58
OlympidBench
29.44
37.50
36.02
GRPO
48.92
43.12
72.70
38.78
19.60
Minerva
30.93
31.85
32.26
PSFT
33.61
38.25
69.60
43.33
27.63
GRPO
50.58
44.14
72.20
40.78
26.07
MMLU
63.10
45.39
62.44
MMLU-Pro
47.90
40.67
50.55
Finding 3: PSFT reduces the alignment tax. Figure shows the alignment tax results with various tasks (e.g., GSM8K, MMLU, GPQA) the out-of-domain tasks. It indicates that: PSFT and PSFT+DPO could largely maintain the general ability during the SFT stage compared to the standard SFT. (b) PSFT-Prolong, even being degraded by possible overfitting issues, achieves relatively good results on alignment tax, implying the robustness of PSFT in practical scenarios.
4.4 FURTHER EXPLORATIONS OF PSFT ON VISION LANGUAGE MODELS
Setup. (1) Model and Datasets. We employ the Qwen2.5-7B. 1eeem2.5-7B-L--Isttmeee -L-Isrrutt (Bai et al., 2025) vision language model (VLM). We utilize the text-only OpenR1-Me -Mthh--49--696 dataset (Face, 2025) and the Geometry-3k (Lu et al., 2021) dataset, which consists of text-image geometry problems. (2) Train- ing Pipeline. We conduct SFT/PSFT on the VLM using the text-only dataset to inject long CoT reasoning ability. After that, we perform RL training the text-image pair dataset to further en- hance the visual reasoning ability. (3) Evaluation. Motivation and additional experimental details provided in Appendix C.7.4.
geal
R03
(a) SFT/PSFT training.
(b) RL training.
Figure 6: Training dynamics of Entropy and performance VLM experiments.
Finding 1: PSFT functions well multimodal scenarios. Based Figure 6, PSFT still effec- tively avoids entropy collapse while achieving comparable in-domain performance. By maintaining sufficient entropy space, the model maintains excellent performance throughout the RL process. suggests that incorporating text-only long CoT data can enhance the reasoning ability of VLMs.
Finding 2: PSFT preserves the general ability, while SFT compromises. Table shows that PSFT and SFT successfully inject the long reasoning data into the VLM models, achieving great improvement on in-domain text-only benchmarks. However, MMLU and MMLU-Pro indicate that
8

---

Published as a conference paper at ICLR 2026
SFT greatly weakens performance in general domains. Table 5 shows similar results. In-domain text-only long CoT data can improve a VLM's performance on in-domain multimodal reasoning. However, results MMMU and MMMU-Pro indicate that SFT suffers significant drop in multi- disciplinary multimodal understanding. In contrast, PSFT consistently maintains strong perfor- mance, achieving the best overall results across both in-domain and out-of-domain benchmarks.
IN-DEPTH ANALYSES
5.1 CLIPPED TOKENS IN PSFT
Figure 7 presents representative clipped tokens observed during the PSFT training process in Exp. 4.1. The clipped token mostly concentrates on uncertain words like "wait", "alternatively". and similar words that reflect certain "long thinking patterns". With the training in progress, these token-clip weights more pronounced, while other to- kens become smaller. The "thinking pattern" has been gradually and smoothly incorporated into models via PSFT with minimal disturbance to the general capabilities.
chesot Alternat viveltiivell velyl Treaa Ly
ais Wai
Alternatively maybe
auudar Him First
Letwart
PSrtapsundt
LLtt Toua
(a) Clipped token
(h) Clipped token
cloud epoch 1.
cloud in epoch
Figure Evolution of clipped tokens.
5.2 PARAMETER ANALYSIS ON CLIPPED VALUE
As shown in Figure 8, PSFT without clipping maintains higher entropy but suffers from large, un- stable gradient norms and highly fluctuating downstream results. By introducing the trust-region clipping mechanism, PSFT strikes better balance: it prevents entropy collapse, stabilizes gradient updates, and achieves steady improvements in performance. Across different clipping thresholds, with medium values of (e.g., 0.28) showing particularly favorable trade-offs between stability and performance. Flexible choices can be made within an appropriate range of for practical demands.
PSFT ino cipi
PSTTi- Sanue
PSFTic 0200
10
PSFTlc - 0200
PSFT (ro cipp
0.55
PSFTi 02811
F0.50
10
0.40
10
PSFT)/ -.00
PSPflc -10
0.35
157Tic w0.41
200
400
500
200
4003
Step
Step
Sup
(a) Entropy
(b) Gradient Norm
(c) Performance
Figure 8: Training dynamics and in-domain results of PSFT with different clipped values.
5.3 UPDATE FREQUENCY OF THE OLD POLICY
Figure 9 and Table show that dynamically updating the old pol- icy THy essential for effective PSFT. While fixed references (no up- dates) still yield improvements over the base model, their overall perfor- mance remains poor. contrast, frequent
Table 6: Results with different frequencies.
Benchmark
AIME-24
AIME-25
TruthfulQA
IFEvaluss
11.25
8.75
66.10
73.94
PSFT (no upd)
13.02
11.15
66.61
76.46
PSFT (upd. 16)
21.56
23.54
66.10
69.33
PSFT (upd.8)
19.38
21.98
67.16
73.03
every 4 steps-deliver substantial gains on in- domain tasks. For update intervals of 16 steps (batch size 256, mini-bate size 16) and steps (mini-batch size 32), we use the same learn- ing rate of le 6. These results further indicate that smaller batch sizes, combined with relatively higher learning rates, can enhance in-domain performance, albeit at the expense of generalization due to high fluctuations.
PSFT (upd.4)
22.50
22.92
66.72
72.08
SFT
22.08
23.02
63.14
54.42

---

Published as a conference paper at ICLR 2026
FSFT Ina update]
27.5
PSFT (no updatel
0.7
PSRT (update 16 steps)
JUME 2004 3269
25.0
steps)
0.6
PSFT (update steps)
PSFT (update 8 steps)
PSFT (update steps)
22.5
PSFT lupdate stens)
Entroyy
0.5
20.0
SFT
0.4
0.3
17.5
0.2
15.0
0.1
12.5
10.0
0
200
400
800
800 Step
1000
1200
1400
200
400
600
800
1000
1200
1400
Step
Figure 9: Training Dynamies under different old policy model updating frequencies.
6 RELATED WORK
Supervised Fine-tuning. SFT, typically the first post-training step, sets the stage for more robust fine-tuned model. Several studies have noted that the standard cross-entropy (CE) loss may not be the most suitable objective for SFT (Li et al., 2024b; Xiao, 2024), arguing that training with CE at this stage tends to encourage the model to memorize the training data rather than acquire more generalizable capabilities. One line of research focuses on improving target performance by addressing the distribution mismatch between expert demonstrations and the evolving target policy. iw-SFT (Qin & Springenberg, 2025) reinterpret standard SFT as optimizing loose lower bound on the RL objective. This lower bound becomes increasingly loose as the model distribution drifts away from the reference policy. iw-SFT addresses this gap by introducing importance reiieigh,nng, which assigns higher weights to more preferred trajectories and thus tightens the bound toward the true RL objective. DFT (Wu et al., 2025) views SFT as a flawed policy gradient and introduces simple mobability-basee reweighting that improves targeted performance. Our work aims to prevent entropy collapse while enhancing target performance, without compromising general capability, and leaving room for further optimization.
Reinforcement Learning. Following SFT, RL is employed to optimize reward signals directly. Policy gradient methods (Sutton et al., 2000) provide the foundation but are prone to instability and high variance. To address this, TRPO (Schulman et al., 2015) introduces a trust-region constraint, limiting each policy update to a small KL-divergence neighborhood ensure stable improvement. PPO (Schulman et al., 2017) further simplifies this approach with a clipped surrogate objective, striking a balance between stability and efficiency. GRPO (Shao et al., 2024) removes the critic and computes advantage using the relative rewards of outputs within each group, treating all tokens in sequence equally. We extend the benefits of these methods to the supervised setting.
Combination of SFT and RL. Recent studies have also investigated hybrid objectives that cou- ple SFT loss on offline data with RL loss on online rollouts (Kang et al., 2023; Yan et al., 2025). LUFFY (Yan et al., 2025) exemplifies this direction by mixing offline demonstrations with online trajectories at a fixed proportion in each training batch. Extending this approach, Hybrid Post- Training (HPT) (Lv et al., 2025) adaptively switches between SFT and RL based on rollout per- formance, balancing guidance and exploration to enhance reasoning. This idea constructing a composite loss has also been adopted in several recent frameworks (Zhang et al., 2025; Liu et al., 2025). Our method is orthogonal to these methods.
CONCLUSION AND FUTURE WORK
Motivated by TRPO and PPO, we propose Proximal Supervised Fine-Tuning (PSFT). PSFT pre- serves smooth entropy curve while achieving performance comparable to standard SFT, and it exhibits strong generalization. Furthermore, using PSFT as a cold-start model facilitates more effec- tive post-training techniques such RL optimization and DPO. Overall, PSFT presents promising alternative to traditional SFT. In the future, we will explore the universality of our PSFT on more diverse, industry-level datasets and models.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Our work is easy to reproduce, as mainstream RL frameworks can support this algorithm with min- imal code modifications. You can replace rollout generation with SFT demonstrations and assign constant value to all advantages in the sequence for training. We provide our code, built upon the open-source verl framework, in the supplementary material. Detailed hyperparameter settings are provided in the Appendix.
ETHICS STATEMENT
All datasets and models used in this study for our experiments are publicly available and open- source.
ACKNOWLEDGEMENT
This work supported by the AI for Science Program, Shanghai Municipal Commission of Econ- omy and Informatization AI 2022-C77-EGG/--BBXXXX00111 and the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001).
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1399 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le. Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arXiv preprini urXiv:2501.17166 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
0803005457v 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:21 10.14168. 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023,
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.226177 2025.
Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling Ilm evaluation across 285 graduate disciplines. arXiv preprini arXiv:2502. 147399 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled pacaeval: A simple way debias automatic evaluators. arXiv preprint 4rX1v:240..04475, 2024.
Hugging Face. Open rl: A fully open reproduction of deepseck-rl, 2025. whhhgginnfacce co.bblog/ppee-rll
URL https:

---

Published as a conference paper at ICLR 2026
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cogni- tive behaviors that enable self improving reasoners, or, four habits of highly effective stars. arXiv preprint ar arXhv:25033.13077 2025.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter. Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https:ttp/ zenodo. wwt.//rorrs///ooooooooooooooor.o /
reco
Etash Guha, Ryan Marten, Sedrick Kch, Negin Raoof, Georgios Smymis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reason- ing models. arXiv preprint (iX.::25566041775, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12445 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: (t..v::1133.08888 2021.
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general Ilm capabilities? understanding transferability of Ilm reasoning. arXiv preprint arXiv:2507.0043: 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamina- tion free evaluation of large language models for code. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: http:s/:///peeeveee..o net wttpeeewwwwweeefrrrrreeeeee forum?id- chfJJYC3iL.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference machine learning, pp. 267-274, 2002.
Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference-guided policy optimization. arXiv preprint arXiv:2305. 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gon- zalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11993 2024a.
Yang Li, Youssef Emad, Karthik Padthe, Jack Lanchantin, Weizhe Yuan, Thao Nguyen, Jason We- ston, Shang-Waa Wen Li, Dong Wang, Ilia Kulikov, et Naturalthoughts: Selecting and distilling reasoning traces for general reasoning tasks. arXiv preprint arXiv:2507.0199 2025.
Ziniu Li. Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised fine-tuning of large language models. arXiv preprint a40iv:2408. 16673, 2024b.
12

---

Published as a conference paper at ICLR 2026
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07995 2021.
Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. Uft: Unifying supervised and reinforcement fine-tuning, arXiv preprint arXiv:2505. 2025.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105. 240.0126000004165, 2021.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hanmaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Lixuan He, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, et al. Towards a unified view of large language model post-training. arXiv preprint arXiv:2509.04419, 2025.
OpenAl. Learning to reason with Ilms, 2024.
URL https:://ppnttt com/
Bearning-to-reasonn
Chongli Qin and Jost Tobias Springenberg. Supervised fine tuning curated data reinforcement learning (and can be improved). arXiv preprint 10XXi:2207.128566 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728-53741 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on muchine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprini arXiv: 2409.19255 2024.
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS), volume 12, 2000.
David Vilares and Carlos Rodriguzu HEAD-QA: A healtheare dataset for complex reason- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Lingnis- tics, pp. 960-966, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v N..18653/11/P19-099 1/P19-1092.91 URL https: http:/wwccccc acl org/ anthology 1P1--1999.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for I1m reasoning. arXiv preprint arXiv:2506.01939 2025.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Jiang, et al. Mmlu-pu pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 11195266-955299 2024.
13

---

Published as a conference paper at ICLR 2026
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint (iXii::2508.066233 2025.
Lechao Xiao. Rethinking conventional wisdom in machine learning: From generalization scaling.
arXiv preprint arXiv:2409.15111 15166, 2024.
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123, 2024.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning reason under off- off-policy guidance. arXiv preprint arXiv:2504. 14945, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.00 09388, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprini arXiv:2503. 14476, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal I1m truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169- 186.186. Springer, 2024.
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy meets off- off-policy experts: Harmonizing supervised fine fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508. 1408, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024a.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo. Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024b. Association for Computational Linguis- tics. URL http: /ttt:://arxin org/ abs / 2203. 13372.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. fnolluition-folloiingg evaluation for large language models, 2023. URL https:
/ /arxiv.. www.vorg/aaee aas///a//// 2311 1.....000 07911.
Ruochen Zhou, Minrui Xu, Shiqi Chen, Junteng Liu, Yunqi Li, Xinxin Lin, Zhengyu Chen, and Junxian He. Does learning mathematical problem-solving generalize to broader reasoning? arXiv preprint ar0iv:2507. (5000999... 2025.
Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, and Rui Wang. Weak-to-strong preference optimization: Stealing reward from weak aligned model. arXiv preprint 24XX:::2100.86400 2024.
Wenhong Zhu, Ruobing Xie, Weinan Zhang, and Rui Wang. Flexible realignment of language models. arXiv preprint 250.:::2006./7004 2025.
Daniel M. Ziegler, Nisan Stiemnon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1909 08593, 2019.
14

---

Published as a conference paper at ICLR 2026
A LIMITATION
Our experiments are currently limited in terms of model scale, dataset size, and data diversity. Future work is needed to evaluate the effectiveness of PSFT on industrial-scan models, larger datasets, and mixed data. We have not yet tested our approach on the latest model architectures, such as MoE or hybrid models, which will be considered in future studies.
B LLM-USAGE STATEMENT
We use LLMs to polish writing and correct grammar.
C APPENDIX
C.1 CONVERGENCE ANALYSTS
Assumption (Smoothness of the NLL) The negative log likelihood is L-smooth, i.e.
Vilog (a s) S L,
V(s,a) E S x A.
Assumption 2 (Bounded stochastic gradients) There exists G 0 such that
8)]l S. V(s, a) E S x Theorem (Convergence Rate ) Under Assumptions and 2, running gradient ascent with learn- & for T iterations yields:
V log To(a
ing rate ag Vi
T (.P.T(.))))) (0.)]1 < PSFT (8*) L PSPT (8:))
(1+ In
(8)
avT
2T
Proof. Denote by F(0) == I,PSFT (0) the objective. Let gr be the (mini-batch) stochastic gradient iteration t:
9t = (0) luuss(r (r (0)) V log n 85)], so that B[g 0. = VF
Because r. (0) < in the trust region, Assumption 2 implies E gl < =1++))
(1) Smoothness of F. For any 0, A,
F(0 + -) < F(0) - (F Lelig ||||2
Laip : L- + cc.
The same inequality with the opposite sign also holds, F is Seip-shmoth
(2) One-step progress. Lelip smoothness
With the ascent update oc+1
02 - a-st and the lower bound of
F(0++1) 2 F(O,) + at (F())) Lolie
afl1gel1?
(9)
2
Taking conditional expectation [x[] = B[. O.).
E] 2 (())) + arl (F.))) - Tretip at 262
(10)
2
(3) Bounding the gradient norm Rearranging equation 10 yields
at V((O)I1 < E|F((==1) F(O.) + Lelip af
Summing = 1 to T and taking full expectation,
a VF(() < EF(0r+1) F(01) + Telip 6
(11)
Let Ap = F F(O)) where Ft := sups F(0). Because F F(OT+1) ) 5 F*, equation 11 becomes
T
Lelpp 22
T
0x E V(FO)) < AF + 2
Σ
2
15

---

Published as a conference paper at ICLR 2026
(4) Choosing the step size.
Choose at
and assume S 1/ Leip (so that 1 Latiet v2
2 2)
Then
T
Σ
d:
2
a
2 (12VT-22 2 avT,
Vt
1=1
Σ
2
S a² (1
fm1
tm
(4) Averaged bound.
Divide equation
by T and use the two bounds above:
B())(( Ar + avT
aLeip G 01+ ln
2T
Substituting Lelip L(1 + e) and G (1 e)G gives
VL PSFT (0.)]1 2(L (0') (O)))(O))
aLG ( + In?)
+
T
avT
2T
When T + oo the second term vanishes as O(n T/T), so the dominant rate is (1/vTT). This consistent with the optimal upper bound of the standard non-convex SGD, indicating that under reasonable assumptions of smoothness and gradient bound, the stochastic gradient rise of PSFT has the same theoretical convergence rate that of ordinary NLL-SFT.
C.2 ENTROPY AS ADVANTAGE ESTIMATION
We perform training on the high-entropy tokens (Wang et al., 2025).
0.50
PSFT-Ent(>0.67)
26
PSTT-EME-O 671
PSTT Cnt
0.45
PSFT-Ent(=0.3)
AIME 2024 Aus 9321%
24
PSTT
PSFT
Diptaction
0.40
22
0.35
20
0.30
18
0.25
16
200
400
600
800
1000
1200
1400
200
400
600
800 Step
1000
1200
1400
Step
(a) Entropy
(b) Performance
Figure 10: Training dynamics and in nn-domain results of PSFT with and without entropy token selec- tion.
Benchmark
AIME-24
AIME-25
TruthfulQA
PSFT PSFT-Ent(( > 0.3)
26.98
28.75
62.90
25.21
28.65
62.48
PSFT-Ent( > 0.672)
24.17
25.10
63.71
Table Results under different baselines.
Results.
We believe that this sparse update may also contribute to more robust generalization.
C.3
MORE LARGE MODELS
We test PSFT ---00------57---A- MoE model.
16

---

Published as a conference paper at ICLR 2026
60.0
0.45
PSFT
PSFT
59.5
0.40
SFT
59.0
Entropy
0.35
HIME 2024 Ango
58.5
0.30
58.0
0.25
www
57.5
57.0
0.20
56.5
0.15
56.0
0
100
200
300
400 Step
500
600
700
800
100
200
300
400
500 Step
600
700
800
(a) Entropy
(h) Performance
Figure 11: Training dynamics and in-domain results PSFT with different methods.
Benchmark
AIME-24
AIME-25
TruthfulQA
IFEval
PSFT
61.67
48.38
80.19 77.89
88.52 50.56
SFT
59.89
46.88
Table 8: Results under different baselines.
Results. The results show that PSFT serves powerful and practical tool for industrial-scale model training.
C.4 RECENT BASELINES
Baseline 1: DFT. DFT (Wu et al., 2025) views SFT as a flawed policy gradient and introduces simple probability-basee reweighting that improves targeted performance.
la"l sg (no (az 8t)) log no (at st)
CDTT (0) (e
(12)
Baseline 2: Iw-SFT iw-SFT (Qin & Springenberg. 2025) reframes standard SFT as optimizing loose lower bound of the RL objective. This bound becomes increasingly loose as the model dis- tribution diverges from the reference policy. To mitigate this mismatch, iw-SFT introduces impor- lance re-weighting, assigning higher weights to more preferred trajectories and thereby tightening the bound toward the true RL objective. In addition, iw-SFT incorporates clipping range in the importance weights; following the settings in the original paper, we adopt hyperparameter values of 0.2 and 1.8.
Te (az 8t)
Liw Liw-srT(0) = Be >DD
log go (a sx))
(13)
Aeee (aī st)
Benchmark
AIME-24
AIME-25
TruthfulQA
IFEval
Base
11.25
8.75
66.10
73.94
PSFT (Ir 1e-6, bs 32)
19.38
21.98
67.16
73.03
PSFT (Ir 3e-6, bs 32)
22.50
22.92
66.63
71.32
iw-SFT
17.64
20.31
63.67
43.92
DFT
10.73
10.10
54.53
47.34
Table 9: Results under different baselines.
Results. As shown in Figure 12, DFT exhibits markedly different training dynamics and yields poor performance, which may indicate that it only benefits from domain-specific signals. In
17

---

Published as a conference paper at ICLR 2026
0.8
PSFT
20
PSFT
18
0.7
iw-SFT
0.6
DFT
16
DIFT
0.55
14
0.4
12
AIME 2024
0.3
10
0.2
0.1
0
200
400
600
800 Step
1000
1200
1400 1600
200
400
600
800
1000
1200
1400
1600
Step
(a) Entropy
(b) Performance
Figure 12: Training dynamics and in-domain results of PSFT with different methods.
trast, iw-SFT shows training behavior similar to PSFT, though PSFT ultimately achieves better per- formance.
C.5 LOWER LEARNING RATE
In this section, examine the impact of the learning rate both PSFT and SFT. As shown in Figure 14, a large learning rate induces greater update fluctuations, which in turn results in a higher clip fraction. Moreover, increasing the learning more rapidly. Based on Table 10, PSFT still achieves stable performance due to the gradient clip- ping mechanism, and the evaluation on IFEvalosse shows that PSFT outperforms SFT with small learning rate.
enables the model to fit the data distribution
ISFT 2.-66.m. mini but 32, baz 256
0.05
PSFT in 16 mini, boz 32, 256
PSFT. 3e.6. min 32, biz 256
0.05
PSFT. 30 minL 32, biz 256
256
SET
0.5
5..04
0.00
0.2
zan nenu 390
Delk io
Entropy
(b) Clip Fraction
(c) Performance
Figure 13: Training dynamics and in-domain results of PSFT with different clipped values.
Benchmark
AIME-24
AIME-25
TruthfuiQA
IFEval
Base
11.25
8.75
66.10
73.94
PSFT (Ir 1e-6, bs 32)
19.38
21.98
67.16
73.03
PSFT (Ir 3e-6, bs 32)
22.50
22.92
66.63
71.32
SFT (Ir 2c-5, bs 256)
22.08
23.02
63.14
54.42
SFT (Ir 5e-6, bs 256)
21.25
22.50
66.54
68.87
Table 10: Results under different learning rates across the methods.
C.6
CODING TASK
We use the OpenRI-cooo -code dataset and filter it for lengths less than 8192, composing the 13k dataset. We use LiveCodeBench the evaluation (Jain et al., 2025).
Results. These results show that while both SFT and PSFT effectively learn the target domain, PSFT better preserves mathematical instraction-folloiing ability.
18

---

Published as a conference paper at ICLR 2026
1.6
2250-000000 4 P PHINTTEN 32
3 36 06
1.2
0.6
0.4
0
100
200
300
400
500
100 150
200
250
300
350
400
450
500
100
150
200
250
300
350
400
450
500
(a) Entropy
(b) AIME-24 Performance
(c) LiveCodcBench Performance
Figure 14: Training dynamics and in-domain results of PSFT with different clipped values.
C.7 EXPERIMENTAL DETAILS
We perform SFT, PSFT, and RL training using the verl framework (Sheng et al., 2024), and employ LLama-Factors (Zheng et al., 2024b) for DPO training. The loss is aggregated using token-mean in verl. For SFT and PSFT, we use weight decay of 0.1. All experiments are conducted with full fine-tuning.
Evaluation Benchmark. (i) In-domain tasks: AIME24, AIME25, AMC, MATH-500 (Hendrycks et al., 2021), OlympidBench (He et al., 2024), and Minerva (Lewkowycz et al., 2022). (ii) Out- of-domain tasks: GPQA (Rein et al., 2024), ARC-C (Clark al., 2018), TruthfulQA (Lin et al., 2021), MMLU-Pro (Wang et al., 2024), SuperGPQA (Du et al., 2025), HeadQA (Vilares & Gómez- Rodríguez, 2019) and IFEval (Zhou al., 2023).
C.7.1 MATH REASONING EXPERIMENTS
The detailed training configurations for SFT/PSIT are presented in Table 11.
Method
Train batch size
Mini batch size
Learning rate
Train cpochs
High clip
Cutoff
PSFT
256
32
le-6
10
0.28
10k
SFT
256
2e-5
10k
Table 11: PSFT/SFT experiment configuration
C.7.2 EXPLORATION ON MODEL POTENTIAL IN RL
The detailed training configurations for RL are presented in Table 12. For Qwen models and LLama models, we use the same configuration.
Config Ir
SET + GRPO
PSFT + GRPO
1e-6
le-6
kl_coef
0.0
0.0
mex-prompt.length
2k
2k
max response length
10k
10k
overlong buffer.len
2k
2k
train train_batch_sif batch_size
256
256
ppo_mini batch
32
32
clip_ratio_lor clip-ratio__igg
0.2
0.2
0.28
0.28
temperature
1.0
1.0
rollout.n
total training steps
100
100
Table 12: RL experiment configuration
C.7.3 ALIGNMENT EXPERIMENTS
Evaluation benchmark. We adopt the same evaluation framework as described Zhu et al. (2024). Since short CoT models not solve the complex problems, the ARC,
19

---

Published as a conference paper at ICLR 2026
GSM8K (Cobbe et al., 2021), MMLU (Hendrycks et al., 2020), GPQA, and TruthfulQA to eval- uate the alignment tax. We evaluate our models on three aligmment benchmarks: MT-Bench (Zheng al., 2024a), AlpacaEval (Dubois et al., 2024), and Arena-Hard (Li et al., 2024a). We use Qwen3- 30B-A3-Instruct-2507 (Yang et al., 2025) as the judge model to provide alignment evaluation. We -hanness- (Gao et al., 2024) evaluate the alignment tax performance.
Method
Train batch size
Mini batch size
Learning
Train epochs
High clip
Cutoff_len
PSFT
256
32
lc-6
0.28
6k
PSFT prolong
256
32
1c-6
10
0.28
6k
SFT
256
2e-5
5
6k
Table 13: PSFT/SFT experiment configuration
Method DPO
Train batch size
B
Learning rate
Train epochs
Cutoff
64
0.01
5e-7
4k
Table 14: DPO experiment configuration
The detailed training configurations for SFT/PSFT on alignment tasks are presented in Table 13. The detailed training configurations for DPO are presented in Table 14.
Training.
Config
MT-Bench
Alpaca-Eval
Arena-Hart
temperature
0.0
0.7
1.0
top-p
0,95
0.7
Table 15: Alignment benchmark generation
We adopt the settings listed in Table 15 for evaluation generation. To assess alignment tax, we use the corresponding tasks in Ilm-eval- as summarized in Table 16.
Evaluation.
task
GSMSK
MMLU
TruthfulQA
CMMLU
GPQA
setting
gsm8k_cot
truthfulqa.nm Imc
cmmlu cot_n.soot
Table 16: Alignment tax evaluation using im-eval-harnesr
C.7.4 VLM EXPERIMENTS
Motivation. In real-world scenarios, high-quality multimodal reasoning data scarce, whereas large-scale textual reasoning data for LLMs is both abundant and of high quality. However, prior work has shown that directly incorporating such data can be harmful, as may disrupt the alignment of existing MILMs and impair their visual capabilities. Motivated by this, design framework in which PSFT leverages unimodal reasoning data to update only the LLM component while keeping the multimodal component fixed. We then investigate the model's capabilities after the SFT and RL stages. We hope that this pioneer exploration based on PSFT could shed light on the future multimodal reasoning learning paradigm.
Evaluation benchmark. (i) In-domain text-only tasks: AIME24, AIME25, AMC, MATH- 500 (Hendrycks et al., 2021), OlympidBench (He et al., 2024), and Minerva (Lewkowycz et al., 2022). (ii) Out-of-domain text-only tasks: MMLU (Hendrycks et al., 2020) and MMLU-Pro (Wang et al., 2024). (iii) In-domain text-image tasks: Geometry-33 33 (Lu et al., 2021), Math Verse (Zhang al., 2024), MATH (Lu et al., 2024). (iv) Out-of-domain text-image tasks: MMMU (Yue et al., 2024) and MMMU-Pro (Lu et al., 2024).
C.7.5 TEXT-ONLY REASONING EXPERIMENTS
The detailed training configurations for SFT/PSFT are presented Table 17.
C.7.6 EXPLORATION ON MODEL POTENTIAL IN RL
The detailed RL training configurations are listed in Table 12, with the exception that the response length is set to 5k.
20

---

Published as a conference paper at ICLR 2026
Method
Train batch size
Mini batch size
Learning rate
Train epochs
High clip
Cutoff_ien
PSFT
256
32
le-6
10
0.28
6k
SFT
256
2e-5
10
6k
Table 17: PSFT/SFT experiment configuration
C.8 SYSTEM PROMPT
For the reasoning task, use the system prompt follows: Please reason step by step, and put your final answer within \boxed[}.
For the alignment task, we use the system prompt as follows: You are a helpful assistant
C.9 VLM CASE STDUY
We provide two case studies, shown in Figure 16 and Figure 17, to demonstrate the feasibility of injecting text-only reasoning trajectories into VLMs.
21

---

Published as a conference paper at ICLR 2026
System Prompt You are helpful assistant.
[Task Description]
Find PS
[Output]
To find the length of PS, start by analyzing the given information and the properties of the triangle and the right triangle formed by the altitude.
7
1. Identify the given lengths: PQ = 9 9 SR =
RQ =
2. Since PR is the altitude from P to QR. PR is perpendicular to QR. This makes APRQ right triangle with <PRQ 90".
3. The length of PQ the hypotenuse of PPQQ. We can use the Pythagorean theorem to find the length of PR. The Pythagorean theorem states that in right triangle, the square of the length of the hypotenuse (the side opposite the right angle) is equal to the sum of the squares of the lengths of the other two sides. Here, PQ is the hypotenuse, and PR and RQ the legs of the right triangle.
-Q²=PKP + HQ²
Substituting the known values:
9² P + 72
81 PR² 19
Solving for PR
PR 81 49
PR 32
Taking the square root of both sides:
PR = 32 = 4v2
4. Now, we need to find length of PS. Since and are segments of the same line PS and PQ, we can use the fact that PS = PR+ SR.
PS PR SR
Substituting the known values:
PS = 4
Therefore, the length of PS is:
4
Figure 15: Base model reasoning trajectory.
System Prompt You are helpful assistant.
[Task Description]
Find PS
Provide the answer directly without reasoning.
[Output]
7
14
**Final * The length PS is 14
Figure 16: PSFT model without reasoning.
22

---

Published as a conference paper at ICLR 2026
System Prompt ( You are a helpful assistant. )
[Task Description]
Find PS
[Output]
Okay, so I need to find the length of PS in the given triangle. Hmm.
Let's look at the image again.
It seems like triangle PSQ divided by the altitude PR, which is the height from point P to the base SQ. Since SQ divided by R into two equal parts (each being units long). that makes triangle PPR and PQR right triangles, right?
perpendicular to SQ. So
Wait, let's check. If SQ is divided by R into two equal parts (each 7), then SQ must be 7 + 14. So the base SQ 14 units long, and PR is the height. Since PR is perpendicular to SQ, triangles PPR and PQR should be right-angled at R. So PR is the height, PS is one leg (7 units?), and PQ is the other leg (9 units). So using the Pythagorean theorem for triangle PQR: In triangle PQR: PR Q PQ"
So PR = 92 P 49 81 PR = -9- = 32 PR = ((2)) = ((1+2) = 4+ (2) Wait, but that's the height PR. However, I need PS, which is also part of the right triangle PPS with PR and PS. So in triangle PPS:
PS² + PR² = PQ?? Wait, no, no. Wait.
Wait, no. Let's clarify:
Since R is the foot of the altitude from P to SQ, which is split into two equal parts (each 7), SQ is 14, PR the height. So triangles PPS and PRQ both right triangles.
Wait, actually:
Wait, so triangles PPS and PRQ are both right-angled at R? So:
In triangle PPS:
PS = PR + SR²² No, but since SR is 7, then PS would V(PR² + SRR)? Hmm.
Wait, perhaps:
Wait, let's think of it:
Point R divides SQ into two equal parts, so the entire base SQ is 14 (since 7+7). So in right triangles PPR and POR. both have right angles R, so:
In triangle PPR: PS PR + SR', but wait, PR is the height, so PR is the same both triangles. So: So:
In triangle PPR: PS² + PR PS2 + (height)? (PS)?? No. perhaps:
Wait, perhaps:
Wait:
Wait, perhaps since R divides SQ into two equal parts (each 7). and PR is perpendicular,
PR is the height base SQ, hence:
In right triangle POR, we have QR=7, PQ=9, so PR + QR² PQ².
Thus PR = PQ² QR² 92 - 72 = 49 32.
Thus PR = /332)
/(16 2) = 44 ((2)
Thus, since in right triangle PSR. PS2 = PR SRR but SR=7. Therefore p5² = (4 / (2)) + 7 16 49 32 49 81
Thus PS = (81) 9
Wait, that seems to make sense.
Wait, checking:
PS=9? But 9 squared is 81, which is the sum (4 + ((2) squared (which 16*2=32) and squared (49). 32+49=81, /(81)= Yes, that seems to check out.
So the answer is 97 But wait, hold on. Wait:
Wait, but looking at the figure: PR is the height, in the right triangle PQR: PR is the
height, QR=7, PQ=9. So PR = VPQ2 QR?) = V(92 72) = vi81 - 19) = ((22)
4 * (2). So that's correct.
Then in the right triangle PPS: PS² = PR + SH² = (4 * ((2)) + + 16 2
32 + 49 = 81, so PS = V(81) = 9.
Yes, that seems correct.
**final *
Figure 17: PSFT model with reasoning.
23


Reviews:
[{"review_id": "AoB6uwMNGx", "reviewer": "Reviewer_XwCA", "summary": "This paper proposes a novel supervised fine-tuning method called Proximal Supervised Fine-Tuning (PSFT), which aims to address the issues of degraded generalization, entropy collapse, and excessive policy updates commonly found in traditional SFT. Inspired by TRPO and PPO from reinforcement learning, PSFT introduces a clipped probability ratio to constrain the magnitude of policy updates. The authors conduct extensive experiments across various domains, showing that PSFT achieves comparable in-domain performance to SFT while offering advantages in out-of-domain generalization and entropy stability.", "questions": "See Weaknesses.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Originality: The paper introduces the trust-region mechanism from reinforcement learning into supervised fine-tuning, proposing PSFT. This is a novel and reasonable perspective that offers fresh insights into improving SFT.\n- Quality: The authors validate the effectiveness of the proposed method through extensive experiments and compare it with several existing methods. Results show consistent performance improvements across multiple benchmarks.\n- Clarity: The structure of the paper is generally clear, although there are some notational errors that need correction.", "weaknesses": "1. Although PSFT is inspired by TRPO/PPO, it remains largely heuristic, lacking rigorous mathematical analysis on convergence, stability, or generalization error.\n2. While PSFT provides a better cold-start for RL, the RL stage still uses standard GRPO/DAPO. The paper does not explore joint optimization of PSFT and RL objectives.\n3. All experiments are conducted on 7B–8B scale models. The effectiveness of PSFT on larger models (e.g., 30B+) remains unverified.\n4. Current experiments mainly focus on mathematical reasoning and long CoT data. It is unclear whether PSFT is also applicable to dialogue, code, instruction following, or other general SFT tasks.\n5. Equation (3) is incorrect — the objective function is not in the right form.\n6. The assumption $\\hat{A}_t=1$ is too strong. The paper does not justify why this assumption is reasonable.\n7. In Equation (7), the meaning of $r_t$ is not clearly explained.\n8. The meaning of $π_{old}$ is ambiguous. It is unclear why the same \\*π\\*old is used throughout training on a fixed dataset, and thus the effectiveness of this approach remains questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qw3YPmFZMx", "reviewer": "Reviewer_Z7c6", "summary": "The paper introduces Proximal Supervised Fine-tuning, a method for supervised fine-tuning that takes inspiration from trust-region policy optimisation (TRPO) and proximal policy optimisation (PPO) from reinforcement learning. The supervised fine tuning setting is matched to the RL context by setting the advantage function to be a constant function and matching the objectives. Then the importance sampling of TRPO and clipping of PPO are applied. Experiments are carried out on a wide number of domains and applications. \n\nThe novelty of this paper is in an interesting composition of existing methods.", "questions": "- Line 58: \"It maximally preserves...\" — Do you have a proof to supporting the claim that your method \"*maximally* preserves the model's general capabilities\"?\n- Line 102: \"... far from the reference policy, enabling more targeted and ...\" — isn't the reference policy $\\pi_{\\theta_\\mathrm{old}}$? If so, is this the policy from the previous update step? If this is the case, is calling it a reference policy valid, since reference policy implies a notion of a ground-truth. Furthermore, if this is the policy from the previous step, how does this method ensure the training to be \"more targeted\" — what is the target if the target keeps changing at each step? \n\t- Alternatively, does $\\pi_{\\theta_\\mathrm{old}}$ refer to the initial policy? If so, couldn't the notation be changed to $\\pi_{\\theta_0}$, or something else to that effect?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a simple yet effective idea.\n- Generally well written, except for the points mentioned below.\n- Good set of experiments with clear descriptions of findings. \n- Code provided + open-sourced datasets and models.", "weaknesses": "- There are inconsistencies in the description of the mathematical notation:\n\t- - Section 2: a MDP usually has a reward function as well with respect to which the *best decision* is made.  You have omitted this. Why?\n\t- Line 78 onwards: What does the $\\*$ in $a^\\*_t$ signify: the optimal action? If so, state this in the text. What is the distinction between the action in Equation 2 and 3, where Equation 2 contains the $\\*$ and Equation 3 does not?\n\t- Line 83: The authors should define what an advantage is, at least in text. \n- References are missing at places (below are some examples):\n\t- Line 84: Please add a reference after \"policy gradient theorem\". \n\t- Line 105: \"... can be difficult to optimize in practice.\" — please state why and provide a reference.\n\t- Line 115: \"... preventing entropy collapse...\" — please say what entropy collapse is and provide a reference.\n\t- Line 257: \"... DAPO ...\" — please add a reference to DAPO. \n\t\t- Can you elaborate on why DAPO was chosen?\n- I am wondering if the framing of this paper can be improved / made more accurate by changing it from applying Reinforcement Learning to Supervised Fine Tuning to applying the specific techniques (importance sampling and clipping) to the supervised objective, and observing / realising similar benefits. In particular, because the advantage is set to a constant 1, I am hesitant to classify the presented work as RL. \n\n### Suggestions\n- Line 27: Please expand PPO and GRPO here since this is the first time these abbreviations are used.\n- Line 31: \"These reasoning models offer an abundant and valuable latent thoughts (Ruan et al., 2025) across the internet\" — It is unclear to me what this sentence means. Are the authors saying that foundation models are used to generate the thought data?  \n- Line 68: \"the state space (partial sequences), ...\" — Elaborate on what you mean by \"partial sequences\", since it is unclear here. E.g., partial sequences of what?\n- Line 115: \"... while retaining general capabilities...\" — can you expand on the specific capabilities you are targeting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C3pfRDLznC", "reviewer": "Reviewer_UvXJ", "summary": "This paper introduces a new supervised fine-tuning algorithm for LLMs that addresses the problems of overfitting and entropy collapse by incorporating trust-region updates inspired by TRPO and PPO. First, the paper reformulates the supervised fine-tuning (behavior cloning) objective in terms of the policy gradient objective and proposes an analogous clipped PPO-style objective.\nThe paper evaluates the proposed algorithm on several math reasoning benchmarks and investigates its effectiveness for RL post-training, human alignment, and vision-language models.", "questions": "- The analogy between PSFT and RPO/PPO-like objectives is not accurate: (1) In the ppo objective in equation 5 the expectation is taken w.r.t the old policy $\\pi_old$, while in the PSFT objective, the expectation is w.r.t the offline dataset. (2) The probability ratio in PPO and PSFT loss is $\\frac{\\pi(a \\mid s)}{\\pi_{old}(a \\mid s)}$, but it should something like  $\\frac{\\pi(a \\mid s)}{\\pi_{D}(a \\mid s)}$.\nCan the authors comment on these differences and how they would explain the performance gains? \n- The method is compared with two baselines, vanilla SFT and SFT with KL. Are there any other SFT baselines that the authors can compare to?\n- Performing an SFT warm-up seems to contradict the paper’s stated goal of replacing vanilla SFT with a more stable method that prevents entropy collapse. The paper claims that the warm-up stage helps mitigate the bias resulting from the misalignment between the data distribution and πold\\pi_{\\text{old}}πold​, but this warm-up phase may also affect some of the model’s capabilities. It is not clear how this aligns with the core motivation of the proposed approach.\n- Most of the evaluation is on Math benchmarks; can the authors show some results on coding or general reasoning tasks?\n\nI am willing to increase my score if the authors addressed my concerns.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is clear and easy to follow.\n- The proposed method is simple, and the results seem promising.", "weaknesses": "- The analogy drawn between the proposed PSTF objective and TRPO/PPO-style objectives may be oversimplified.\n\n- The paper lacks comparisons with stronger supervised fine-tuning baselines.\n\n- The evaluation is primarily limited to mathematical reasoning benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GtAUuET74i", "reviewer": "Reviewer_q3ok", "summary": "Introduces PSFT, an algorithm to replace SFT that achieves comparable performance but archives strong generalization. It maximally preserves the model’s general capabilities (prevents entropy collapse and overfitting), and achieves good performance, especially in out-of-distribution tasks. PSFT also is compatible with RL fine-tuning afterwards, and boots RL performance by initializing with a more general model.", "questions": "- How does PSFT compare to doing RL directly from the pre-training model without an SFT-like stage? In other words, can you outperform PSFT if you estimate the actual advantages instead of setting them all to 1.\n- nit: the results plots are kind of too small to see, can you make them bigger?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Very good contribution: the PSFT algorithm is simple, yet shown to be extremely effective. I believe this is a great and timely contribution to the community. It is incredibly important to have algorithms that can fine-tune pre-trained models yet maintain their generalist performance and not overfit.\n- Very clear presentation. The motivation is clear, and the preliminary covers the background work sufficiently. Explanation of the method is well done, and the results section makes the take aways crystal clear. The writing is very well done\n- Comprehensive experimental evaluation shows that PSFT works very well, and evaluates on multiple base models on extensive benchmarks.", "weaknesses": "- The paper focuses on an algorithm that can finetune a pre-trained model with generalization ability. However, there is not much comparisons with baselines that aim to finetune a model that preserves generalization ability. The only baselines are naively doing SFT and SFT with KL constraints, and it would be nice to see comparisons with more sophisticated methods that aim to keep generalization ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mmnNbltULQ", "reviewer": "Reviewer_4cnt", "summary": "The paper proposes Proximal Supervised Fine-Tuning (PSFT), which improves the supervised fine-tuning by utilizing a clipped surrogate objective to enforce trust-region-like constraints. The method can stabilize optimization and reduce the loss of generalization during supervised fine-tuning. Experiments demonstrate the effectiveness and statement of the method.", "questions": "There are different conclusions in Table 1 and Table 2 for the in-domain performance, where PSFT performs better in Table 1 but not in Table 2. This also happens before and after GRPO in Table 2, where PSFT performs better than SFT after GRPO but not before. Are there any analyses or discussions on that?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The main idea is well-motivated and straightforward.\n\n2. The proposed method is easy to implemented with a simple objective.\n\n3. Extensive experiments demonstrate the statements and efffectiveness of the proposed method.", "weaknesses": "1. Although the proposed method is well-motivated and empirically demonstrated, it seems lack of theoretical support.\n\n2. It is not clear whether the empirical gains obtained by the proposed method or the lower effective lr and gradient clipping. It should be compared to simple baselines like: lower LR, gradient clipping, sample filtering, or reweighting. \n\n3. The method is only compared with the common SFT and SFT SFT-KL, lack of recent RL-based fine-tuning baselines for comparisons. \n\n4. It is not clear whether the method also work on larger model sizes and datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)