Anchored Supervised Fine-Tuning
Authors: He Zhu
Junyou Su
Peng Lai
Ren Ma
Wenjia Zhang
Linyi Yang
Guanhua Chen
Venue: ICLR 2026 Poster (2026)
Keywords: SFT
Primary area: foundation or frontier models, including LLMs

Abstract: Post-training of large language models involves a fundamental trade-off between
supervised fine-tuning (SFT), which efficiently mimics demonstrations but tends
to memorize, and reinforcement learning (RL), which achieves better generaliza-
tion at higher computational cost. Dynamic Fine-Tuning (DFT) recently emerged
as a promising middle ground, reweighting SFT objectives with token probabili-
ties and achieving improvements in certain reasoning domains, though it exhibits
instability in other tasks. We provide a analysis of DFT through the reward-
weighted regression (RWR) framework, revealing that it corresponds to a spe-
cific auxiliary distribution choice that yields provably tighter RL bounds than
standard SFT. However, our analysis also uncovers a critical limitation: this con-
struction lacks distributional anchoring, leading to progressive drift that under-
mines training stability. To address this, we propose Anchored Supervised Fine-
Tuning (ASFT), which augments DFT’s reweighting with lightweight KL regu-
larization to preserve tightness while ensuring stability. Empirically, ASFT con-
sistently outperforms both SFT and DFT across mathematical reasoning, medical
knowledge grounding, and code generation, achieving substantial improvements
with minimal computational overhead. Our RWR framework provides a system-
atic lens for understanding post-training methods and demonstrates that principled
theoretical analysis leads to both stronger guarantees and practical gains.

Published as a conference paper at ICLR 2026
ANCHORED SUPERVISED FINE-TUNING
He 2 Junyou Su Peng Lai Ren Ma², Wenjia Zhang', Linyi Yang', Guanhua Chen 11 1 "Southern University of Science and Technology. "Peking University *Shanghai Artificial Intelligence Laboratory {zhuhe, 1ysu25]estu. pku..pkkuee... eda.eddu.c.. cn, chengh33sustech..et
ABSTRACT
Post-training of large language models involves fundamental trade-off between supervised fine-tuning (SFT), which efficiently mimics demonstrations but tends to memorize, and reinforeement learning (RL), which achieves better generaliza- tion at higher computational cost. Dynamic Fine-Tuning (DFT) recently emerged as a promising middle ground, reweighting SFT objectives with token probabili- ties and achieving improvements in certain reasoning domains, though it exhibits instability in other tasks. We provide an analysis of DFT through the reward- weighted regression (RWR) framework, revealing that it corresponds to spe- cific auxiliary distribution choice that yields provably tighter RL bounds than standard SFT. However, our analysis also uncovers critical limitation: this con- struction lacks distributional anchoring, leading to progressive drift that under- mines training stability. To address this, we propose Anchored Supervised Fine- Tuning (ASFT), which augments DFT's reweighting with lightweight KL regu- larization to preserve tightness while ensuring stability. Empirically, ASFT con- sistently outperforms both SFT and DFT across mathematical reasoning, medical knowledge grounding, and code generation, achieving substantial improvements with minimal computational overhead. Our RWR framework provides a system- atic lens for understanding post-training methods and demonstrates that principled theoretical analysis leads to both stronger guarantees and practical gains.
INTRODUCTION
Large language models (LLMs) have become central substrate for modern Al systems, powering instruction following, tool use, and multi-ster reasoning at scale (Brown et al., 2020 Touvron et al.
2023 Achiam et al. 2023 Guo et al. 2025). Post-training is crucial to adapt pretrained models to tasks and human preferences. This process typically involves two primary paradigms: supervised fine-tuning (SFT), which is off-policy method that imitates expert demonstrations collected from a fixed dataset, and reinforcement learning (RL), which is an on-policy approach that optimizes outcome-based rewards by directly interacting with the model's own outputs (Ouyang et al., 2022 Rafailov et al., 2023 Shao et al. 2024). While SFT is data- and compute-efficient excelling at rapid acquisition of desired behaviors, it tends memorize surface patterns rather than learn robust, generalizable strategies (Chu et al.; 2025 Zhang et al.) 2021 Feldman 2020). In contrast, RI. leverages outcome-driven updates and exploration to discover more transferable behaviors, but substantially more expensive and unstable in practice (Schulman et al.] 2017 Ziegler et al. 2019 Stiennon et al., 2020). This fundamental trade-off motivates methods that retain SFT's efficiency while inheriting RL's generalization benefits (Bai et al. 2022 Lee al. 2023 Yuan et al. 2024).
growing body of work re-examines SFT through an RL lens, arguing that the implicit reward induced by maximum likelihood is pathological and that principled reweighting trust regions are needed. Among these approaches, Dynamic Fine- Fine-Tuning (DFT) (Wu et al.) (225a) has gained significant attention by identifying pathological reward structure in standard SFT that leads to bounded variance when model probabilities approach zero. DFT addresses this through probability- based reweighting, achieving remarkable empirical improvements mathematical reasoning tasks. However, our preliminary experiments reveal that DFT's effectiveness is domain-specific; it excels
"Equal Contribution. 'Corresponding Author.
'The code is available at /ttttpa://gichhb.ccom wwt.huub.ooo//uucco/ccee chi 5

---

Published as a conference paper at ICLR 2026
Accuracy Domain)
MedMCQA Accuracy Domain)
Divergence
0.28
0.2
2.225
015
00
0.2
Traniing
Teaining
S11+
Figure 1: Training dynamies comparison across fine-tuning methods on medical knowledge tasks. Left: MMLU accuracy (out-of-domain evaluation); Center: MedMCQA accuracy (in-domain eval- uation); Right: KI. divergence from base model. DFT exhibits severe distributional drift (high KI. divergence) while ASFT maintains stability through KL anchoring and achieves superior perfor- mance on both tasks.
in reasoning- intensive domains yet exhibits instability in thowledge-intensivy tasks and lacks theo- retical grounding for its design choices.
To address these problems, provide principled theoretical analysis of DFT within reward- weighted regression framework inspired by reward-weighted regression and importance sampling theory (Rubinstein & Kroese, (016). From the theoretical perspective, we reveal that DFT corre- sponds to specific auxiliary distribution construction that yields provably tighter lower bound on the RL objective compared standard SFT. This causes critical limitation: the absence of distri- butional anchoring mechanisms leads to progressive drift away from the reference distribution. The distributional shift makes the bound increasingly loose and increases importance weight variance, explaining the training instabilities of DFT.
To address these fundamental limitations, we propose Anchored Supervised Fine-Tuning (ASFT), a lightweight extension of DFT that incorporates a KI. divergence regularization term to prevent distributional drift while preserving the tightness benefits of adaptive reweighting. As demonstrated in Figure D ASFT maintains stable KL divergence while achieving superior performance across both in-domain and out-of-domain evaluations.
Empirically, ASFT consistently outperforms both standard SFT and DFT across mathematical rea- soning, medical knowledge, and code generation tasks. On mathematical reasoning benchmarks with 100k training samples, ASFT achieves average improvement of +4.85 points (18.6% over DFT and +1 +17.89 points (142%) over the base model. In medical knowledge tasks with 10k samples, ASFT delivers +8.28 points (24.8%) improvement over SFT and +10.65 points (33.9%) the base model, requiring only 3% of the training cost of full RL approaches.
Our contributions threefold: (1) We provide rigorous theoretical explanation for DFT's domain-specific effectiveness and inherent instabilities, grounding its heuristic design within the for- mal wwward-weighted regression framework and proving that it achieves strictly tighter bound than SFT while suffering from uncontrolled variance growth. (2) We propose ASFT, a simple yet prin- cipled method that resolves DFT's stability issues through lightweight KL anchoring while main- taining its tightness advantages, requiring minimal computational overhead compared to full RI. approaches. (3) We demonstrate that ASFT delivers superior performance across both reasoning- intensive and knowledge-intensiv domains, achieving better generalization than SFT, greater sta- bility and broader applicability than DFT, and RL-comparable performance with SFT-level compu- tational efficiency.
2 RELATED WORK
Supervised fine-tuning and reinforcement learning. Supervised fine-tuning (SFT) and rein- forcement learning (RL) are the two dominant paradigms for post-training large language models. SFT can viewed as optimizing a stable but loose lower bound on the RL objective, which ensures efficiency and robustness but often leads to memorization and limited generalization (Wei et al.
2

---

Published as a conference paper at ICLR 2026
2022 Chung al. 2024 Zhang et al. 2021 Chu et al. 2025 In contrast, RL directly optimizes
outcome- based rewards, achieving tighter alignment and stronger generalization but at the cost of instability, high variance, and heavy computation (Oyyang al. 2022 Schulman et al. 2015 2017). Recent work seeks to bridge this trade-off by either tightening the SFT bound via reweighting and importance weighting (Wu et al. [025aa] Qin & Springenberg 2025 stabilizing RI. through trust-
region and hybrid methods (Sheng et al.
Zhu et al. 2025).
Importance weighting and policy optimization. The connection between supervised learning and reinforcement learning through importance weighting has deep theoretical roots (Kahn & Mar- shall [953; Dayan Hinton, 1997). In the context of policy optimization, importance sampling enables off-policy learning by reweighting samples from behavior policy estimate gradients for a target policy (Metelli et al. 2018) Jiang & Li 2016), though the resulting weights can suf- fer from high variance when distributions are misaligned (Andradóttir et al., [995). To address this, trust-region methods (Schulman et al. 2015) and proximal policy optimization (Schulman 2017) constrain policy updates to remain close to reference policy. Building on these deaas, recent advances in language model fine-tuning have explored importance-
super-
vised objectives (Qin Springenberg (2025,, proximal supervised ane-tuning (Zhu et al., (025), and probability-based reweighting in dynamic fine-tuning (Wu et al.) (202a)). as well as unified frameworks that combine SFT and RL principles (Lv al., [025] Wu et al. 2025b). Our work follows this line of research by anchoring the auxiliary distribution itself to the base model, thereby extending importance-wen weighted approaches with a mechanism that stabilizes training while retain- ing their theoretical tightness.
PRELIMINARIES
In this section, we establish the theoretical foundation connecting supervised fine-tuning and re- inforcement learning through the reward-weighted regression framework, setting the stage for our analysis of existing methods and our proposed ASFT approach.
3.1 PROBLEM FORMULATION AND BASIC FRAMEWORKS
We consider language modeling where trajectories T (az, y) consist of input prompts a and gener-
ated responses y. A parametric policy (T) = ( x)) = Tr= Ta( Vec, x) assigns probabil- ity via autoregressive decomposition. The reinforcement learning objective maximizes expected reward (0) ME [R(T]] where R(T) X' x y + [0, 1] evaluates trajectory quality. In contrast, supervised fine-tuning performs behavior cloning on expert demonstrations D
{(z, y")) sam- [log To > x)).
pled from reference distribution (T) by minimizing LSFT(#) = (ma >-
3.2 THE REWARD-WEEE WEIGHTED REGRESSION FRAMEWORK
Building on prior work in weighed regression and importance sampling (Peters & Schaal 2007 Rubinstein & Kroese 2016 Qin & Springenberg 2025). we adopt the reward-weighted regression (RWR) framework for language model fine-tuning. This framework provides princi- pled connection between SFT and RL objectives by leveraging importance sampling and auxiliary distributions construct tighter bounds on the RL objective.
Under sparse rewards where R(T) 1ly y²] and the assumption that supp(To) S supp(Tret), we can establish the following fundamental result:
Proposition 1 (SFT as RL Lower Bound). The RL objective satisfies:
J(0) 2 Cref ErED [log [T)
where D ((x,y") | = 1) and Czef
P Trtte (T E D*).
This reveals that SFT optimization implicitly maximizes a lower bound on the RL objective. How- ever, this bound becomes increasingly loose my diverges from Neef during training.
3

---

Published as a conference paper at ICLR 2026
Within the RWR framework, generalize tighter bounds through auxiliary distributions. For any distribution (()) with appropriate support:
g(T)
J(0) 2 Craf BrED+
log TT TT(TT)
(2)
(T)
The choice of auxiliary distribution determines both the tightness of the bound and the stability of the resulting optimization procedure. This sets up the fundamental trade-off between validity and tightness that we address this work.
3.3 DYNAMIC FINE- TUNING: AN EXISTING APPROACII
Dynamic Fine-Tumme Tuning (DFT) (Wu al. 2025a) addresses SFT's limitations by identifying a patho- logical reward structure in standard SFT. When viewed as policy gradient method, SFT's implicit reward (y x) Toly exhibits inverse-probability weighting that causes unbounded vari-
f[y=y"
ance when Th x) approaches zero.
DFT addresses this through probability reweighting:
(DF = - (<n [x) To("" 2)]
(3)
where sg[-] denotes the stop-gradient operator.
While DFT achieves empirical improvements, its theoretical properties within the RWR framework remained unclear-as gap address in our analysis.
METHOD
4.1
THEORETICAL ANALYSIS DFT WITHIN THE RWR FRAMEWORK
We found that DFT can be precisely characterized within the RWR framework through a specific auxiliary distribution construction. This analysis reveals both DFT's strengths and fundamental limitations.
Key Finding DFT corresponds to a specific auxiliary distribution choice. We discovered that the DFT objective is mathematically equivalent to choosing the auxiliary distribution:
((T) = Tref (T D+) sg [Pa (T)
(4)
E nent (++
sg Po (T)
This construction directly recovers the DFT sequence-level objective:
LDFT = - E (sg(p((T)))))))))))))))))) log (T()]
(5)
Key Finding 2: DFT achieves provably tighter bounds than SFT. We proved that this auxil- iary distribution yields strictly tighter lower bound the RL objective compared to standard SFT whenever the policy assigns non-uniform probabilities to demonstrations (detailed proof in Appendix(D4)
Theorem (Strict Tightness). The DFT auxiliary distribution yields strictly tighter lower bound than standard SFT whenever Var (po > on D
This theoretical result explains DFT's superior empirical performance in domains where the policy distribution exhibits sufficient variance across training examples.
DFT suffers from distributional drift. However, our analysis also revealed crit- ical limitation: the policy distribution progressively diverges from the reference distribution during training. As optimization proceeds, becomes increasingly concentrated on trajectories with high pe(T), creating feedback loop where the model focuses diminishing subset of training data. This distributional drift makes the bound increasingly loose and the importance weights increasingly high-variance, reducing effective sample size and destabilizing optimization.
Key Finding 3:
4

---

Published as a conference paper at ICLR 2026
We formalized this instability by noting that the fundamental inequality 2 1 + log (used to derive the RL lower bound) achieves equality if and only if L. In DFT's case, u 9=(r)*
Me(r)
so the bound is tight only when <T(r) = qo (T), i.., when Po(T) is constant on D However, as training progresses, po(r) becomes increasingly non-uniform, making the inequality strictly loose. This leads to deteriorating bound quality, reduced effective sample size, and training instability.
4.2 ANCHORED SUPERVISED FINE-TUNING (ASFT)
To address DFT's distributional drift while preserving its tightness advantages, we propose An- chored Supervised Fine-Tuning (ASFT). Our method adds lightweight KI. regularization term that constrains the policy within a trust region of reference checkpoint:
CASPT (0) = CDFT (0) + [DLL (Tbase () s)]]
(6)
where Thase a fixed reference policy (typically the pretrained model) and À > 0 controls anchoring strength.
Theoretical Guarantees. This design preserves DFT's tightness benefits since the KL term does not alter the lower-bound structure, while providing explicit variance control that prevents the expo- nential growth that destabilizes pure DFT training. The anchoring mechanism creates a trust region around the reference policy, allowing controlled exploration of tighter bounds without sacrificing distributional stability.
Practical Implementation. Following standard practice in language model training (Ouyang al. 2022 Shao et al. 2024). we implement ASFT at the token level by distributing sequence-level weights across tokens using normalized per-position allocation, ensuring mathematical equivalence to our theoretical framework while enabling efficient computation. Our method requires minimal computational overhead compared to standard SFT adding only simple KL penalty yet delivers RL-comparable generalization performance along with SFT-level efficiency.
5 EXPERIMENTS
5.1
SETUP
Models. We conduct fine-tuning experiments using LLaMA-2-78 Touvron et al. 2023) and Touuvooo Qwen2.5-7B(Qwee et al.) 2025), widely adopted models in the field. We select LLaMA-2- 7B specifically to avoid potential contamination from prior supervised knowledge. Qwen2.5-7B, on the other hand, is state-of-the-an model that is broadly used in current research. For knowledge- intensive (medical) tasks, we utilize both LLaMA-2-7B and Qwen2.5-7B to evaluate knowledge fine-tuning. For mathematical reasoming tasks, we focus exclusively on Qwen2.5-7B due its su- perior reasoning capabilities, whereas LLaMA-2-7B is less competitive in mathematical reasoning. This setup enables a systematic study of knowledge and reasoning learning across both fact-based and reasoning- intensive domains.
Datasets. We evaluate ASFT on two domains: (i) Mathematical reasoning, using 10k/30k/ 100k samples from NuminaMath CoT (LI et al. 2024) for training, and testing Math500 (Hendrycks) et al. 2021). Minerva Math (Lewkowycz et al. (022), OlympiadBench (AI Mathematical Olympiad
2024.. AIME 2024 (American Institute of Mathematics 2024), and AMC 2023 (Mathematical As-
sociation of America 2023); (ii) Medical knowledge, using 10k/30k/100k MedMCQA (Pal et al.)
2022) samples for training, and testing MMLU-medical (Hendrycks al. 2020). MedQA (Jin
et al. 2021) and the MedMCQA test set.
Training and Evaluation Settings. All methods are implemented using the AdamW optimizer with a constant learning rate. For mathematical reasoning, SFT and DFT use model_mat lenghh 2048, global batch 256, learning-rate 5 x 10-5, and are trained for epoch. ASFT follows the same configuration with coefficient X 0.05. For medical knowledge, we set model Laaeeelma____e length 512, global batch size 64, learning rate 2 x 10-5, and train for epochs, with ASFT again using 0.05. At the evaluation stage, for math,
the default chat template and Chain-of- -finn-of-Though
5

---

Published as a conference paper at ICLR 2026
Medical Benchmarks
Math Benchmarko
Methods
MedQA
MMLU
MedMCQA
AIME24
Math500
Minerva
Olymplad 7.69
ACM23
Avg.
Base
29.85
30.52
33.76
31.38
1.63
28.79
9.26
15.65
12.61
Soale:
GFT
33.31
13:20
10:4
1100
113.0
11.30 12:04
12.26
1128
16.73 (432
SPT w/ KL.
29.22 (8:3
30.63 p0:1
33,01
30.95 joz0
0.41
1121
42.21 117.0
12.05 12.79
12.08
17.19
-151
16.79 943r
DPT
14
13:0
31.20 17:56
29.19
19:19
4.18
59.51
pwom
17.10 17.88
24.95
10236
31.00
11505
27.77 15:.6
ASFT 39.28
46.37
40.45
42.03
3.33
59,60
19,91
24.50
36,41
28.75
Datonet Scale:
SET
53.54 13
38
17
56,03 13:30
56.02 14.64
2.71 1106
44.74
115.95
13.21
13.95
13.44
1505
21.56 159)
19.93 17:3
GET w/
30.56
10.36
28
Jo06
116.22
1377
13:18
1333
18.60
DFT
31.26 (1:43
33.08 12:36
35.09
11:33
33.14
11.76
3.34
11:00
57.93
(20.4
23.28
154.02
25.31
137.22
28.44
11279
27.66
115/8
20.61
110.0
30,00
(1435
Datovel Soade: 1008
SFT
33,46
38.01 17:09
35.67
35,71 14.33
0.83
47.30
13:46 16.30
14,16 16:47
20,00 1435
19.15 1551
SFT w/ KL
30.09 19.24
31.62
11.00
33.85
10.09
31.85 1044
1.41
1021
16.81
1100
14.13 14.97
13.74 te0
20,00 14.39
19.22
(5:))
DFT
36.61
41,26
36.31
12:09
38.06
6.26
56.88
21.18
22.68
27.19
26.04
ASFT
40.61
110.76
42.02
37.32
53.55
39.98
15.00
6.66
15.0
59.99
131.20
23.55
25.57
1:7.88
36.72
133.7
30.50
$17.00
Table 1: Performance comparison of fine-tuning methods on medical benchmarks (left, base: LLaMA-2-7B and math benchmarks (right, base: Qwen2.5-7BB under different dataset scales. Bold numbers indicate the best performance in each group, and rows with blue background highlight our ASFT approach. Arrows with 1 and |l indicate performance improvements and degradations relative to the base model.
(CoT) prompting, report average accuracy over 16 decoding runs (iemperature 1.0, max length 4096). For medical, we use standard prompt templates and multiple-choice accuracy, Baselines include SFT, SFT w/ KL, and DFT (Wu et al. 2025b).
5.2
MAIN RESULTS
Our experimental results demonstrate that ASFT consistently delivers superior performance across both knowledge-intensse innteniivv and reasoning rassoinngintensive domains while maintaining training stability. As shown in Table [ across both medical knowledge and mathematical reasoning tasks, ASFT consis- tently delivers strong and stable improvements over baselines. In knowledge-intensive domains, ASFT not only avoids the severe performance degradation observed with DFT (which drops by an average of -2.19 points at 10k samples), but also achieves substantial outperfrioniiiii the base model by + +10.65 points (a 33.9% relative improvement) at 10k scale, and maintaining robust advan- tages as the dataset size increases (10k: +10.65, 30k: +10.63, 100k: +8.60). This stability across scales highlights ASFT 's scalability and addresses the distributional drift issues that limit DFT in such settings. For mathematical reasoning, both DFT and ASFT surpass standard SFT, but ASFT maintains consistent edge and greater training stability. With 100k samples, ASFT improves over the base by +17.89 points (versus DFT's +13.43), and the advantage is even more pronounced on challenging benchmarks like AMC23 (36.72% vs. 27.19% for DFT), reflecting superior generaliza- tion. Overall, ASFT 's improvements are not only larger but also more consistent across diverse benchmarks, while DFT's gains are more variable especially on tasks requiring broad mathemati- cal reasoningg-undersoorrinn the robustness and effectiveness of ASFT.
5.3 ABLATION STUDY
We conduct two ablations: (1) comparing forward vs. reverse KL regularization, and (2) analyzing the impact of learning rate and batch size. These studies clarify the effect of KL direction and the robustness of ASFT to key hyperparameters.
Forward vs. Reverse KL. Here we follow the standard notation: let Q denote the policy model (i.e., the fine-tuned model with parameters 0), and P denote the reference model (the pretrained base model). In our approach, we use the forward KL divergence, DKI, Q). to regularize the policy model. For comparison, we also experiment with the reverse KL divergence, DKL P). As shown in Figure 2 forward KL leads to consistent improvements. Theoretically. forward KL en- courages mode-covering behavior, preventing the model from collapsing and ensuring it maintains the broad distribution of the base model. In contrast, reverse KI. is mode-seeking. which can cause

---

Published as a conference paper at ICLR 2026
McdOA
MMLU
MedMCQA
0.00
0.575
6.55
0.500
0.30
0.30
0.33
0.500
0.22
6.22
66770
0.00
Confficiont (A)
Coefficient (A)
Coefficiint (A)
Figure 2: Comparison of forward KL. (FKL) and reverse KL (RKL) regularization effects across different coefficient values (A) on MedQA, MMLU, and MedMCQA benchmarks. Performance is measured in accuracy, with horizontal dashed lines indicating baseline performance of Base, SFT, and DFT models.
HedMCQAA (ln-domain)
Hedo& (ost-af-donahh)
NNU (ont-of-domall)
Figure 3: Comparison of model performance across three benchmarks (MedQA, MMLU, MedM- CQA) for five models (LLaMA-2-7B, LLaMA-2-70B. Qwen2.5-7B, Qwen2.5-32B, Qwen2.5-72B) using four fine-tuning strategies (Base, SFT, DFT. ASFT). Each subplot shows the scores for specific benchmark, highlighting the relative effectiveness of different fine-tuning methods across models.
the model focus excessively few figh-probabilityy sequences. We further sweep the regular- ization coefficient and find that the optimal range is around À = 0.1; both excessively small and large values lead to under-anchoring over-regularization, respectively.
Training Hyper-Parametery Ablation. We conduct an ablation study to evaluate the robustness of ASFT with respect learning rate and batch size, using LLaMA-2-7B and 10k samples from MedCAQA. For learning rate, we examine six values ranging from 5e-6 to 2e-4. ASFT consistently outperforms both SFT and DFT across all configurations, with the best performance observed intermediate rates (le-5 and 2e-4). Extreme low (5e-6) or high (2e-4) rates lead to minor drops, highlighting that moderate learning rates are preferable but ASFT remains robust overall. For batch size, we sweep values from 32 to 256. The results show stable performance across the full range, with only small fluctuations. This indicates that batch size is not a sensitive factor for ASFT, and standard settings suffice. Overall, ASFT demonstrates both strong robustness and low sensitivity to key hyperparameters, maintaining consistent advantage over SFT and DFT in medical QA fine- tuning tasks. The detailed results for learning rate and batch size sweeps are shown in Appendix E.2
6 ANALYSIS AND DISCUSSION
6.1 SCALING ANALYSIS ACROSS MODEL SIZB
We evaluate fine-tuning methods medical-domain datasets using 10k training samples (Figure see 3, detailed results in Appendix E.I). Our experiments cover both LLaMA-2 (7B and 70B) and Qwen2.5 (7B, 32B, and 72B) models. Across all model sizes, ASFT consistently outperforms Base, SFT, and DFT, and its improvements remain robust as models scale, demonstrating effective and stable adaptation under low-resource settings.
7

---

Published as a conference paper at ICLR 2026
Methods
MedQA
MMLU
MedMCQA
Avg.
LLaMR-2-17
29.85
30.52
33.76
31.38
Supervised Fine-Tuning
DFI
24.67 15.18
22.82
17.70
30.43
L3.33
25,97
45.41
1w-S*T
28.36
35.89
15.37
34.88
11.12
33.04
11.66
SFT
33.31 13.56
33.52
13.00
33.28
10.48
33.44
12.06
ASFT
39.28 19:43
46.37
113.85
40.45
16.69
42.03
110.65
Reinforcement Learning
GREO
30.48 70.63
32.46
11.94
34.64
10.88
32.53
11.15
DAPO
39.75 19.90
48.63 -18.11
38.37 14.61
42.25
(10.87
Continual RL
SP" DAPO
36.84
16.99
44.76
114224
39.11
15.35
40.24
18.86
ASFT + DAPO
41.32 111.47
49.54
119.02
41.45 17.69
44.10
112.2
Table 2: Comparison of different post-training strategies medical benchmarks. Bold numbers indicate the best performance in each group, and rows with blue background highlight ASFT approach.
6.2 COMPARISON WITH REINFORCEMENT LEARNING METHODS
Table presents the performance comparison of different fine-tuning methods several medical reasoning benchmarks. Our proposed method, ASFT, consistently outperforms all SFT-based ap- proaches, including standard SFT, SFT with KL regularization, DFT, and iw-SFT, demonstrating the effectiveness of our adaptive weighting strategy. For example, ASFT achieves an average score of 42.03, which is substantially higher than iw-SFT (33.04) and SFT (33.44). However, expected, ASFT still falls short advanced reinforcement learning- based methods such as DAPO, which achieves average of 42.25, indicating that while method narrows the gap with RL approaches, RL still maintains a slight advantage in these tasks. The experimental results validate our theoretical analysis from Section 4 The comparison between SFT and DFT (33.44 25.97) confirms DFT's distributional drift problem in knowledge-intensive tasks, while SFT vs ASFT (33.44 vs 42.03) demonstrates the effectiveness of our anchored auxiliary distribution construction. Using final ac- curacy as proxy for bound tightness, the performance ordering SFT < ASFT < DAPO (33.44 42.03 < 42.25) empirically supports our reward weighted regression framework, showing that ASFT achieves tighter lower bound on the RL objective as proven in Theorem [ improvement of ASFT over standard SFT while remaining computationally efficient demonstrates the practical value of our theoretically grounded approach.
The substantial
6.3 ASFT AS ENHANCED INITIALIZATION FOR REINFORCEMENT LEARNING
The results in Table 2 further demonstrate that ASFT provides superior initialization point for sub- sequent RL finentuning.a Starting from ASFT and continuing training with DAPO yields consistent gains over SFT DAPO (44.10 10 vs. 40.24 average, +3.86 points), with the largest improvements on MMI.U-medical (+4.78) and MedQA (+4.48). ASFT + DAPO also surpasses standalone DAPO (04.10 42.25), indicating that KL-anchored fine-tuning not only improves direct supervised per- formance but also creates more stable policy foundation for RL optimization. This finding sug- gests that the distributional stability provided by ASFT's KL anchoring not only improves direct fine-tuning performance but also creates a more robust foundation for advanced RL algorithms, extending the practical utility of our method beyond standalone applications.
Bece-tunnngSStttinggggg
6.4 CROSS-DOMES VALIDATION ON CODE GENERATION
To examine the generality of ASFT, we fine-tune LLaMA-2-1 2-B-- on 10k samples from the Magicoder-
Evol-Instruct-I 110K (Wei et al. 2023) dataset using the same setup as Section 5 but for 2 epochs. Evaluation is performed with the evalplus framework (Liu al.] 2023) HumanEval, Hu- manEval+ (Chen et al.) 202T), MBPP, and MBPP+ (Austin et al. 2021). The results are sum- marized in Table 3 ASFT achieves the highest average score, notably improving HumanEyal and
8

---

Published as a conference paper at ICLR 2026
Methods
HvmanEval
HumanEval+
MBPP
MBPP+
Avg.
LLaMA-2-17
17.1
14.6
28.3
21.2
20.3
SET
23.2 16.1
19,5 14.9
34.1 15.8
28.6 17.4
26.4
16.1
iw-SFT
23.8 16.7
21.3
16.7
31.0 12.7
15.3
25.7 15.4
DFT
15.9 41.2
12.8
41.8
28.3 1..00
22.3
11.1
19.8
Jo.5
ASFT
27.2 110.1
16.8
32.5
14.2
26.7
15.5
27.0
16.7
Table 3: Performance (%) on code generation with I.LaMA--77B. Bold numbers indicate the best performance in each column, and rows with blue background highlight ASFT approach.
HumanEval+ while remaining competitive on MBPP, confirming that the anchoring mechanism gen- eralizes effectively to code generation.
6.5 COMPUTATIONAL EFFICIENCY AND MEMORY ANALYSIS
While ASFT demonstrates superior performance across both knowledge-intensive and reasoning- intensive domains, the KL divergence computation against the reference model introduces signifi- cant computational overhead that merits careful analysis. During training, ASFT requires maintain- ing the reference model Tbase in memory alongside the training model πy. effectively doubling GPU memory consumption from 98GB to 88.02GB for full tull-parameter fine-tuning on LLaMA-2-7B.
The KL computation additionally introduces approximately 23.7% training time overhead compared to standard SFT (0.648 vs 0.524 hours), though remaining substantially more efficient than full RI. approaches like GRPO (51.24 hours). All training times mea- sured on a single NVIDIA A A100 GPU. More critically for deploy- ment, inference requires loading both models simultaneously, cre- ating scalability concerns where memory efficiency paramount.
Method
Time
Memory Acc (GB)
(hrs)
SET
0,524
38.96
33.04
DFT
0.521
38,90
25.97
1M SPT
8,287
33.04
GRPO
51.24
481.98
32,53
DAPO
ASPT-T.RA
ASFI
0.648
88.02
42.03
To address these practical limitations, leveraging the mathematical properties of low-rank adaptation to enable memory-efficient implementation. The key insight exploits LoRA's parameter decomposition AW = BA where the fine-tuned model becomes (e(2)²) Theae [y]=; Whsse BA). This pa- using
propose ASFT-LoRA,
Table 4: Training time, mem- ory, and accuracy of different methods on L.AMMA---BB.
rameterization enables computing single model instance by dynamically switching between Whase and Wase BA during for- ward passes, eliminating the need for separate model copies while preserving theoretical anchoring guarantees. As shown in Table 4 ASFT-LoRA (rank=8, Ir=5e-4) reduces memory consumption to 40.70GB (comparable to standard SFT) and training time to 0.594 hours (13.4% overhead over SFT), while achieving substantial improvements over SFT baselines (39.45 vs 33.04 overall accuracy). Notably, ASFT-LoRA with appropriate learning rate selection achieves 93.9% of full-parameter ASFT's performance (39.45 vs 42.03 overall accuracy) while requiring less than half the mem- ory, demonstrating the versatility of our anchoring framework across different parameter--fffciency regimes. We observe that LoRA- BaRA-bard based training benefits from higher learning rates (5e-4) compared full fine-tuning (2e-5), consistent with recent findings on low-rank adaptation (Hu et 2021)
6.6 ALTERNATIVE ANCHORING MECHANISMS.
Our theoretical framework reveals that the key to stable optimization lies in providing distributional anchoring to prevent drift, and principle, any regularization term that constrains the policy dis- tribution can serve this purpose. While KL divergence provides effective anchoring, we investigate whether other terms terms-parttin pere--cartttlllyy the standard SFT loss itself offer a more lightweight alter- native. The SFT loss (fF = -E(r.9" flog Tyy"" x>) naturally encourages the model to stay close to demonstrated behaviors, suggesting it may provide implicit anchoring effects as observed in Figure

---

Published as a conference paper at ICLR 2026
Math
Medical
0.300
B554
Baze
0.275
511
0.250
0.225
0.375
0.200
0.350
0.325
0.150
0.500
0.125
0.275
0.180
0.250
(o)
Conffidient
Figure 4: Comparison of KL-based anchoring (ASFTx1) versus SFT loss-based anchoring (ASF(sFF) across different coefficient values (a or X) on math reasoning and medical knowledge tasks. ASFT KL maintains stable performance across all values, while ASFT SFT shows domain- specific sensitivity: degradation at large coefficients for math tasks and volatility for medical tasks.
We therefore explore a hybrid objective that combines DFT's adaptive reweighting with SFT-based anchoring:
CASFT-SFT(O)) = CDFT (0) + aCsFT(0)
(7)
where a controls the anchoring strength. Compared KL-based anchoring. this approach is compu- tationally more efficient as it does not require maintaining separate reference model during train- ing. As shown in Figure we conduct a grid search over a E (0.01, 0.05, 0.1,0.5, 1.0, 5.0, 10.0) on both math and medical tasks.
Interestingly, we observe domain-specific patterns: on mathematical reasoning tasks, ASFT-SFT achieves competitive performance at moderate values (0.05-0.5), but degrades significantly at larger coefficients (dropping from 0.275 to 0.168 as a increases from 0.01 to 10.0). This suggests that for reasoning tasks where the base model already possesses strong prior knowledge, moder- ate SFT anchoring can effectively stabilize training by activating these capabilities without over- volatile constraining exploration. In contrast, medical knowledge tasks, ASFT-SFT shows behavior across different a values, with performance fluctuating between 0.340 and 0.370. This instability likely reflects the challenge of balancing knowledge injection (requiring departure from the base distribution) with stability preservation in knowledge-intensive domains. Throughout both settings, ASFT with KI anchoring (green lines) maintains consistently superior and stable perfor- mance across all coefficient values, demonstrating that while SFT loss provides useful anchoring, divergence offers more robust distributional control, particularly for tasks requiring significant distribution shift from the base model.
7 CONCLUSION
We present Anchored Supervised Fine-Tuning (ASFT), principled approach that addresses the fundamental trade-off between supervised fine-tuning's efficiency and reinforcement learning's gen- cralization. By grounding Dynamic Fine-Tuning the reward-weighted regression framework, show that DFT achieves tighter RL lower bounds than SFT but suffers from distributional drift. ASFT resolves this through lightweight KL anchoring, preserving tightness while ensuring stability. Empirically, ASFT consistently outperforms SFT and DFT across mathematical reasoning, medi- cal Kndw injection, and code generation. The method achieves substantial improvements with minimal computational overhead, demonstrating that principled theoretical analysis can lead to both stronger guarantees and practical gains.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This project supported by National Natural Science Foundation of China (No. 62306132), Guangdong Basic and Applied Basic Research Foundation (No. (025A1510011544) We thank the anonymous reviewers for their insightful feedback this work.
ETHICS STATEMENT
All experiments in this work conducted using publicly available datasets and standard bench- marks. The proposed Anchored Supervised Fine- Tuning (ASFT) method intended for research purposes improve model generalization and stability. When deploying models trained with ASFT in high-stakes applications, we recommend thorough validation, human oversight, and appropriate safeguards to ensure outputs are accurate, reliable, and aligned with ethical guidelines. We acknowl- edge potential risks of harmful or biased content and encourage the use of input filtering, output moderation, and ongoing monitoring to mitigate such risks. Our goal is to contribute to robust, efficient, and trustworthy Al systems that can be safely deployed.
REPRODUCIBILITY STATEMENT
We provide comprehensive implementation details in Section including specific hyperparameters, datasets, and evaluation protocols. All experiments use publicly available datasets and standard benchmarks. We have released our implementation code to ensure reproducibility of our results. The code is available /httts:://gitiiii com/ httm//hhhhcchhe/555555
LIMITATIONS
We evaluate ASFT on three mamnains-mathematiion reasoning, medical knowledge, and code gen- eration. Although results consistent across these areas, broader evaluation across diverse task types and domains is needed to fully establish the method's generalizability.
Our theoretical analysis shows that DFT's effectiveness is fundamentally dependent the prior knowledge embedded in the base model, which accounts for its strong performance reasoning tasks with well-established priors, but weaker results in knowledge-intensive domains requiring more substantial distributional shifts. Although our KI.-based anchoring mitigates DFT's instability, the method's reliance model priors persists. Our ablation study alternative anchoring mech- anisms (Section 6.6) suggests that domain-adaptive anchoring strategies could enable more robust generalization across domains; however, developing a systematic framework for such adaptation remains an open challenge.
Additionally, ASFT incurs higher computational costs than standard SFT due the KL divergence computation, requiring doubled GPU memory (from 38.96GB to 88.02GB for LLaMA-2-7BB and 23.7% training time overhead. Our ASFT-LoRA variant (Section (.5) provides practical solution by reducing memory to 40.700 70GB with only 7.3% overhead, though with some performance trade- off.
Our work lacks systematic analysis of what capabilities or behaviors SFT may lose when transition- ing to ASFT. While we observe consistent improvements, a more comprehensive understanding of the trade-offss involved in our reweighting scheme would provide valuable insights for practitioners.
FUTURE WORK
A promising direction for future research is extending ASFT to continual learning settings, where models must adapt to new tasks while preserving previously acquired knowledge. The anchor- ing mechanism in ASFT naturally aligns with continual learning goals: the KL regularization that prevents distributional drift during single-task fine-tuning could adapted to prevent catastrophic forgetting when learning sequential tasks. By maintaining anchors to both the base model and

---

Published as a conference paper at ICLR 2026
checkpoints from previous tasks, ASFT's framework could potentially enable stable, progressive capability acquisition across multiple domains.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 2023.
AT Mathematical Olympiad. Ai mathematical olympiad prize datasets, 2024. URL https:// www. www.kagglle..o.... comp:ee.ccommoooooooooo httm///mmmttiiioooommemoo ai mall-mattemattia ical -wlcccl-olllyyy olyooyympaa.... prize
American Institute of Mathematics. Aime 2024 competition mathematical problems, 2024. URL
https://wwr //www//wwwwww. maa. wwwpaaarrgmaaaaaaaaaaaeae wttgg//aath-cmmpeetteee
aime
S. Andradóttir. D. P. Heyman, and T. J. Ott. On the choice of alternative measures in importance sampling with Markov chains. Operations Research, 1995.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dasgupta, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Hase, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Sastry Girish, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning, 2025.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finetuned lan- guage models. Journal of Machine Learning Research, 25(70): 2024.
Peter Dayan and Geoffrey F. Hinton. Using expectation-maaxiiizziioon for reinforcement learning. Neural Computation, 9())271-2778 1997. doi: (0.1166//eco.1199..92222
Vitaly Feldman. Does learning require memorization? short tale about a long tail. Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12941 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 106.0000.9.33 2021.
12

---

Published as a conference paper at ICLR 2026
Nan Jiang and Lihong Li. Doubly robust off- off-po value evaluation for reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, pp. 652-661,20 2016.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang. and Peter Szolovits. What dis- does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
H. Kahn and Andrew W. Marshall. Methods of reducing sample size in monte carlo computations. Oper. Res., 1953.
Harrison Lee. Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning 2023.
from human feedback with ai feedback. arXiv preprint arXiv:2309.0022
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Soke et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zi- han Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu.
Numinamath. wtt.://tttttttteeffeeeeeeeeeeee.
numina maath-oodels-ann
2024.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In shirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://
openreview net/ fortt//frum?ii
qvx61 DCu7
Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Lixuan He, Xuekai Zhu, Kaiyan Zhang, Bingning Wang. Ning Ding, and Bowen Zhou. Towards a unified view of large language model post-training. 2025. URL htttts:///rrrt
.ogg/orggsss......... wwwp./559...0 04419
Mathematical Association of America. Ame 2023 competition problems, 2023. 7/ 7/www. maa wttp://aaaatccccpeeeii..oo
URL https:
Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimization via importance sampling. In Advances Neural Information Processing Systems, volume 31, 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference health, inference, and learning, pp. 248 260. PMLR, 2022.
Peters and S. Schaal. Reinforcement learning by weight- regression for operational space control. In Proceedings of the 24th Annual International Conference on Machine Learning, pp- 745-750, 2007.
Chongli Qin and Jost Tobias Springenberg. Supervised fine tuning on curated data is reinforcement learning (and can be improved). 2025. hRtps:t::///axiii org abs /2007 12856
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2.5 technical report, 2025. URL https://axxxx
org/abs 212...... 15115
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information ProiiiooPPooooi Processing Systems, volume 36, 2023.
13

---

Published as a conference paper at ICLR 2026
Reuven Y. Rubinstein and Dirk P. Kroese. Simulation and the Monte Carlo Method. Wiley Publish- ing, 3rd edition, 2016. ISBN 11 1118632168.
John Schulman. Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402..33000 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize from human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. 2022.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct. arXiv preprint arXiv:2312.02120, 2023.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng. Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learn- ing perspective with reward rectification, 2025a. URL http:: arxiv. ort.v/./rggbbb//55000 05629
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hswa Husan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.05553 2025b.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Wenhong Zhu, Ruobing Xie, Rui Wang, Xingwu Sun, Di Wang and Pengfei Liu. Proximal super- vised fine-tuning, 2025. URL htt.hhtss////ssttseetttss wwwpa/wwwwww.w////////eee
17884
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint Xrxiv: 08593, 2019.
A LLM USAGE
In the preparation this paper, we only used large language models (LLMs) as an assistive tool for grammar correction and text polishing.
14

---

Published as a conference paper at ICLR 2026
B LIMITATIONS
We evaluate ASFT on three domains mate--mmttttaaalllllllla reasoning, medical knowledge, and code gen- eration. Although results are consistent across these areas, broader evaluation across diverse task types and domains is needed to fully establish the method's generalizability.
Our work lacks systematic analysis of what capabilities or behaviors SFT may lose when transition- ing to ASFT. While we observe consistent improvements, more comprehensive understanding of the trade-off's involved our reweighting scheme would provide valuable insights for practitioners.
C BROADER IMPACTS
ASFT offers a lightweight and computationally efficient alternative standard supervised fine- tuning, requiring only KL regularization rather than complex reward modeling. This simplicity makes advanced fine-tuning techniques more accessible researchers and practitioners with lim- ited computational resources. The method shows potential as a practical path forward for improving language model post-training, particularly in specialized domains requiring both knowledge injec- tion and reasoning capabilities. However, as with any fine-tuning approach, careful evaluation and validation remain essential when deploying models high-stakes applications such as medical or educational settings.
D THEORETICAL FOUNDATIONS
This appendix provides detailed derivations for the key theoretical results presented in Section 3
D.1 DERIVATION OF SFT AS RL LOWER BOUND
We provide a complete proof of Proposition I following the theoretical framework established in (Qin & Springenberg 2025), showing that SFT optimizes a lower bound on the RL objective in sparse reward settings.
Proof of Proposition
We start with the RL objective under sparse rewards R(T) I[ y"]:
J(0) E. [R(r) Er [Ty
y²]
(8)
Since only observe trajectories from the reference distribution Tret importance sampling to rewrite this expectation. Under the assumption supp(ræ) S supp(Tret):
(T(T) [[y = s']
J(O) = E
(9)
Tree
Now apply the fundamental inequality u 2 + log for all u 0. Setting
ma(r)
no + log
(T) 1 log (T(T) log गeof (To
(10)
Tra
Fea (T)
Substituting this into our importance-saneen
expression:
((0)= 2 Er F [[1 + log TT(T) - log Thee (T)e 12y y = y'Il
(1I)
E af [[y y²]] + B [T y² log TT(
(12)
Eron [Tyy = log (r)e(())
(13)
The first and third terms are constants independent of 0. Let Cref = Ey [4y = y²'] = IP Pr,., (T
D+). Then:
J()) 2 Gref + Ersxre [Ty y²] [og + const
(14)
15

---

Published as a conference paper at ICLR 2026
The expectation indicator weighted trajectories can be rewritten expectation the filtered dataset D = =[xx R(x, y²) = 1}:
E B My v"i log To GeelBr [log (T(
(15)
Dropping the constant terms, we obtain:
(()) Cet [log (T(
(16)
This completes the proof. Note that the right-hand side is precisely the SFT objective (up to scaling), establishing that SFT optimizes lower bound on the RL objective.
D.2 FROZEN AUXILIARY DISTRIBUTION AND OPTIMIZATION VALIDITY
We formalize how DFT's stop-gradient mechanism ensures valid lower bound optimization by treat- ing the auxiliary distribution as constant during each update step.
Lemma (Frozen-q Surrogate). At iteration k, define the auxiliary distribution
qu(T) X Tye((T D*) (T)
(17)
where O2 is the current parameter. Then optimizing
(x(T))
log To (T(
max MaxE
(18)
8
(T)
with qu treated as constant (independent of 0) yields valid lower bound on (0) that differs only by a f-independent constant.
Proof. Following the derivation in Appendix D.3 we have:
(T)
(T)
J(0) 2 ETANIC
R((T] Tref(T) (T) + ETwwtes
R(T) log To(T) log
(T)
(T) (T) R(7) log log (u())) (19)
When is constructed from frozen parameter 0g (via stop-gradient), the first and third terms be- come constants independent of the optimization variable 9. Therefore, maximizing the second is equivalent to maximizing valid lower bound on J (().
The stop-gradient operator sg]:] in DFT implements this frozen-q mechanism: at each iteration, gk depends on t& from the previous step, but is treated as fixed when computing gradients with respect the current
This formalization clarifies that the "dropped constant terms" in derivations legitimate be- cause they are genuinely 9-independent within each optimization iteration, making DFT a valid iterative lower bound maximization procedure.
D.3 FROM IMPORTANCE SAMPLING TO IMPORTANCE- WEIGHTED LOWER BOUNDS
We now derive the generalized amportaace--eighter framework that enables tighter bounds through auxiliary distributions.
Starting from the ayportance-sampler RL objective:
Te R((T)
J(0)= E.poxKeet
(20)
(T)
We introduce auxiliary distribution ((T) with supp(g) supp(mo) and rewrite the importance ratio:
q(T) Te(T)
J(Ø)= = REyasttet
(T) (()) R(())
(21)
16

---

Published as a conference paper at ICLR 2026
Now apply the inequality 2 + log to the ratio (s(r), (T(
((F)
(T
2 1 + log
1 + log TT( - log ((T)
(22)
((T)
((()
Substituting this bound:
(((T) (1 + log (o() log R(T))
J(0)= 2 Rrot
(23)
(T)
((T)
q(T) R((T) log log
(T) I((T)
+ Bryweed
(24)
(Ta
((T)
-Brot,e
R((T) (og
(25)
(T)
When treating as fixed (independent of 0, see Lemmal we can drop them from the optimization objective:
1b. the first and third terms are constants, so
q(T)
J(0) 2 E, Bywxiet
R(T) log ml()) + const
(26)
(Te((T)
For sparse rewards R(() = 1ly = y"], this reduces to:
q(T) Bly y²1 log [og
((0) E Brott
+ const
(27)
(ree(()
Converting to expectation over the filtered dataset D+:
g(()
((0) 2 rED!
log To(T)] + const (T)
(28)
where Cref P Pr. (T D
Key Insights:
1. When ((T) (T) recover the standard SFT bound from Proposition
|
2. As q(r) - (T). the bound becomes tighter since the inequality approaches equality.
ma(r)
1 + log
TT(T) q(r)
((r)
3. The choice of q involves fundamental trade-off: tighter bounds (by making g closer to Tp) versus stability (by keeping g close to Tref).
This theoretical framework provides the foundation for both existing methods like DFT and our pro- posed ASFT approach, which aims achieve tight bounds while maintaining optimization stability through principled anchoring mechanisms.
D.4 PROOF OF TIGHTNESS
We provide the detailed proof that the DFT auxiliary distribution g yields a strictly tighter lower bound than standard SFT whenever the policy distribution is non-degenerate.
Proof of Theorem
Let X = PP(T) with T ~ Tree ( and f(x) log x. We compare:
E X [(X)
Bgrr Crer (())),
BDrT Cref
(29)
X
The difference is:
E[X](X)] E(F(X))
BOFT BSFT Cref
(30)
[[X]
Cref
[EXX)(X) ((X]EL((X)))
(31)
[[
Cov(X, f(x))
(32)
X
17

---

Published as a conference paper at ICLR 2026
Since f()) = log is strictly increasing on (0, 1), variables X and f(X) comonotone, yielding
f 2 0 with equality iff is constant. Therefore BDFT 2 BSFT, with strict inequality when Var(X) > 0.
18

---

Published as a conference paper at ICLR 2026
E EXPERIMENTS
E.1
MODEL SCALE RESULTS
Model
Methods
MedQA
MMLU
MedMCQA
Avg.
Basc
29.85
30,52
33.76
31.38
SFT
33.31 13.46
33.52 13.00
33.28 L0.48
33.37
11.99
SFT KL
29.22 10.63
30.63 10.11
33.01 L0.75
30.95
J0.43
iw-SFT
35.35 13.50
38.92 18.40
34.74 10.98
36.34
14.96
LLaMA-2-7B
DPT
29.69 J0.16
26.69 13.83
31.20 12.56
29.19
J2.19
GRPO
30.48 10.63
32.46 1,99
34.64
10.88
32.53
11.15
DAFO
39.75 19.90
48.63 118.11
38.37 14.61
42.25
110.87
ASFT
39.28 19.43
46.37
115.85
40.45 76.69
42,03 110.65
Basc
47.37
65,32
47.2
53,30
DFT
36.84 U10.53
46.77
418.55
42.39
L4.82
42,00 111.30
LLaMA-2-70B
SFT
41.24 J6.13
47.02 118.30
36.43 110.78
41,56 111.74
ASFT
49.57 12.20
65.86 10.54
50.61 13.40
55.35 12.05
Base
62.53
74.84
58.45
65.27
SFT
59.07 13.46
71.19 13.65
49,80 L8.65
60.02 15.25
SFT
KL
61.12 41.41
69.77 45.07
51.52 L6.93
60.80 14.47
Qwen2.5-7B
DFT
36.45 J26.08
54.65
120.19
45.90 (12.55
45.67 119.60
GRPO
63.00 10.47
76.16
11.32
58.93 10.48
66.03 10.76
DAPO
63.94 11.41
73.82 41.02
58.88
65.55 10.28
ASFT
61.98 J0.55
76.60 111.66
59.00 10.55
65.86 10.59
Base
61.90
62.50
45.09
56.50
Qwen2.5-32B
SFT
67.09 15.19
76.12 113.62
61.20 11
68. 1.66
DFT
68.42 16.52
57.03 45.47
42.51 L2.58
55.99 40.51
ASFT
71.80 19.90
79.23 116.73
58.76 113.67
69.93 13.43
Base
75.57
78.28
64.91
72.92
SFT
28.04 (47.53
34.57
J43.71
38.92 (25.99
33.84 139.08
Qwen2.5-72B
DFT
71.01 14.56
79.04
10.76
64.69 J0.22
71.58
11.34
ASFT
77.69 12.12
87.08
18.80
71.00 66.09
78.59 15.67
Table 5: Performance of various fine-tuning methods on medical domain datasets for both LLaMA- 2 and Qwen2.5 series. Bold numbers indicate the best performance each group, and rows with blue background highlight our ASFT approach. Arrows with 1 and indicate performance im- provements and degradations relative to the base model. All values are reported in percentages.
19

---

Published as a conference paper at ICLR 2026
E.2
TRAINING
Average Performance across Batch Size
Average Performance across Learning Rate
Mode
Matel
6.4
T
6.8
6.10
545
5.05
5.0
5.00
Batch Size
Learning
Figure 5: Ablation study of ASFT on MedCAQA (10k) using L-MMA-2-7B. Right: Effect of hatch size, indicating stable performance across wide range (32-256). ASFT consistently outperforms SFT and DFT under all settings. Left: Effect of different learning rates on average performance, showing that intermediate rates (le-5 and le-4) achieve the best results.
20

---

Published as a conference paper at ICLR 2026
F DATASET EXAMPLE
F.1 MEDICAL KNOWLEDGE DATASETS
Medical Knowledge Training Set
Instruction:
Disulfiram a type af:-
A. Aversion therapy
B. Anticraving therapy
C. Detoxification
D. Opioid management therapy
Response:
Deterrent agents/Aversive agents Disulfiram (tetracthyl thiuram disulfide) When alcohol ingested by person who is on disulfiram, alcohol-derived acetaldehyde cannot oxidized to acetate and this leads an accumulation of acetaldehyde in blood. This causes the impoant disulfiam-ethanol reac- tion (DER) characterized by flushing, tachycardia, hypotension, tachypnoea, palpitations, headache, sweating, nausca, vomiting, giddiness and sense impending doom associated with severe anxicty. When we try to decrease the frequency behavior because can cause a really bad consequence will be called as an aversion or Aversive therapy. Anti-craving agents Acamprosate, naltrexone and SSRIs (such as fluoxetine) are among the medications tried anti-craving agents in alcohol depen- dence. Detoxification Detoxification the treatment of alcohol withdrawal symptoms, i.e. symptoms produced by the removal of the 'toxin' (alcohol). The drugs of choice for detoxification are usually benzodiazepines.
So the answer to this question is A
Figure 6: An example from the medical knowledge training set illustrating instruction-responee pair on alcohol dependence therapy
Medical Knowledge Test Set
Question:
Which of the following is not for myelinated fibers:
Options:
A. Impulse through myelinated fibers slower than non-myelinated fibers
B. Membrane currents are generated at nodes of Ranvier
C. Saltatory conduction of impulses is seen
D. Local anesthesia is effective only when the nerve not covered by myclin sheath
Answer: A
Figure 7: An example from the medical knowledge test set illustrating multiple-choice question on physiology (myelinated nerve fibers)
21

---

Published as a conference paper at ICLR 2026
F.2
MATH REASONING DATASETS
Math Reasoning Training Set
Instruction:
Let a, b, and he the roots of the equation - 2x - 5 0. Find
1-2
Response:
+ 2 transform the polynomial that 6 and 2 are the roots for We then have:
First, substitute
+ 2 are the new roots, i.e. 2,
(= 2) 2( + 2) 5 2² 62 12: + 2: 4 -
6z 10:
We interested in - 6-2 c-2" which are the reciprocals of the roots of x" + 6z2 10x 1.
By Vieta's formulas, the of the reciprocals of the roots of polynomial =² pz + q2 + q/r. For our polynomial, p 6, q 10, and
so:
10 -1
Figure 8: An example from the math reasoning training set illustrating an instructiin-respposee pair on algebraic root manipulation
Math Reasoning Test Set
Question:
What the smallest positive perfect cube that can be written the sum three consecutive integers?
Answer:
27
Solution:
The sum of three consecutive integers takes the form (k: - 1) (k) + (I: + 1) 34 and hence is multiple of 3. Conversely, if number is multiple of 3, then n1/3 1, n/3, and n/3 + are three consecutive integers that give Therefore, number is of three consccutive integers if and only if is multiple of 3. The smallest positive perfect cube that multiple of is 3 27
Figure 9: An example from the math reasoning test set illustrating a problem with its solution on sums of consecutive integers
22


Reviews:
[{"review_id": "mXDKPJrFKL", "reviewer": "Reviewer_Tq5U", "summary": "The paper analyzes Supervised Fine-Tuning (SFT) and Dynamic Fine-Tuning (DFT) through the lens of reward-weighted regression (RWR), showing that both can be seen as optimizing lower bounds of an underlying RL objective. Within this framework, DFT corresponds to using an importance-weighted auxiliary distribution that yields a strictly tighter bound than SFT but suffers from instability due to uncontrolled distributional drift. To address this, the authors propose Anchored Supervised Fine-Tuning (ASFT), which augments DFT with a KL regularizer towards a fixed base model, acting as an “anchor” that stabilizes training. Experiments on math, medical QA, and code tasks demonstrate that ASFT outperforms SFT and DFT, and on medical QA it approaches RL methods at lower cost.", "questions": "1.Empirical scope and “bridge SFT–RL” framing. Could you either (i) provide preliminary results or analysis of ASFT under a genuinely RLHF-style setting (e.g., preference-based rewards or open-ended instruction-following), or (ii) clarify and narrow the scope of the “bridging SFT and RL” claim to supervised/offline, correctness-style regimes? Any concrete evidence or discussion in this direction would help readers understand how far your conclusions are meant to generalize.\n\n2.RL comparisons beyond medical QA. For math reasoning and code, do you have any results—perhaps at smaller scale or with fewer training steps—where ASFT is directly compared to GRPO/DAPO or another RL baseline, so that we can calibrate how close ASFT gets to “RL-level” performance beyond medical QA? If such experiments are infeasible, could you provide a more detailed justification or expectation (e.g., based on reward sparsity or domain differences) for whether you believe the medical-domain findings should transfer to these tasks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper makes a clear and conceptually grounded contribution to understanding the link between supervised fine-tuning and reinforcement learning–style objectives. By framing SFT and DFT under a unified reward-weighted regression (RWR) objective, it shows how auxiliary distributions determine the tightness and stability of RL lower bounds. Identifying DFT as a particular importance-weighted auxiliary distribution, and then introducing the “anchored” variant (ASFT) with a KL regularizer towards the base model, is an elegant way to stabilize DFT while preserving its tighter connection to the RL objective.\n\nThe theoretical development is careful and transparent: each step from SFT → DFT → ASFT is well-motivated, with assumptions stated and a useful strict-tightness result for DFT. Empirically, the work provides consistent evidence that ASFT improves over SFT and DFT across math reasoning, medical QA, and code generation, and acts as a strong initializer for RL methods such as DAPO in the medical domain. Overall, the paper is clearly written, theoretically sound within its setting, and practically relevant. Its main value lies less in radically new algorithms and more in clarifying and stabilizing the intermediate regime between SFT and RL, offering both conceptual insight and a simple recipe that practitioners can readily adopt.", "weaknesses": "1.Empirical scope w.r.t. the “bridge SFT–RL” claim. The experimental coverage across math reasoning, medical QA, and code generation is solid, but all setups remain in a supervised/offline, correctness-style regime with static datasets. There are no studies on preference-based post-training, open-ended instruction following, or safety/factuality trade-offs, which are central targets of RLHF-style training. Given the paper’s framing as a general method that “bridges SFT and RL”, the current evidence still primarily supports gains on a few curated accuracy benchmarks. Either tempering the scope of the claims or adding at least one preference/RLHF-style experiment would make the story more convincing.\n\n2.Uneven RL comparisons across domains. The paper provides a careful comparison between ASFT and RL-style methods (GRPO/DAPO), and shows that ASFT is competitive and that ASFT+DAPO further improves performance—but this analysis is restricted to medical QA. For math reasoning and code, the baselines are primarily SFT/DFT/iw-SFT, without RL counterparts. As a result, it is difficult to calibrate how close ASFT gets to “RL-level” performance in these domains, or whether its advantages as a cheaper alternative or initializer hold once RL is available. Even a smaller-scale RL comparison on one math or code task would significantly strengthen the empirical case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ViEiOH01Zi", "reviewer": "Reviewer_XKZp", "summary": "The paper proposes Anchored Supervised Fine-Tuning (ASFT), a variant of SFT and DFT (dynamic fine-tuning) based on a Reward-Weighted Regression (RWR) perspective. It shows that SFT is a lower bound on the RL objective and that DFT provides a tighter bound but suffers from instability. ASFT introduces a reverse-KL anchor to the base model to stabilize DFT while preserving its theoretical benefits. Empirical studies on math, medical QA, and code generation tasks demonstrate consistent improvements over SFT/DFT and comparable or better results than RLHF-style methods (GRPO/DAPO).", "questions": "Please consider to address questions in the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Using a reverse-KL anchor to control DFT drift is a natural, well-motivated modification that stabilizes training. \n\n+ The RWR derivation linking SFT, DFT, and ASFT is internally consistent and clearly written. The proofs in the appendix (esp. the tightness lemma and DFT-as-RWR derivation) are correct under the stated sparse-reward and support assumptions.\n\n+ The paper evaluates on multiple domains (math, medical, and code) across LLaMA-2/3 and Qwen models, and includes practical metrics such as GPU memory and wall-time comparisons. The authors acknowledge compute cost, and ASFT-LoRA offers a realistic, resource-efficient version.\n\n+ Ablations studies on reverse vs. forward KL, $\\lambda$ sweeps, and model-size scaling are reported and analyzed.", "weaknesses": "- The paper asserts that the KL anchoring “preserves tightness,” but technically ASFT optimizes a regularized surrogate, not the same lower bound. This needs clarification. \n\n- The theory relies on sparse rewards and full-support overlap; it’s unclear whether results extend to open-ended generative or preference-learning tasks.\n\n- The transition from sequence-level RWR theory to token-level training is described heuristically rather than derived rigorously.\n\n- Results lack multiple random seeds or confidence intervals; improvements may be within run variance for some metrics.\n\n- Full ASFT doubles memory usage (requires both model and reference for KL), and while ASFT-LoRA helps, it trades accuracy for efficiency.\n\nMinor\n- Evaluation focuses on reasoning and domain-specific QA; general instruction-following, safety, or long-context benchmarks are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BSS6TmYGjz", "reviewer": "Reviewer_RAzX", "summary": "- The paper proposes that SFT maximizes a lower bound on the RL objective\n- It introduces auxiliary distribution to control the tightness of the bound, and shows that DFT is a special case which achieves tighter lower bound than SFT\n- It shows that DFT suffers from distribution shiift as training proresses\n- To mitigate the dirtribution shift, ASFT augments the DFT objective with a KL regularization term between the currect model and reference model (base model)", "questions": "No questions other than those mentioned in the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- ASFT shows better performance than SFT and DFT on math reasoning and medical knowledge\n- The paper theoretically derives the importance weighted lower bounds and proves the tightness of the lower bound for DFT", "weaknesses": "- ASFT requires an additional copy of base model to be stored in memory, as well as an additional forward pass through it\n- While using LoRA mitigates the memory overhead, it causes a significant performance drop (the performance of ASFT+LoRA is almost equivalent to simple SFT as seen in Table 4)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qk63921C2J", "reviewer": "Reviewer_DPLk", "summary": "The authors indicate that DFT suffers from distributional shift and propose ASFT to prevent it. ASFT is a post-training method which uses KL divergence to anchor it to a reference base model to prevent distributional shift during training.", "questions": "* Would anchoring prevent overfitting when we have a small dataset?\n* Have the authors considered reporting the mean and standard deviation across several independent runs? This will strengthen the findings further\n* Edit suggestions for better readability\n1. The second sentence in lines 72 to 75 is a repetition \n2. Font size is very small for Table 1\n3. Typo in line 310\n4. Font in Figure 3 is illegible \n5.  Consistent color coding of methods across plots will improve readability - it is a little confusing now and readers have to check the legend across plots every time", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Identifies the shortcoming of DFT and suggests a novel way to handle it\n* The method is theoretically grounded using a reward-weighted regression framework\n* Empirical analysis and ablation studies are thorough and evaluation is done for mathematical reasoning and knowledge intensive tasks\n* Clearly stated contributions backed by theory and experimental analysis", "weaknesses": "* Requires the base model as a reference - memory and computational overhead. \n* ASFT-LoRA is proposed to avoid memory overhead, the performance significantly drops when compared to ASFT and is very close to the performance of SFT\n* DAPO still performs better than ASFT on an average, more analysis and discussion on why and where RL remains preferable would be nice\n* No reporting of standard deviation of the runs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)