NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning
Authors: Huayu Chen
Kaiwen Zheng
Qinsheng Zhang
Ganqu Cui
Yin Cui
Haotian Ye
Tsung-Yi Lin
Ming-Yu Liu
Jun Zhu
Haoxiang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Supervised Learning
GRPO
LLM
Primary area: generative models

Abstract: Reinforcement Learning (RL) has played a central role in the recent surge of LLMs' math abilities by enabling verification-driven training through binary verifier signals. In contrast, Supervised Learning (SL) is rarely considered for such verification-driven training, largely due to its heavy reliance on reference answers and inability to reflect on mistakes. In this work, we challenge the prevailing notion that self-improvement is exclusive to RL and propose Negative-aware Fine-Tuning (NFT) --- a supervised approach that enables LLMs to reflect on their failures and improve autonomously with no external teachers. In online training, instead of throwing away self-generated negative answers, NFT constructs an \textit{implicit} negative policy to model them. This implicit policy is parameterized with the same positive LLM we target to optimize on positive data, enabling direct policy optimization on all LLMs' generations. 
We conduct experiments on 7B and 32B models in math reasoning tasks. Results consistently show that through the additional leverage of negative feedback, NFT significantly improves over SL baselines like rejection fine-tuning, matching, or even surpassing leading RL algorithms like GRPO and DAPO. Furthermore, we demonstrate that NFT and GRPO are actually equivalent in strict-on-policy training, even though they have entirely different theoretical foundations. Our experiments and theoretical findings bridge the gap between SL and RL methods in binary-feedback learning systems.

Published as a conference paper at ICLR 2026
NFT: BRIDGING SUPERVISED LEARNING AND REIN- FORCEMENT LEARNING IN MATH REASONING
Huayu Chen
Kaiwen Zheng!?
Qinsheng Zhang?
Ganqu Cui'
Yin Cui
Haotian Ye
Tsung-Yi Lin
Ming-Yu Yu Liu?
Jun Zhu
Haoxiang Wang
'Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch ML Center, Al Institute, Tsinghua
2NVIDIA
SStanford
ABSTRACT
Reinforcement Learning (RL) has played central role in the recent surge of LLMs' math abilities by enabling verification-driven training through binary ver- ifier signals. In contrast, Supervised Learning (SL) is rarely considered for such verification-driven training, largely due to its heavy reliance on reference answers and inability to reflect on mistakes. In this work, we challenge the prevailing no- tion that self-improvement is exclusive to RL and propose Negative-aware Fine- Tuning (NFT) - a supervised approach that enables LLMs to reflect on their failures and improve autonomously with no external teachers. In online training, instead of throwing away self-generated negative answers, NFT constructs an im- plicit negative policy to model them. This implicit policy is parameterized with the same positive LLM we target to optimize on positive data, enabling direct policy optimization all LLMs' generations. We conduct experiments 7B and 32B models in math reasoning tasks. Results consistently show that through the additional leverage negative feedback, NFT significantly improves over SL baselines like rejection fine-tuning, matching, or even surpassing leading RL algo- rithms like GRPO and DAPO. Furthermore, we demonstrate that NFT and GRPO are actually equivalent in strict-on- -oollicy training, even though they have entirely different theoretical foundations. Our experiments and theoretical findings bridge the gap between SL and RI. methods in binary-feedback learning systems.
INTRODUCTION
The recent surge in math reasoning abilities of Large Language Models (LLMs) is largely driven by a fundamental shift in their learning paradigm, from imitation to self-improvement (DeepSeek-A]
2025 OpenAT 22244] Chen et al. (2025c). Instead of relying reference answers supplied by human annotators or stronger models (Liu et al. 2024 OpenAl 2023), the new paradigm requires only a question dataset with a binary verifier judge the correctness of self-generated answers. By reflect- ing on their own generation, LLMs can improve autonomously. This approach not only eliminates the need for costly data annotation but also removes competence ceilings imposed by external teach- ers, offering promising path toward general intelligence (DeepSeck-Al 2025 NVIDIA 2025).
Reinforcement Learning (RL) appears to be a natural fit for such serification-driven training. Spe- cific algorithms like PPO (Schulman et al. (017)) and GRPO (Shao et al., 2024) are explicitly designed to maximize reward signals, which can conveniently take the form of a binary verifier outcome. In contrast, Supervised Learning (SL) is rarely considered for realizing such verification- driven training. A common view holds that SL is inherently designed to memorize the positive data, rendering unsuitable for self-refiective learning from negative mistakes (Chu al. 2025a).
In this work, we challenge the prevailing notion that verification-drive driven training is exclusive to RL, and demonstrate it can be similarly achieved within the supervised learning paradigm.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Reinforcement Learning
Supervised Learning
REINFORCE
PPO
GRPO
NFT
RFT
Equivalent gradient when palicy
Consistent Maximum Likelinood Objective
On-Policy
Off-Palicy
Dense Rewards
Binary Labels
Leverage Negative Data
Positive Only*
Figure I: A spectrum of online algorithms for LLM fine-tuning. NFT bridges reinforcement learn- ing and supervised learning methods through the leverage of negative feedback via supervision.
We start with a simple SL baseline: Rejection 2023). At each iteration, an LLM generates answers to questions. verifier helps reject all negative answers. The remaining positive ones are compiled into a dataset to fine-tune the LLM itself in a supervised manner. RFT has been demonstrated effective (Bai et al. 2022. Yuan et al.) 2023a Song et al. 2023 Touvron et al. 2023 Xiong et al. (025)_ However, it prevents any learning from negative feedback. LLMs are encouraged to reinforce what they already perform well, rather than reflect on their mistakes
(RFT) Yuaa et al. 2023b Dong et al.
ability we believe critical for achieving general intelligence.
To overcome this limitation, we propose Negative-aware Fine-Tuning (NFT), an online learning algorithm that enables LLMs to learn from their negative generations (Sec. 3). Like RFT, NFT fine-tunes a positive LLM on positive answers via supervision. Crucially, instead of throwing away negative answers, NFT also constructs policy is parameterized with the same positive LLM we target to optimize on positive data, enabling direct policy optimization all LLMs' generations (Figure 2 NFT has minimal memory overhead, as only a single model is maintained throughout training.
implicit negative policy to model them. This implicit
To understand the connection between NFT and RL approaches, we conduct in-depth comparison between NET and GRPO (Sec. 4. Surprisingly, we find the two methods are actually equivalent in strict-on-policy training. despite that they originate from entirely different theoretical frameworks (Figure T. Notably, the "advantage normalization" characteristic of GRPO is already implicitly re- flected in NFT's loss function. Their main difference arises in off-policy settings, regarding different strategies for clipping model gradients when the learned policy deviates from the old policy. These observations suggest a strong connection between SL and RL binary-feedback learning systems.
We evaluate NFT on 7B and 32B Qwen models and report two key findings: 1. Supervised Learning alone can significantly enhance I.LMs' math reasoning with no external teachers. NFT matches or even surpasses ofate-of-the-art RI. algorithms like GRPO (Shao et al.) (024] and DAPO (Yu et al.) 2025). 2. The performance gap between RFT and RL in online training largely stems from SL' past inability to leverage negative feedback, rather than any inherent superiority of RL. Through the additional leverage of negative data, NFT substantially mitigates this gap.
BACKGROUND
2.1 MAXIMUM LIKELIHOOD VS. POLICY GRADIENT
Supervised Learning essentially aims to learn model (ala)) to fit the data distribution TTa (()/g)). This can be realized by employing the maximum-likellhon objective:
max E (a)n(( log no(alg) g) <> min DKr. (To(al))
In LLM fine-tuning, q usually means the question prompt, and a the answer. To perform Maximum- likelihood training, we need a dataset {q, a π {a|g)}} draw training samples from.
Reinforcement Learning, by contrast, maximizes pre-defined reward r(g. a) for a Te(a
max J(0): Ea-ro(aly) a)) 0
Directly bropaprop through (() is non-trivial, r(-) can be an arbitrary scalar function whose gradient is unknown. Luckily, ( can be estimated, making policy optimization feasible.
VoJ(O) Eaanolaa Φ) Vo (r(g, a) log (a(ag))
(1)
2

---

Published as a conference paper at ICLR 2026
question
Target Positive Policy
(x+y)
Tt (g.
Acc: Ta
max log(
a
4
answers
Tg (q, a))
Data Generation
Implicit Negative Policy
Illustration of the NFT algorithm. Data Collection: An LLM т generates answers to of math questions. Generation results split into two sub-datasets based on their answer cor- rectness. Policy Optimization: By constructing an implicir policy for modeling negative data, NFT enables direct policy optimization on both positive and negative answers
Figure 2:
cikimuum-ikelihoodd
Eq. [ is known the Policy Gradient (PG) algorithm (Sutton al. 1999). In sequential decision- making problems such as language reasoning, a can be interpreted as the token decision for each step and r(g. a) can be replaced with advantage functions A(q, g, a) (Schulman al. 2015).
2.2 MATH REASONING RL: FROM POLICY GRADIENT TO GRPO
Eq. T requires training to be on-policy, where Te can only be updated single time after data collec- tion. To break this limitation, importance sampling can be applied (Schulman al. 2017). Suppose the policy for collecting the RL dataset is denoted as Told. we have
Ta(a)
V (0) = Eantaalaa (alg)
Tota (a) r(g. Vo
o log (a(()) = Eavawilag [r(q. a) Vo a))
where Ro(g, a) =
Tal malalq) is the likelihood Ratio between two policies.
{a)(a/
In math reasoning tasks, PPO (Schulman et al. 2017) and subsequent GRPO (Shao et al. 2024) algorithms further apply gradient clipping constrain the distance between my and Mold!
CRROO (0)
Σ
Σ
min R/ a) a) (iipiiiRRg -
(3)
Radla malla: q.ace)) and Áq,a is the estimated advantage value. Note that have dropped some auxiliary loss terms, such as KL and entropy regularization, as they have been pointed out to be unnecessary by recent studies like DAPO (Yu et al. 2025).
where Rg a)
GRPO proposes efficient way estimate Åq.a- Collect K answers a= 1:K and their binary reward 1.KK (0, 1} for each question. The advantage is defined to be the normalized reward:
Aq.a
[r(q. a) mean(r }] kk)
(4)
Later studies (Liu et al. 2025b) suggest removing the st cl term from Eq.
3
METHOD
3.1 PROBLEM SETUP
Dataset. Given a of N math questions {qI:N), pretrained LLM aaa()))) and verifier for judging the correctness. In every iteration, we generate dataset D = {q, a l:k n,, rl: p where r' {0, 13 is the correctness label, and K the number of answers collected for each question.
Dataset D ~ can be split into two subsets: D and D=. D* contains all positive answers, and D contains the rest negative ones. We denote the underlying answer distribution of D as x (:lq))
Learning Target. We to optimize the old policy into new policy क - The target n (alg) can be formalized using Bayes' Rule:
Total (annaa))((--lp/) a)
++(a|)) = T (alq,r=1))
= ΣA Tou (al)p((==1lg...p a)
(5)
3

---

Published as a conference paper at ICLR 2026
π
πI
TT(aqq)
Ta (alg)
(1-Th - (1-T))T((alql (alq)
Tt
πI
Generation Policy
Target Positive Policy
Negative Policy
750
π
Left: Policy Splitting. The generation policy can be split into positive policy and a negative policy, and re-expressed as their linear combination. Right: Policy Improvement. By iteratively optimizing towards its positive split, an LLM policy To can improve continuously.
Figure 3:
where A means all possible language space for a.
Discussion, An obvious solution for learning x is to fine-tune solely on correct answers (D+) and discard D totally (RFT) (Dong al. 2023 Xiong et al. 2025). However, this approach prevents the model from any learning on its negative feedback (D ). We posit that the ability to reflect on one's own failures is not merely desirable, but central general intelligence, marking shift from pure imitation to self-reflective learning. Though traditionally viewed as distinctive strength of RL
(Chu et al., 2025a Yue al. 2025), we ask: Can self-reflective improvement be similarly achieved within the SL paradigm?
3.2 DIRECT OPTIMIZATION OF LANGUAGE MODELS WITH NEGATIVE ANSWERS
In this section, we discuss how to leverage negative data D to directly optimize To. Despite seemingly impossible at first thought, we find the target policy n and the negative policy are tightly coupled, making feasible training Tg directly from D
First, we formalize the definition of the negative policy ग similar to Eq.
Tota a q) p00= a)
π T (a qq) == π (a q, r=0) = ΣA Tota (a ()//)/(11 p(r-pp11110 1 a)
(6)
Combining Eq. and Eq. 6 we make a key observation that
Fa (a)g) + ra] π (alg) = vata (a) (a))),
(7)
where ra = Ea Tola (aa((a)))pr=1lqq0 a = p(r=1lq) is the correctness rate of LLM Told over question In practice, R mean can be estimated using the rewards in dataset D.
Implicit negative policy. Eq. reveals tight coupling between x and (Figure 3). Given that we already have π as the pretrained L.M and Fg is estimable, learning л from negative data should, in principle, shape the target policy क in manner analogous to SL on positive data.
To realize this idea, we construct an implicit negative policy, denoted म by re-parameterizing the target policy oh using the relationship in Eq. 7
(alq) = Tolal (alq)) nT (alq)
1 Tq
Thus, training To on negative answers directly leads to optimizing the underlying positive LLM क5. (Figure 2h. We have the following guarantee:
Theorem 3.1 (Policy Optimization with Negative Answers). Consider the maximum-likellhann objective for training the implicit negative policy Tg
B(q.a)-D- log Tota (a q) Tq (alg)
max LE p(g))= (a g) [log To (alq)) & min
(8)
6
1 riq
Assuming unlimited data and model capacity, the optimal solution for solving Eq. is
Vq, a: के (alq) + (alg)

---

Published as a conference paper at ICLR 2026
Algorithm Negative-aware Fine-Tuning (NFT)
Input: Language model æ, prompt set qlt.. verifier r(-).
2: Def max.x((x,e)) e):
I/Straight through Max Operator I/Clip Value while Keeping Gradient
3: Return wwwppggaddienem.m..............e ara (max(2e <) - x)
4: for each iteration do
5: for each sampled prompt q do
6: Generate K answers a 1:K and verify their correctness 1
Data Collection Calculate correctness rate fg meanfr "i and token-level likelihood (Tou(a;) q, a<z)) 3:1aaly ) M
7:
8: D + {4. fgg L.KK ,I 1:K 1r0 rq
m
Prompt Filtering
9:
end for
10: Initialize T..
for each mini batch {q. a, fg: me) in D do
12:
a)) r (aala.aa c.ac)) Vi
I1 Positive Likelihood Ratio
smi(a: q..)))
13:
=r==0:
14:
(g =
II Implicit Negative Likelthood Ratio If Clip Negative Likelihood Ratio Maximum Likelihood Training
15:
[g max
16:
0 0+- A E, log a.
(Eq. 10
end for
18: x + Tg. start the next iteration
19:
end for
Proof in Appendix A Theorem 3.1 demonstrates the feasibility of policy optimization on negative data only. To further utilize positive data, we additionally conduct supervised training on D+:
+ (alg) marla
(alg) Tota (a
1 Fg
(0)==) =r
NET
log
+ r)
log
1 Fq
Note that we subtract a baseline term log Fold (ald from the loss. Since this term is unrelated to 0, it does not affect the loss gradient and thus the optimal solution. Told q)) is the old data likelihood before optimizer update. At the start of training, we have Ti = r such that C (a.q.r) 0.
We name our method Negative-aware Fine-Tuning (NFT) as it enables the additional leverage of negative data for policy optimization compared with RFT.
Continuous Reward. Though we mainly consider binary rewards for math reasoning, NFT is directly applicable to continuous rewards. Following[Levine 2018), we can define optimality reward r E [0, 1]. meaning the answer has a probability of for being positive, and 1 -- for being negative.
The loss definition Eq. and its convergence property stay unchanged (Proof in Appendix B
Memory Efficiency. NFT is memory-efficient. In practice, we keep only single model copy in memory. The old policy likelihood Tau q) can be pre-computed during data generation.
3.3 PRACTICAL ALGORITHM
We introduce several improvements over Eq. 9 and propose practical objective of NFT:
fg Rt (g a)
CFF (8) Σ w(q)) Σ
log (g. + - r) log max_v(
(10)
1 1 fg
g.a.r
To (ar q, ace)
1 Er(a,a)
where
(g
and fg
r(q,a) a).
Tatal (ara ace)"
K
ala
Pseudo code is in Algorithm T Below, we explain our key design choices.
Token-level loss. Eq. essentially deals with sequence data, where answer likelihood ( q) =
II, (a, lq, a<)) is heavily correlated with answer length. This introduces high variance in gradient estimation and causes numerical instabilities during training. Following Schulman et al. (2017); : Yu
et al. (2025), view each token decision individual unit and up their loss in 1qg 10
5

---

Published as a conference paper at ICLR 2026
Clipping negative likelihood ratio. The negative loss in Eq. 10 involves logarithm whose argu- ment must remain positive, imposing (1 fg Rg fq) > When Rg is unoptimized, this requirement may not be satisfied, potentially leading to training collapse. We therefore enforce minimum value of e >> 0 for the negative likelihood ratio. To preserve gradient flow after clipping, further apply straight-thruugh gradient (Bengio al. 2013). Details are in Algorithm
Prompt weighting. To focus training on more informative instances, we weight each prompt by w()), assigning higher importance hard questions with low Tq. An ablation study is posted in This design also helps align NFT with RL algorithms like GRPO (Sec.
Sec. 5.4
4
UNDERSTANDING THE GAP BETWEEN NFT AND GRPO
Despite originating from entirely different theoretical foundations, NFT and GRPO exhibit signif- icant similarities. Notably, we find GRPO and NFT are equivalent in on-policy training. To understand this, we calculate and compare their loss gradients:
Proposition 4.1 (Algorithm Gradient Comparision). Suppose there are fgK positive answers and (1 - K negative ones for given question
(a) GRPO Gradient: Consider only binary reward {0, 13 in Eq. GRPO loss gradient satisfies
FaCorro(0) (0) = Er I [Rg(g, a) < <1+ <1+ I R2 ( volt(laa)
where A =
Fg T 1-fa and Ag -
respectively normalized advantages for answers.
(b) NFT Gradient: Let w(q) v(1 frg) NFT loss gradient satisfies
COC (FT( (0)=
(rA (g a)
1
+ (1 - (1-r)Ag max
fg Rb
E)
) VeRi(qq a).
pa
Proofs are provided in Appendix A Comparing VeLGRPO (8) and VoLo (0). the only differ-
Wor
W
NFT GRPO
ence between GRPO and NFT is their strategy for clipping model gradients when training data becomes off off-policy (Figure 4. GRPO simply zeros out the gradient when the learned policy my shifts far away from the old policy æ,, while NFT takes "softer" decay schedule.
GRPO
0
1.0
on policy
on-palicy
Surprisingly, we find GRPO and NFT to be equivalent when training is totally on-policy, despite their distinctively different derivations:
R5
1-c
1.0
1
Figure 4: Gradient weight for NFT and GRPO.
Proposition 4.2 (On-policy Gradient Equivalence). Following the set of Proposition 4.1 and let E < 1, GRPO and NFT loss gradient are equivalent in on- on-policy training:
Ry(q, a)
VOL VgLGRPO (0)
Implicit group normalization. Proposition shows the "normalized advantage" term is implicitly present within NFT's loss function. This partially justifies the Group Normalization design choice for GRPO, which was initially introduced only as an empirical technique (Shao et al.) 2024). In Appendix A. we further demonstrate that by adjusting w(q) = fq. NFT also aligns with Dr. GRPO (Liu et al. 2025b). Our findings suggest strong connection between RL and SL frameworks.
sshows
5 EXPERIMENTS
We seek to answer the following questions through our experiments.
How does NET perform comparison with existing RL algorithms such as GRPO? (Sec. 5.2)
2. How does negative data contribute to NFT's performance gain? (Sec. (5.3)
Which empirical design choices are important to the effectiveness NFT? (Sec. 5.4

---

Published as a conference paper at ICLR 2026
Qwer-Melh-78
Eurus-2-78-PRIME
80
Owen-DPO-R1-
Ze-roo
PRIME 78-Zer
NFT-7B-Zero (Durs)
RWF++
Accurac
30
26.
20
AIME24
Minerva Math
OlympiadBench
MATH500
Average
Figure 5: Comparison the released NFT-7B with other zero-style math models of Qwen series.
Table I: NFT performs competiivelly compared with other algorithms. We report avg8 32 for AIME24, AIME25, and AMC23 and avgel for others. Numbers within 1 % of the max are bolded.
Model
AIME24
MATH500
AIME25
AMC23
Olympiad
Minerva
Average
Owen2.5-Math-
13.3
69.0
5.5
45.8
34.7
21.3
31.6
Preference fine-turing
+ DPO
29.8
79.8
13.8
83.2
48.0
39.0
48.9
Reinforcement fine-tuning
GRPO
30.2
80.4
17.1
79.5
51.8
38.2
49.5
Dr. GRPO
31.8
15.7
80.2
49.6
38.2
49.8
+ DAPO
33.1
81.6
18.7
85.0
49.9
39.3
51.2
Supervised fine-tuming
RFT
33.7
79.8
13.4
79.7
44.3
38.6
48.3
NFT
32.0
83.2
18.3
88.5
47.3
40.8
51.7
Qwen2.5-32B
4.1
68.6
1.0
45.0
31.1
27.9
29.6
DAPO
44.1
89.2
33.4
90.9
54.1
47.5
59.9
RFT
29.9
86.2
19.1
92.4
45.3
44.1
52.8
NET
37.8
31.5
93.8
55.0
48.9
59,2
5.1 EXPERIMENT SETUPS
Training. We perform online fine-tuning on Qwen2. 5-Math-7a and Qwen2. 5-32B (Yang al, 2024) to enhance their math abilities without relying on external teachers. The training dataset the publicly available AAPO-Math-174 Yu et al.) 2025). which consists solely of math ques- tions paired with ground-trunt truth answers in integer form. During training, all models are tine-tuned for approximately 5,000 gradient steps with batch size of 512. Generation temperature is 1.0.
Evaluation. We evaluate models on six validation benchmarks and report their average accuracy: AIME 2024, AIME 2025, AMC 2023 (Li et al.] 2024). MATH500 (Hendrycks al. 2021
OlympiadBench (He et al. 2024), and Minerva Math (Lewkowyzz et (022).
Baseline methods. We compare against
of online fine-tuning algorithms, including Iterative
DPO (Rafailov et al. 2023] Xiong et al., 2023 Guo et al. 2024), GRPO (Shao et al. 2024), Dr.
GRPO (Liu al. 2025b), DAPO (Yu al. 2025 and RFT (Dong al. 2023 Yuan al. 2023b
5.2 NFT PERFORMANCE EVALUATION
Model comparison. By applying NFT Owen2.5-Math-77 we release NFT-7B-Zero (Fig- ure 5). NFT-7B-24 Zero achieves competitive performance all benchmarks compared to other zero-style 7B math models (Cui et al. 2025 Liu et al. 2025b: Xiong et al., 2025 2023). This pro- vides strong empirical evidence for the effectiveness of the NFT algorithm and demonstrates that SL alone can enable effective verification-drive
training in math tasks.
Algorithm comparison. To isolate the contribution of the algorithm itself, benchmarked various online algorithms using identical training data, infrastructure, and general hyperparameters (Table 1.. Results show that NET matches surpasses state-of-the-art methods such DAPO. Fig-

---

Published as a conference paper at ICLR 2026
Accuracy
Training Accuracy
60
NFT (aurs)
NFT (ours)
RFT
RFT
DAPO
DAPO
30
1000
2000
5000
2000
5000
Gradient Step
Gradient Step
ANC 2:24
ver I52c
222
11.
125
(ouro
DAPO
DXPO
105
280
1000
204
606
100m
Tel
1000
Pia
Xil
Step
Ooynzivtteerh
Math
(our)
Stop
Figure 6: Training and validation accuracy curves for 7B experiments. We conducted 3-4 random and independent experiments for each algorithm and report their mean + std results.
Validation Accuracy
55
50
Negative Data Contribution
45
RFT
DAPO
Positive Data Contribution
NFT (ours)
30
Gradient Step
Figure 7: Average accuracy across 6 benchmarks for 32B experiments. More curves in Appendix D
ure and II present training curves across multiple runs. NFT exhibits convergence speed and final performance on par with DAPO, further supporting its stability.
5.3 BENEFITS OF NEGATIVE DATA
Negative feedback enhances performance and exploration. Table T shows that NFT con- sistently outperforms RFT by a clear margin, highlighting the benefit of incorporating nega- tive feedback during training. Notably, we ob- serve clear divergence in training dynamics between RFT and NFT. Across both 7B and 32B settings, RFT tends reduce entropy over time, whereas NET and RL methods like DAPO encourage increasing entropy (Figure 8 This
Entropy
Entropy
1.00
NFT (ours)
1.0
RFT
DAPO
0.5
0.25
0.0
2000
4000
2000
4000
Gradient Step
Gredieet Step
Figure 8: Entropy curves for 7B and 32B runs.
suggests more exploration YYu et al. 2025), potentially hindering why NFT outperforms RFT.
Negative feedback becomes increasingly important larger models. The performance gap be- tween RFT and NFT widens faster over training in 32B experiments compared with the trend in 7B (Figure II. Similarly, DeepSeck-aae AAl (2025) also notes RL offers greater benefits over SFT in larger models. A potential explanation could be the increasing importance of negative data: Larger models already memorize well enough, so the ability to reflect on mistakes becomes a new bottleneck.
RFT remains strong baseline. Although surpassed by numerous algorithms, RFT still deserves attention due to its extreme simplicity. In 32B settings (Figure [I]. learning from positive data (RFT) contributes to 80% of the total gain achieved by our best-pertorming model, while negative
8

---

Published as a conference paper at ICLR 2026
Wgww
640)
Wiiidedonn Accemmy
0.55
Valldatton Accurscy.
640) =
(4g)
0.50
r= 0
r.10
0.45
2:20
0.2
0.4
9.6
0.3
1.0
MFT
AFT
14
1.0
2.0
4.0
Figure 9: Effect of prompt weighting.
Figure 10: Effect of negative ratio clip value
data only accounts for the remaining 20%. These findings echo recent studies Yue et al. 2025 Xiong et al.] 2025 Liu et al.] 2025a Zhao et al. 2025 Wang et al.] 2025), which suggest RI. primarily amplifies existing capabilities large models rather than fostering new skills. How to exploit negative feedback remains open challenge with heavy potential.
5.4 INGREDIENTS BEHIND NFT'S S EFFECTIVENESS
We discuss two empirical design choices that notably help NFT achieve strong performance. Prioritize harder questions. We find that assigning higher weights to difficult questions with a low answer correctness rate fg can enhance model performance. We achieve this mainly by selecting w(q) in Eq. 10 from 3 choices: (1) ((q) = 1. (2) w(q) -Tq. which aligns NFT with Dr. GRPO in on-policy training (Sec. 4.. (3) (()) = (11 (/l-rr)/ra which aligns NFT with GRPO. Figure 9 visualizes different w w'q). and their effect for NFT and RFT. We find choices (2) and (3) perform similarly, both outperforming constant weighting choice (1).
Avoid overpenalizing mistakes. The clip value of NFT (Eq. 10] sets an upper bound on the penalty weight when the likelihood ratio R2 for negative answers increases. When is small (e.g., near zero), the algorithm empirically assigns high penalties to rising likelihoods of incorrect answers (Figure 10). However, experiments show that overly aggressive penalization with -> 0 degrades overall performance, We thus adopt default setting of e = 1.0.
RELATED WORKS
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the frontier of LLM rea-
soning (DeepSeek-A) 2025 OpenAl 2024 Team et al. 2025 Chen et al., 2025c). Compared with
previous RL practices that rely strong reward models (Wang al. 2023b Yuan et al., 2024
Zhang et al. 2024) to simulate human feedback (Ouyang et al. 2022 Christiano et al., [0117] Cui ground truth verifier for providing reliable binary supervision (Lam- bert et al. 2024 Shao et al. 2024). Moreover, unlike preference-bart based learning algorithms such as
al.] 2025), J, RLVR turns
DPO [Rafailov et al. 2023 Cai al.
2023 Azar et al. 2024 Ethayarajh et al. 2024 Wang al.,
2023a Chen et al. 2024; Hong et al.
2024 Xu et al. 2023). RLVR does not require paired prefer-
ence data, rendering more flexible and memory-etheient, Despite the demonstrated effectiveness
of RL algorithms in verification driven training (Li et al. 2023 Ahmadian et al.) 2024| Hu 2025
Yu et al. 20255] Chu et al., 2025b; Yuan et al.) [025]] Yan al. 2025). recent studies suggest that
supervised learning (SL. may also suffice for achieving verification-driven training in LLMS (Dong
al.
2023). Our method further addresses SL's inability to incorporate negative feedback (Hua
al.
2024). bridging both the theoretical and the performance gap between the two fields.
A key design of NFT involves implicit policy modeling for direct policy optimization. This design, emphasizing direct optimization via implicitly defined models, shares conceptual similarities with some existing approaches. In preference- based training, DPO (Rafailov et al. 2023) introduces an
implicit reward model parameterized by the policy network to allow optimizing policies directly. Recent visual modeling efforts also leverage implicit conditional or residual models parameterized by generation networks avoid guided sampling (Chen al. 2025b
Zheng al. 2025

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
In this work, we introduce Negative-aware Fine-Tuning (NFT), a supervised approach that enables LLMs learn from their own negative generations. In online training, NFT substantially improves upon supervised learning baselines through the additional leverage of negative feedback, achieving performance comparable to leading RL algorithms like GRPO. Notably, we unveiled a theoretical equivalence between NFT and GRPO under strict-on un-pocyn- conditions. These findings bridge the conceptual and practical gap between SL and RL paradigms in verification-die driven training.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models (LLMS)) solely writing assistant for language polishing and improving clarity of presentation. The LLMs were not involved in research ideation, methodolog- ical design, experimental execution, result analysis. All scientific contributions and substantive writing were carried out by the authors.
ACKNOWLEDGMENTS
This project supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan the Ministry of Education of China (No. JYB2025XDXM101), NSF of China Projects (Nos. 62550004, U25B6003, 92370124, 92248303); Beijing Natural Science Foundation 1.24701 I; the High Perfor- mance Computing Center, Tsinghua University. J.Z was also supported by the XPlorer Prize.
We especially thank Lifan Yuan for collaboration. We thank Wei Xiong, Zekun Hao, Yuxuan Tong, Chang Zhou, and Jiashu Xu for the insightful discussion.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in llms. arXiv preprint arXiv:2402. 14740, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. AISTATS, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08077. 2022.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308. 3432, 2013.
Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, and Guannan Zhang. Ulma: Unified language model alignment with demonstration and point-wise human preference. arXiv preprint arXiv:2312.02554, 2023.
Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. Noise contrastive align- ment of language models with explicit rewards. In NeurIPS. 2024.
Huayu Chen, Kai Jiang, Kaiwen Zheng, Jianfei Chen, Hang Su, and Jun Zhu. Visual generation without guidance. In ICML, 2025a.
Huayu Chen, Hang Su, Peize Sun, and Jun Zhu. Toward guidance-free ar visual generation via condition contrastive alignment. ICLR, 2025b.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereasonn-nemotron Advancing math and code reasoning through reinforcement learning. arXiv preprint, 2025c.
Paul F Christiano, Jan Leike, Tom Brown. Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurlPS, 2017.
10

---

Published as a conference paper at ICLR 2026
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-trainn tess-training. arXiv preprint 2r 2025a.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg. A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02544 2025b.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprini arXiv:2502.01456. 2025.
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. TMLR, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXlv:2402.01306, 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao. Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04799 2024.
Chaoqun He. Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0333 038744 2021.
Jiwoo Hong, Noah Lee, and James Thome Orpo: Monolithic preference optimization without reference model. arXiv preprint trXiv:2403.0769/ 2024.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262 2025.
Ermo Hua, Biqing Qi, Kaiyan Zhang, Yue Yu, Ning Ding, Xingtai Lv, Kai Tian, and Bowen Zhou, Intuitive fine-tuning: Towards simplifying alignment into a single process. arXiv preprint arXiv:2405.11870, 2024.
Hynek Kydlíček. Math-verify: Math verification library, 2024, URL huggingfaace/math-erri.
com/
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. TV"" ulu 3: Pushing frontiers open language model post-training. arXiv preprint 2rXiv:2411.15124 2024.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. arXiv preprint arXiv: 0188805000099 2018
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole et al. Solving quantitative reasoning problems with language models. In NeurIPS, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 2024.
11

---

Published as a conference paper at ICLR 2026
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi MhiiQuan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310.
2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXfv:2412.19437. 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang. Chao Du, and Min Lin. There may not be aha moment in r1- r1-zero-like training- pilot study, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
NVIDIA. Cosmol-reasonl: From physical common sense to embodied reasoning. arXiv preprint arXiv:2503. 2025.
OpenAL. Gpt-4 technical report. arXiv preprint 2rXI::2303.08774, 2023.
OpenAl. Openai ol system card. arXiv preprint arXiv:2412.16723 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In NeurlPS, 2023.
John Schulman. Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXh::2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 19256,9 2024.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. 17492, 2023.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NeurIPS, 1999.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Chaoqi Wang. Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Gen- cralizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2300. 16240, 2023a.
Peiyi Wang. Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wa, and Zhifang Sui. Math-shepherd: Verify and reinforce 1lms step-by-step without human annotations. arXiv preprint arXiv:2312.089.. 2023b.
12

---

Published as a conference paper at ICLR 2026
Yiping Wang. Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.2057,, 2025
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji. Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312.11456 2023.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint (tti:::504.113433 2025.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16662. 2023.
Jianhao Yan, Yafu Li. Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off off-policy guidance, 2025. URL https::/erxiv..rgg/ abs/
2504. 14945
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via seff-improvement. arXiv preprint 2rXiv:2409.12122 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source I1m reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 2025.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. In NeurlPS, 2023a.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.0198) 2024.
Yurun Yuan, Fan Chen, Zeyu Jia, Alexander Rakhlin, and Tengyang Xie. Trajectory bellman residual minimization: A simple value-based method for Ilm reasoning, 2025. URL http::ttts:/aatt
exg/abs/25055 15311
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scal- ing relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023b.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity Ilms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mchran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.1 15240, 2024.
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.0P 07912, 2025.
Kaiwen Zheng, Yongxin Chen, Huayu Chen, Guande He, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Direct discriminative optimization: Your likelihood-based visual generative model is secretly a gan discriminator. ICML, 2025.
13

---

Published as a conference paper at ICLR 2026
A PROOF OF THEOREMS
Theorem A.1 (Policy Optimization with Negative Answers). Consider the maximum-likelihodd objective for training the implicit negative policy Tg.
min [- B(q.a)-D- log
Totd (a. q) T (al9)
max B (a
[log To (alq)]
1 1- r.q
Assuming unlimited data and model capacity, the optimal solution for solving Eq. i
Vq,as a: nh (a/g) = (alg)
Proof. The proof is quite straightforward. First, we show that maximum-likeline training leads to the optimal solution TT- (alq) (alq). (a]g))
max Ep(g)= (ale) [log To (a
0
+ max E p())) (ala) [log To (alq) - log (alq)
+ min BninEp(g) DKL (T (a q) T (al4)
Since DRL [ (alg) (a/g)) 2 0. The equality holds if and only Vig Tg = x-. We thus have
Ag, a Tg. (alg) π (a)g).
(11)
Next, we prove ng.
Note that that during training, "g only an implicit policy defined by Th through
T (alg) Tota (a(g) TT (alg) 1- T-
On the other hand, the negative data distribution π has the same relationship with n by Eq.
π (alg) = Told (ala)a T (alq)
1 1- T-7q
We have ensured 0 < rg < 1 during training, combining these observations and Eq. we have
Vq, a: (alg) = - (all))
Proposition A.2 (Algorithm Gradient Comparision). Suppose there are faK positive answers and (1 - KK negative ones for given question q
(a) GRPO Gradient: Consider only binary reward {0, 1} in Eq. 3 GRPO loss gradient satisfies
V (0) = - r R)(g.a) < <1 <1+ (4000)11111_01144444444444411110 V
where A+
fy V I-fa and Ar = = 1- are respectively normalized advantages for answers.
(b) NFT Gradient: Let w(q) ((1 - ff.. NFT loss gradient satisfies
fg Rb a
VOCNET - R5 a)
+ (1 (1-r)Am max
c] ] ()))(q.a)
1- Fq
Proof. (a) GRPO Gradient: We first copy down the GRPO loss definition from Eq.
C CGRPO Σ min Rb a) Ag. c)) 1'
q.a.t
14

---

Published as a conference paper at ICLR 2026
The normalized advantage can be estimated as
Aqaa == [r(q, a) mean 1: } / stafrlik)
where ( Fig K 1 +1 x 0] = fq
and
stdfrlik] = V liok * (1 fgg) + (1 - -fg)g (0 - f -f)))
1
When a is positive answer, r(g, a) = 1. We have A = -
1-1 > 0.
CORPO (0)
Σ
min R a aaa,tt,, + c) Aq.at
q.atit it
V VeCor (0)
(0) - Σ
A I [Rbbgg.at) < <1+e].
(12)
q
When a is negative answer, r(q.a) = 0. We have Ag = =5=--1 I-fa
Py <0.
[ CCRPO (0)
Σ
max [Rg a 0,, 1 e'} A Aq.a
q.a
V COROR (0) - Σ
Ag I [ > -6
(13)
g.aT,
Combining Eq. 12 and Eq. 13
we have
VOLO GRRPOOOOO (0) = Σ (fr
I
<1+ =
1 - -c] Vo
(a) NFT Gradient: We copy down the NFT loss definition from Eq. 10
fg Rt q, a)
CNFT (0) = Σ w(q)
rlog Ro a) + (1 - r) log max v(
,e)
f
q,a,t
When a is positive answer, r(q, a) = 1.
CNT (0)
Σ
w(q) log Rg a)
q .
Σ
fq log Rt (q.a)
fq
qaa!! .t
Σ a.at
A
log (gg
V
Σ
A
VoRa(q.at)
(14)
Rg a+)))a+)))
q.a i
When a is negative answer, r(g. aa=a)== =0
1 fg R2 (q, w(q) log max v(
CNET( = Σ
q.a
i log
1 - ig Rs (g,
max v(
Rg
fg
q.a
15

---

Published as a conference paper at ICLR 2026
V VgLD
NET
Σ
fg
max
fg Rg
( 9,
_e_ -fg Volls(q.a
lqq
1- fq
1- fg
q,a71
fg
Ry R
Σ
_ "ooli(g,a))
max
1 lq
1- iq
q,07
fg Rg (gqq.a")
= Σ
Ag
max
,e) VoRi
(15)
1 - fa
9,0 .
Combining Eq. 14 and Eq.
15
we have
fg Rg (g.(gg.0
VOLD
CFFT (0) = -- Σ A
ag +(1-r)Ag
max
Voll a).
Fq
Remark A.3 (Dr. GRPO). The main practical difference between Dr. GRPO (Liu et al., 2025b and GRPO is that Dr. GRPO removes the std normalization term when estimating group norumaoora advantages. Following Proposition 4.1 we simply need to set ((q) = 1 - fg instead of w(q) =
11- to align with the Dr. GRPO loss function. tq
Proposition A.4 (On-policy Gradient Equivalence). Following the set up of Proposition 4.1 and let < 1, GRPO and NFT loss gradient equivalent in on on-policy training:
Rg (g a) =1
VOLNET(O)) (0) V V (0)
Proof. The proof is simple. When on-policy, R. a) = 1. Regarding positive answers at, GRPO gradient (Eq.
12 and NFT gradient (Eq. 14 both become D+
(O) V
C
CNFT (0) A V Vo a+).
Regarding negative answers a a, GRPO gradient (Eq.
13 and NET gradient (Eq. 15 both become
V VoLp C VoLD ( Ag Vo
B DISCUSSION FOR CONTINUOUS REWARDS
In this section, we explain how NFTs can be extended to continuous reward settings. First, we define r := p(o g a) E [0, 1], where means the answer is an optimal answer. Then we define positive and negative distributions.
=aa(alg)((==1qqaa) ma
=+(aaq) o=llg,0=1) E. Maul (al(a)))(o=1lqq00
Tola (
<aag)) = (alg, ΣA Tow Tow (1)(0))) )aa((ag) r(q, a)),
T r + [1 rgl (alg) Taaa
Finally we copy training objective Eq. below.
x (a|g)
CNFT ((a.g))~(())) (8) (0)= log T (alg) Takal (a
1 rg mail ②)
+ (1-
loog
(16)
Tg
Theorem B.1 (NFT for continuous rewards). Consider the training objective of Eq. 16 with arbitrary continuous rewards r(g,a) E [0, Suppose we can accurately estimate Tq Emayaa a). Assuming unlimited data and model capacity, the optimal solution satisfies
Vq, a: T - (alg)
16

---

Published as a conference paper at ICLR 2026
Proof.
[ Tota(a
70 (alg)
Vol Vol(a.r)>DD()) Ve r(q. a) l To (alq) + (1 r(q. a))
log
1 1 rq
r(q.a)
+ - (1-r(qq.0) r(q.
1q
(o
Te (alg)
Tosa (ala)) - mT ( (a)
Tg (alq) r(q. To
VoTo(alq))
Tg (a q) (alg)) - T (alg))
When TT (alg) r(g, a) To (a)a) = 0, that is
r(q. a T)
~ (alq) =
= =T (alg)
rq
We have Vo VNLNET (a.ar)p> 0 constantly holds.
C EXPERIMENT DETAILS
General training setup. All algorithms implemented based the official DAPO codebase within the VeRL framework. We use a learning rate of 1c-6 with a linear warm-up schedule across all experiments. At each rollout step, we generate 16 answers for each of 512 sampled questions, then split the data into 16 mini-batches and train the policy network for 16 gradient steps. Models are trained for 320 rollout steps, totaling over 5,000 gradient steps. Unless otherwise specified, we follow DAPO's ' default design choices, including dynamic data sampling, token-level loss normal- ization, and no KL regularization. For training, we restrict context lengths to 4K and use 64 NVIDIA H100 GPUs. For 32B training, we restrict context lengths to 32K (DAPO) or 16K (others), and use 126-256 NVIDIA H100 GPUs.
DAPO. is variant of GRPO that has achieved state-of-the-art AIME performance 32B models. Our NFT implementation is adapted from the official DAPO codebase, based on the VeRL frame- work (Sheng et al., 2024). NFT inherits most of DAPO's hyperparameters and design choices, in- cluding dynamic data sampling, token-level loss normalization, and no KL regularization. We adopt faithful implementation of the official DAPO codebase, keeping all hyperparameters untouched.
NFT. Compared with DAPO, NET modifies the context length to 16K for 32B training. We find this does not significantly affect performance, but noticeably reduces data collection time. Another dif- ference is that we remove the overlong reward shaping technique of DAPO to ensure a binary reward outcome. In our setting, truncated answers are treated as negative, which sufficiently discourages overlong answers. The negative ratio clipping parameter is set to 1.0, and the prompt weight is defined as (()) = 1 Tg.
RFT is simple but effective SL baseline that only fine-tunes LLMs on positive answers and throws away negative data. In our implementation, the main difference between RFT and NFT is that RFT zeros out negative data loss and keeps constant prompt weight w(q) 1 during training. For RFT, we zero out negative data loss in NFT implementation and keep constant prompt weight w(g) during training.
GRPO does not adopt the dynamic sampling technique proposed by DAPO. Rather, it simply lever- ages all data for training, even though the gradient for all- positisieive or all-negative questions should be zeroed out automatically by the algorithm itself (Yu et al. 2025). Other DAPO-related techniques kept, such setting positive clipping parameter to higher value e4 0.28. Since GRPO requires less time for sampling data, we train GRPO models for 580+ rollout steps instead of 320 steps, which roughly takes the same training time compared with DAPO experiments.
Dr. GRPO is modified from our GRPO implementation. The only difference is the removal of the std normalization when computing group-normalizer advantages.
Iterative DPO. Comparing DPO with other RL algorithms in head-to-head fashion is difficult because DPO requires paired data calculate the training objective, while sample 16 unpaired
17

---

Published as a conference paper at ICLR 2026
answers for each question. To solve this problem, take the implementation of InfoNCA (Chen et al. 2024,, a variation of the DPO algorithm that can handle 2 responses per question:
K
(k) (())
log Mala.a")
ClafoNCA p (A =
K
Σ eBfll..aa)
k=1
InfoNCA is guaranteed to become DPO algorithm in K = 2 settings. We ablate 3 E (0.1, 0.02) and report the best averaged validation result. We find InfoNCA training to be unstable and add an SFT regularization term to the original loss function for stabilizing the training process.
Validation details. Validation is performed with a top-p value of 0.7. The temperature is set to 1.0 for 7B models and 0.6 for 32B models, with context lengths matching the training configuration. We use math-ver Ey (Kydliček] 2024) as the verifier during training validation, and s 1mRp (Zen et al. 2025) for final evaluation. The DAPO-17: benchmark consists solely of training questions whose ground truth answers are integers and includes both prefix and lastfix for each question. Accordingly, for validation on AIME and AMC questions, we adapt the prompt format to match the training pattern. For other benchmarks with non-integer answers, question prompts remain unmodified.
D ADDITIONAL EXPERIMENT RESULTS
Validation AccuuatioonAcuuracyy
Training Accuracy
80
NFT (ours)
NFT (ours)
RFT
RFT
DAPO
20
DAPO
Gradient Step
Gradient Step
Joumi
KF (ourg
Stop
Gradkem: Soaz
Gracken
KT Buss)
ReI Hour)
Grudient Seep
Sradle:: Chpp
Gradlee
Figure 11: Training and validation accuracy curves for 32B experiments.
18


Reviews:
[{"review_id": "foRnN2PlPh", "reviewer": "Reviewer_wqPT", "summary": "This paper explores whether supervised fine-tuning approaches can be competetive with reinforcement learning. In particular, the authors identify a means of using negative samples to fine-tune an \"implicit negative policy\". Their approach successfully leverages both positive and negative samples, which can potentially improve over conventional supervised fine-tuning approaches that just use positive samples. Their results show some improvements over baselines, most notably in AMC23.", "questions": "- line 067: throughout the paper, you refer to an \"implicit\" negative policy. It would help to clarify early what it means for the policy to be implicit, and how this actually affects training the LLM\n- Equation 7: Is the key observation being made in equation 7 is essentially the marginalization of $r$?\n- Section 3.2: What is the advantage of this implicit formulation relative to training on an LLM that explicitly conditions on the reward? That is, training the positive policy as pi(a|q, r=1) and the negativev policy as pi(a|q, r=0).\n- Theorem 3.1: This statement is essentially stating that the critical points are the same, which makes sense. But it is not quite clear what the advantage of this this particular formulation.\n- Proposition 4.1: The similarity seems to be driven by the free parameter governing the prompt weighting, $\\omega(q)$. Moreover, the differences pointed out in the gradients seem to originate in the practical modifications, rather than the core algorithm. As such, I am not sure whether these takeaways are related to negative-aware fine tuning in particular.\n- Table 1: Results are a little inconclusive, where in some cases NFT offers a notable improvement (AMC23), but in others losing to RFT or RL based approaches\n- Section 5.3 and Figure 8: I like the fact that the authors investigate the benefits of negative data, but the analysis here seems a little adhoc. For example, couldn't higher entropy be achieved by entropy regularization, which would not necessarily be helpful given that it doesnt use any negative data? The higher entropy could be explained by there being more data fit, which is harder than just fitting positive samples.\n- Section 3.3 (practical modifications) and Section 5.4: I also appreciate that the authors investigate the importance of the practical modifications to the base algorithm. It is puzzling that prompt weighting improves accuracy for NFT well beyond RFT Would this seem to suggest that the modifications are more impactful than the negative data?\n\n### Minor Comments\n- Some ancillary comments do not seem to make any connection to the contribution, such as: \"Later studies (Liu et al., 2025b) suggest removing the std term from Eq. 4\"\n- Notation is sometimes abused and informal, such as line 150: $\\mathcal{D} \\sim \\pi$\n- It is never explicitly defined that \"positive answers\" are correct answers.\n- line 098: minor point but the larger issue in taking a gradient of the RL objective is actually that the parameter $\\theta$ is involved in the sampling distribution rather than just in the log probability (as in supervised learning)\n- Section 3: the paper would benefit from being more clear about what policy is being under consideration. For example, you refer to $\\pi$ and $\\pi_old$", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors posit a clear research question to explore in their paper: the current approach to fine-tuning may be competitive for self-improvement compared to current approaches to reinforcement learning.\n- Motivated by this research question, the authors identify a clear weakness in current apporaches to fine-tuning: it is not possible to use the full dataset when fine-tuning only on positive samples.", "weaknesses": "- While many aspects of the paper are clear, some of the benfits of their negative-aware fine tuning method are not obvious. For example, they choose to use a specific implicit policy approach, but dont compare against an explicit method that conditions on the reward. Also, in their ablations, their practical modifications seem to have a much larger effect on performance than solely using the negative data.\n- The motivating research question does not feature prominently in the experiments, besides comparisons to reinforcement learning. That is, while supervised finetuning approaches appear competetive with reinforcement learning for performance improvement, this paper does not clearly demonstrate that this gap is closed by self-improvement from mistakes in negative data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GspnyFiT0r", "reviewer": "Reviewer_YUof", "summary": "The paper introduces Negative-aware Fine-Tuning (NFT), a ``supervised'' method for large language model training with binary verifier feedback. Unlike rejection-based fine-tuning (RFT), which discards incorrect responses, NFT constructs an implicit negative policy that allows the model to learn from both correct and incorrect generations using the same parameterized model. The method defines a token-level objective combining positive and negative log-likelihood terms and incorporates clipping and straight-through gradients for stability. Conceptually, NFT connects supervised and reinforcement learning: under on-policy conditions, its gradients align with those of GRPO, while maintaining a maximum-likelihood foundation. Experiments on 7B and 32B models for math reasoning tasks show that NFT surpasses supervised baselines and achieves performance comparable to leading RL algorithms.", "questions": "- What confuses me is the following: what would happen if we simply optimize ($\\beta = \\pi_\\text{old}$ due to rendering issue)\n\n    $$\\mathcal{L}(\\theta) = \\mathbb{E}_{x \\sim \\mathcal{D}, y \\sim \\beta} \\left[ r(x, y) \\left[ -\\log\\frac{\\pi^\\theta(y \\mid x)}{\\beta(y\\mid x)} \\right]  + (1- r(x, y) ) \\left[ \\log\\frac{\\pi^\\theta(y \\mid x)}{\\beta(y\\mid x)} \\right] \\right] $$\n\n    This contrastive objective simply reduces the likelihood of negative samples while increasing that of positive ones.\n\n    It would be helpful if the paper could clarify why Eq. (9) provides a more principled alternative to this naive approach, and in particular why the re-parameterization introduced in Eq. (7) is essential.\n\n    An empirical comparison against this straightforward baseline, including the same practical tricks described in Section 3.3, would also make the claimed advantages of NFT more convincing.\n\n\n#### minor comments: writing can be made more crisp\n\n\n- Eq. (9): The notation $(a, q, r)\\sim \\mathcal{D}$ reads as if the dataset is static. It would help to clarify that $\\mathcal{D}$ actually consists of online samples generated by $\\pi_\\text{old}$, rather than a fixed dataset.\n\n- Algorithm 1: My intuition is that the training process should involve \\emph{updating} $\\pi_{\\text{old}}$, yet this does not appear to be reflected in the pseudocode. If my intuition is correct, clarifying whether (and how) $\\pi_{\\text{old}}$ is updated across iterations would improve readability.\n\n- The phrase ``a pretrained LLM $\\pi_{\\text{old}}$'' suggests that $\\pi_{\\text{old}}$ remains \\emph{frozen}, tied to the initial pretrained model. It would be clearer to specify whether $\\pi_{\\text{old}}$ is only the initialization or if it is also updated online during training.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Relevant and timely topic.\n- Comprehensive evaluation across a broad range of reasoning datasets.", "weaknesses": "- The selling point of NFT is that it is an online SFT method. However, being ``online SFT'' does not immediately provide conceptual advantages. If one considers RL as a paradigm of learning under interaction with an environment, then online SFT is simply a special case of RL. The advantages therefore need to be shown practically, for example, through computational or empirical improvements, rather than assumed conceptually. The paper state:\n    > Memory Efficiency. NFT is memory-efficient. In practice, we keep only a single model copy in memory. The old policy likelihood πold(a|q) can be pre-computed during data generation.\n\n    However, this should also applies to GRPO (if KL-free) and DAPO. \n\n- Empirically, NFT does not show notable differences from DAPO, suggesting that the distinction is modest in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "muNTF4LIAn", "reviewer": "Reviewer_RLnb", "summary": "The paper introduces Negative-aware Fine-Tuning (NFT), a supervised learning framework that allows large language models (LLMs) to learn from both correct and incorrect generations without RL. Unlike traditional supervised fine-tuning (or rejection finetuning), NFT constructs an implicit negative policy parameterized by the same model as the positive policy, enabling direct policy optimization on both positive and negative examples. The authors show theoretically that NFT is equivalent to RL under strict on-policy conditions, thereby bridging the conceptual and empirical gap between supervised and RLVR. Experiments on math reasoning tasks with Qwen-2.5 models (7B and 32B) demonstrate that NFT matches or surpasses leading RL methods like GRPO and DAPO while being simpler and more memory-efficient.", "questions": "1. Can NFT generalize beyond mathematical reasoning to less verifiable domains such as commonsense or code generation tasks?\n\n2. Could the authors provide quantitative comparisons of training efficiency (e.g., FLOPs, GPU hours, or convergence speed) against RL-based baselines such as DAPO or GRPO?\n\n3. Could the authors provide generation examples or failure cases?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well written and clearly structured, making it easy to follow.\n\n2. Theoretical contribution — the equivalence between NFT and GRPO is well derived and provides valuable conceptual clarity between reinforcement learning and supervised fine-tuning.\n\n3. The method is simple, effective, and memory-efficient, avoiding the complexity of traditional RL pipelines.\n\n4. Unlike DPO, NFT does not require explicit positive–negative pairs, which improves data efficiency and implementation flexibility.\n\n5.  Empirical results demonstrate strong performance and good test-time scalability, with well-conducted analysis and ablation studies that support the claims.", "weaknesses": "1. The evaluation is limited to mathematical reasoning, with no experiments on logical or commonsense reasoning tasks, limiting evidence of generalization.\n\n2. While Table 2 shows strong performance compared to other RL-based methods, the paper lacks quantitative analysis of efficiency (e.g., training FLOPs, wall-clock time, or GPU hours). Since NFT claims to be simpler and more efficient than RL-based training, this measurement is important to support the claim.\n\n3. Additional discussion on failure cases would improve interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PWHedVHT9F", "reviewer": "Reviewer_Nkge", "summary": "This paper proposes Negative-aware Fine-Tuning (NFT), a new supervised learning algorithm designed to bridge the conceptual and empirical gap between SL and RL in math reasoning.\n\nUnlike traditional SL methods that discard incorrect generations, NFT explicitly models negative data by introducing an implicit negative policy parameterized by the same LLM. The paper shows that NFT can optimize both positive and negative samples via a single model, with negligible memory cost. Empirically, experiments on Qwen2.5-Math-7B and 32B show that NFT outperforms or matches leading RL baselines like GRPO and DAPO on math reasoning benchmarks (AIME24/25, AMC23, MATH500, OlympiadBench, Minerva).", "questions": "Please see the weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The theoretical analysis is interesting to bridge the gap between GRPO and NFT.\n\n2. The empirical observations are interesting. The analysis of entropy trends and scaling effects (negative feedback being more important for larger models) adds valuable interpretive depth.\n\n3. The writing is good and easy to follow.", "weaknesses": "1. It is well known that incorporating negative examples can enhance model performance, both in offline and online settings. Therefore, the observed performance improvement from training with negative data is not particularly surprising.\n\n2. I am not sure how the authors define SFT, as the proposed method seems more consistent with an RL training setup. I am not fully convinced that NFT should be categorized as a purely supervised fine-tuning (SFT) algorithm. It would be great if the authors can further elaborate on this.\n\n3. From Figure 1, it appears that the main difference between NFT and GRPO lies in the use of binary labels, which essentially changes how the advantage of responses is computed. \n\n4. It seems that NFT does not outperform DAPO (even slightly worse). What is the key advantage of NFT compared to GRPO-like algorithms such as DAPO?  What can drive people to use NFT rather than DAPO?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)