Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
Authors: Lu Ma
Hao Liang
Meiyi Qiang
Lexiang Tang
Xiaochen Ma
Zhen Hao Wong
Junbo Niu
Chengyu Shen
Runming He
Yanhao Li
Wentao Zhang
Bin CUI
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models; Reasoning; Reinforcement Learning; Supervised Fine-Tuning
Primary area: foundation or frontier models, including LLMs
TL;DR: ReLIFT, a method combining reinforcement learning and supervised fine-tuning, enhances large language model reasoning by addressing limitations of RL through interleaved training, improving performance across benchmarks with minimal data.

Abstract: Recent advances in large language model (LLM) reasoning have shown that reasoning ability can emerge through reinforcement learning (RL). However, despite these successes, RL in its current form remains insufficient to induce capabilities that exceed the limitations of the base model, as it is primarily optimized based on existing knowledge of the model. To address this limitation, we employ supervised fine-tuning (SFT) to learn what RL cannot, which enables the incorporation of new knowledge and reasoning patterns by leveraging high-quality demonstration data. We analyze the training dynamics of RL and SFT for LLM reasoning and find that RL excels at improving performance on questions within the model's original capabilities, while SFT is more effective at enabling progress on questions beyond the current scope of the model. Motivated by the complementary strengths of RL and SFT, we introduce \textbf{ReLIFT} (\textbf{Re}inforcement \textbf{L}earning \textbf{I}nterleaved with Online \textbf{F}ine-\textbf{T}uning), a novel training strategy. ReLIFT employs RL for general training, but interleaves it with targeted SFT on challenging questions for which high-quality solutions are collected online. By alternating between RL and SFT, ReLIFT addresses model weaknesses as they emerge. Empirically, ReLIFT outperforms previous RLVR methods by an average of +6.7 points across a suite of six benchmarks (five math reasoning and one out-of-distribution). More importantly, ReLIFT surpasses baselines such as individual RL, individual SFT, and various hybrid approaches while reducing the required training time. These results provide compelling evidence that ReLIFT is a powerful and resource-efficient paradigm for developing capable reasoning models. The code is available at \href{https://github.com/TheRoadQaQ/ReLIFT}{here}.

Published as a conference paper at ICLR 2026
LEARNING WHAT REINFORCEMENT LEARNING CAN'T: INTERLEAVED ONLINE FINE-TUNING FOR
HARDEST QUESTIONS
Lu Ma', Hao Liang Meiyi Qiang', Lexiang Tang', Xiaochen Ma', Zhen Hao Wong' Junbo Niu', Chengyu Shen', Runming He', Yanhao Li Bin Cui Wentao Zhang Peking University. "Zhongguancun Academy Beiijng Key Laboratory of Data Intelligence and Security (Peking University) 'Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems
ABSTRACT
Recent advances in large language model (LLM) reasoning have shown that rea- soning ability can emerge through reinforcement learning (RI.). However, despite these successes, RI. in its current form remains insufficient to induce capabilities that exceed the limitations of the base model, as it is primarily optimized based on existing knowledge of the model. To address this limitation, we employ super- vised fine-tuning (SFT) to learn what RL cannot, which enables the incorporation of new knowledge and reasoning patterns by leveraging high-quality demonstra- tion data. We analyze the training dynamics of RL and SFT for LLM reasoning and find that RL excels at improving performance on questions within the model's s original capabilities, while SFT is more effective at enabling progress questions beyond the current scope of the model. Motivated by the complementary strengths of RL and SFT, we introduce ReLIFT (Reinforcement Learning Interleaved with Online Fine-Tuning) a novel training strategy. ReLIFT employs RL for general training, but interleaves it with targeted SFT on challenging questions for which high-quality solutions are collected online. By alternating between RL and SFT, ReLIFT addresses model weaknesses as they emerge. Empirically, ReLIFT out- performs previous RLVR methods by average of +6.7 points across suite six benchmarks (five math reasoning and one out-of- -ifdistriittion). More im- portantly, ReLIFT surpasses baselines such as individual RL, individual SFT, and various hybrid approaches while reducing the required training time. These results provide compelling evidence that ReLIFT is powerful and resource-efficient paradigm for developing capable reasoning models. The code is available at here
INTRODUCTION
Recent advancements in large language model reasoning, demonstrated by OpenAl o-series jtt al. (2024). DeepSeek-RI Guo et al. (2025), Gemini 2.5 [omanici et al. (2025). and Kimi k- series Team al. (025b)) highlight the potential to enhance reasoning capabilities by increas- ing test-time computational resources. These models typically generate long Chains-of-Thoure (CoT et al. 2 responses, exhibiting sophisticated behaviors such reflection Kumar al. (024) and planning Ke et al. (2025). The primary factor driving this progress is large-scale Rein- forcement Learning with Veriiialee Rewards (RLVR), which does not depend on traditional tech- niques like Monte Carlo Tree Search Xie et al.] (2024); Li et al. (2025) or Process Reward Mod- els /Stlur et al. (2024); Zhang et al. (2025). In RLVR, rewards are assigned based whether the model's 's output matches ground-truth solution in mathematics or passes unit tests in code, thereby enabling scalability without the need for demonstration data. Implemented with policy optimization algorithms like Proximal Policy Optimization (PPO) SSchulman et al. (2017) and Group Relative Policy Optimization (GRPO) [Sha al. (2024), the simplicity and directness of large-scale RIVE have been pivotal in incentivizing the reasoning capabilities seen in today's 's most advanced LLMs.
"Corresponding authors: Bin Cui and Wentao Zhang.

---

Published as a conference paper at ICLR 2026
Despite its empirical successes, RLVR is insufficient to foster capabilities that transcend base model's s inherent limitations Yue et al. (2025); Cheng et al. (025): Zhao et al. (002)). Research suggests that RLVR primarily reinforces existing behaviors rather than instilling novel reasoning abilities. For instance, Yue et al. (025) argue that RLVR fails to equip L.MSs with new reasoning skills, noting that RIVR-trrined models underperform base models pass @K higher k values, indicating limited expansion of reasoning capabilities, Similarly, Zhao et al. (2025) show that RL post-training amplifies certain pre-training behaviors while suppressing others. Furthermore, CCheng et al. 2225)) find that RL stifles exploration, causing models to converge on narrow behavioral patterns and plateau on complex reasoning tasks. This limitation is rooted in the on on-policy nature of RL, which learns from the model's own generated responses via iterative rollouts and feedback. Consequently, RL primarily enhances performance by biasing the model toward reasoning paths already knows are likely to yield rewards, optimizing existing knowledge rather than facilitating the acquisition of new information.
In contrast, supervised fine-tuning (SFT) offers complementary approach to enhancing the reason- ing capabilities of LLMs. By leveraging high-quality demonstration data, SFT facilitates the incor- poration of new knowledge and reasoning patterns Mecklenburg al. (2024); Yue et al. (025 Guo et al. (2025), and has been shown to achieve superior reasoning performance compared to RL. par- ticularly in small-scale LLMs Guo al. (2025). Nevertheless, the effectiveness of SFT relies highly on the availability of substantial amounts of high-quality demonstration data. Furthermore, SFT-
trained models often exhibit limited generalization
out-of-distribution (OOD) scenarios, while RL-based approaches tend to demonstrate greater robustness Chu et al. (2025); Chen et al. (2025)
These contrasting characteristies of RL and SFT highlight promising research direction:
How can RL and SFT be effectively combined to improve LLM reasoning and OOD generalization, reduce dependence on expensive demonstration data, and achieve capabilities beyond current cognitive constraints?
To achieve this goal, we analyze the training dynamics of RL and SFT by examining how the ac- curacy of questions changes throughout the training process. We evaluate the model's accuracy on questions at various checkpoints, and classify them into four levels of difficulty. Our findings indi- cate that RL more effective for questions of lower difficulty, whereas SFT proves more beneficial for the most challenging ones. Furthermore, for simpler questions, SFT can degrade the model's existing performance and the response length increases. In contrast, for challenging questions, the improvement provided by RL is less pronounced compared to SFT. Overall, RL efficiently trains the model to generate correct answers for questions can already solve, while SFT is crucial for enabling the model to address questions that surpass its current capabilities.
Based on these analyses, we propose Reinforcement Learning Interleaved with Online Fine-Tuning (ReLIFT)). As illustrated in Figure 2 during RI. training, we collect highly challenging examples according to the accuracy observed during the rollout. When such challenging question is identi- fied, we obtain a high-quality CoT solution and subsequently filter out any incorrect answers. After constructing these examples, we add them to an SFT buffer. Once the number challenging ques- tions in the SFT buffer is sufficient, we perform SET these questions for one step.
We conduct comprehensive experiments on five challenging math reasoning benchmarks and one OOD benchmark. Our proposed ReLIFT method achieves new oftte-of-the-ar accuracy of 52.6% with Qwen2.5-Math-77
which significantly outperforms previous RLVR baselines. Notably, Re- LIFT demonstrates clear superiority over methods including pure SFT, pure RL, and combined RI. and SFT approaches, such as RL with SFT loss, SFT followed by RL, and LUFFY Yan et al.) (2025). ReLIFT requires minimal detailed demonstration data and GPU training hours. Additionally, Re- LIFT generates significantly more concise solutions, which improves both performance and effi- ciency. We further validate the generality of ReLIFT by extending to smaller and weaker base models, consistently observing superior results.
The main contributions of this work are follows:
We present systematic analysis of the training dynamics of RL and SFT in reasoning, empirically demonstrating their complementary roles: RL refines existing skills on solvable problems, while SFT is essential for acquiring new knowledge for difficult ones.
2

---

Published as a conference paper at ICLR 2026
1.0
10"
0.6
10
nig Steps
Pairing Steps
Aireade
Manall
(a) Average Accuracy
(b) Average Response Length
(c) Initial and Final Accuracy
Figure 1: Accuracy and response length changes for Easy, Medium, Hard, and Hardest questions during RL and SFT training. (a) Average accuracy change for each difficulty category. (b) Average response length change for each difficulty category. (c) Number of questions transitioning between different initial and final accuracy categories in RI. and SFT, respectively. The x-axis represents the initial difficulty category, and the y-axis represents the final difficulty category.
We propose ReLIFT, a novel framework that interleaves RL with online fine-tuning, Re- LIFT dynamically identifies challenging examples during RL for targeted fine-tuning to address emerging model weaknesses.
Extensive experiments show ReLIFT achieves state-of ttate-of-the-arr performance on mathemati- cal reasoning and OOD benchmarks, outperforming pure SFT. pure RL, and existing hybrid methods with significantly less demonstration data and computational overhead.
REINFORCEMENT LEARNING INTERLEAVED WITH ONLINE FINE-TUNING
2.1 RL vs. SFT: TRAINING DYNAMICS ACROSS QUESTION DIFFICULTY LEVELS
We begin our study by examining how the model's accuracy evolves questions of varying diffi culty during both SFT and RL training. Specifically, we independently train the Swwen..5-Math-- model using both SFT and RL a subset of the ppen-Rl-math-2201 dataset Face 2025).. Both SFT and RL are conducted for 120 steps, with model checkpoints saved every 30 steps. Detailed ex- perimental settings are provided in Appendix B At each checkpoint, we evaluate model performance on a validation set comprising 1,000 questions. To facilitate a more detailed analysis, we categorize the validation questions into four distinet difficulty levels: Easy, Mediaum, Hard, and Hurdest. To de- prompt the Qwen2.5-Math-77. model eight times and calculate the average accuracy for that question, denoted acc(g) Following the approach in works like Wen al. (2025); Bae et al. (2025); Yu et al. (2025), and to ensure sufficient number of questions in each category, we select these specific thresholds to enable a robust analysis of the training dynamics of both RL and SFT. The criteria for assigning difficulty levels follows:
termine the difficulty level of each question
Easy: 6 to 8 correct answers of (ace(g) 2 0.75).
Medium: 3 to 5 correct answers of 8 (0.375 < aco(g) < 0.625).
Hard: correct answers out of 8 (0.125 acc(1) < 0.25).
Hardest: 0 correct answers of 8 (acc(g) 0), representing items that are beyond the model's current capabilities.
We analyze the average accuracy of LLM trained with RL and SFT on four difficulty levels of ques- tions: Easy, Medium, Hard, and Hardest. As shown in Figure Ta RL outperforms SFT on Easy and Medium questions. Conversely, SFT achieves better results than RL on the Hardest questions. No- tably, after SFT, the LLM fails to solve certain questions that the initial model previously answered correctly. In contrast, RL consistently improves accuracy across all difficulty levels and does not exhibit this issue. We also examine changes in the average response length during RL and SET training. shown in Figure 1b, SFT produces longer responses across all question difficulties. aim- ing to mimic the target long response. contrast, RL begins with shorter responses, and for Medium and Hard questions, the response length increases time, indicating more exploration. For the
3

---

Published as a conference paper at ICLR 2026
N
N
< M
N2 M
< M
RL
RL
RL
RL
RL
RL
FT
RL
GRPO
O2 On
x xi
A, A2 An
processing
policy model
0 O2
verifier
x x
group computation
Ag A2
Inanter N3
4n
01 02 On
A Az An
(9uia) (15.8) s) (4a.
update
filter
Figure 2: Overview of the ReLIFT Training Framework. The model is mainly trained with RL. When it encounters particularly hard questions, high-quality solutions are collected or generated, then stored in a buffer. Once enough hard examples are gathered, fine-tuning (FT) step is performed using these examples. This process adaptively alternates between RL and FT to help the model learn from its mistakes and improve reasoning ability. In addition, N denotes the number of hardest (g. s) pairs in the buffer, while M represents predefined threshold, typically set to the batch size for the FT.
Hardest questions, the response length remains stable, which suggests the model stops exploring further. Ideally, responses to difficult questions should be longer, and responses to simple questions should be shorter. However, neither SFT nor RL fully achieves this objective.
We further analyze the evolution of accuracy during training by tracking both the initial and final accuracy for each question, as illustrated in Figure [cc Our findings reveal substantial differences in how RL and SFT influence the model's performance across questions of varying difficulty, For Easy and Medium questions, SFT results in decreased accuracy for some items, whereas RL generally preserves or enhances the LLM's original ability to solve these questions. However, this pattern reverses for Hardest questions: SFT enables the model to learn and solve greater number of these challenging items, as evidenced by the increased transition of Hardest questions into other cate- gories. In contrast, RL does not yield comparable improvements for Hardest questions, highlighting a limitation in its ability to support learning beyond its current capabilities.
These findings demonstrate that RL and SFT have complementary strengths. RL excels at refining and preserving the model's existing knowledge, making it suitable for questions within its compe- tence. However, SFT is superior for acquiring new knowledge to tackle problems that lie beyond the model's 's initial capabilities. This suggests that accuracy does not improve uniformly and that hybrid training strategy could be optimal. By selectively interleaving SFT steps for the Hardest questions, it may be possible to enhance overall model performance and develop more scalable and effective RL for LLMs.
2.2 REINFORCEMENT LEARNING INTERLEAVED WITH ONLINE
Overview. Motivated by the complementary strengths of RL and SFT in addressing questions of varying difficulty, we propose novel training approach termed Reinforcement Learning Interleaved with Online Fine- Fine-Tuning Tuning (ReLIFT). ReLIFT dynamically integrates RL and fine-tuning (FT) based on the characteristics of the training data and the current capabilities of the model, as illustrated in Figure 2 During RI. training, we identify hardest examples based on rollout accuracy. For each of these hardest questions, we obtain high-quality CoT solutions either by collecting them in advance before training by consulting stronger model human annotators. These curated examples are added to a fine-tuning buffer. When the buffer accumulates enough questions to form a training batch, we perform fine-tuning step using these examples. The underlying intuition is that questions difficult for the model to learn through RL alone may benefit from targeted fine-tuning. Below, we provide detailed description of our method.
Reinforment Learning while Collecting Hard Questions. Owing to the success of Deepseck-Re GRPO has become the de facto approach for RLVR training. GRPO leverages the reward scores of
4

---

Published as a conference paper at ICLR 2026
N sampled solutions from query to estimate the advantage, thereby eliminating the need for an additional value model. Formally, we denote the policy model before and after the update as Te and тg. Given question 9. set of solutions o; generated by aaw. and the reward function R(-), the GRPO objective is defined as follows:
N od
LGRPO(0)
min (r.,(0)),, clip (r..(();11111111 A,),
(I)
ΣN
l0:
i=1 1=1
where (0) ne (os,e 9. 0x,(c) Te.a (Oi 0r,=1))
A; (o,,) - mean ((.(o.) o, new (o), = 1, ,.,,,.,,))) (o), 1,2 NN)
std ({R (o,)
During the rollout phase of GRPO, for each question q, generate set of solutions o,, which are evaluated by the reward function to obtain the accuracy acc(g). We identify particularly challenging questions as those for which acc(g) O. For these particularly hard questions, solicit high- quality CoT solutions 8 from a stronger reasoning model (e.g.. DeepSeek-R1) or human experts, filter out incorrect Co'Ts obtain reliable QA pairs (g. 8). and add them buffer Buffernardest- The data collected from Buffernardest is then added to primary fine- one-tuning buffer, BufferpT- which serves the data for subsequent fine-tuning step. This process is performed online during training. Collecting and training can be done together, and can be formalized as:
Buffer hardest
{ (g, acc(q) = 0, s (q), extract(s) = =a},
(2)
BufferPr
Bufferrr
U Bufferaaddest.
where (g) denotes the CoT response obtained either in advance or online from an external model human annotator, extract(s) indicates the final answer extracted from 8, and is the groundtruth answer for q. We only keep (g. s) pairs where extract(s) = a.
Interleaved Online Fine-Tuning for Hardest Questions. When the size of the fine-tuning buffer exceeds a predefined threshold M (i.e., |BufferrT| 2 M), we sample batch of M (g, s) pairs from the buffer and perform fine-tuning step using these hard examples. The fine-tuning step minimizes the standard cross-entropy loss:
|s|
(C
log Ta (si
821), where (g, BufferFT.
(3)
To prevent fine-tuning updates from overly constraining the model's exploratory behavior, we incor- porate an entropy regularization term into the loss function. The full objective is as follows:
(FT( = (fE(f) + Entropp (0)
|0]
|e| log no (s; q. 82)) a Σ H(To(s: 8.:)),
(4)
where (q. s) E Bufferrr. The term (s 14 (ce)) represents the entropy the token generation distribution (s; g, 8<2), and is hyperparameter controlling the weight of the entropy term.
The frequency this interleaved fine-tuning is adaptive. Early in training, when the model's per- formance is low, SFT is applied more frequently to accelerate the learning of effective reasoning patterns. As training progresses and the model improves, RL is prioritized to further incentivize the model's existing abilities. Furthermore, this online approach obviates the need for large, pre- collected dataset of CoT demonstrations s. Instead, CoTs can be generated dynamically only for the most challenging questions encountered during training.
5

---

Published as a conference paper at ICLR 2026
3 EXPERIMENTAL SETUP
Dataset Construction, Our training set is a subset of 1Math-2MMt Face (225)) with
prompts sourced from NuminaMath 1.5 L.5[Li et al. (2024) and detailed demonstration data generated by Deepseck-RI Guo et al. (2025). We use the default 94k prompt subset and filter out generations that are longer than 8192 tokens yields 46k prompts and hugh-quality demonstrations, which we term OpenRI -Math 466--192.
verified incorrect by Math-verify Face (2024). This process
Evaluation. For our evaluation, primarily consider five widely adopted mathematical reasoning benchmarks: AIME 2024, AIME 2025, AMC Li et al. (2024), OlympiadBench He et al. (2024), and MATH500 (eeddryksss 2025, and AMC, we report avg avg @32 for these datasets. For OlympiadBench and MATH500, we avg @8 as the evaluation metric. Since on mathematical reasoning, we also assess generalization performance on MMLU-Pro Wang et al.)
al. (02T)) Due to the relatively small test sets for AIME 2024, AIME
reinforcement learning training is mainly centered
2024. To prevent data contamination, we shuffle the multiple-choice options. A testing tempera- ture of 0.6 was used for all evaluations.
Baselines. We benchmark ReLIFT against the following baselines using Qwen2.5-Math-77 Yang et al. (2024) as the base model. For SFT methods, we fine- sane-tuning the base model on the OpenRI- 7okh-46k-8/92 dataset. For RL methods, we include GRPO (hao et al. (2024) trained on the same 46k dataset; SimpleRL-ZZero Zeng et al. (2025) which applies GRPO to approximately 24k math- ematical samples from GSMSK [Cobbe et al. (222)) and MATH dataset Hendrycks et al. (202I); OpenReasoner-Zeet Hu et al. (2025). PPO-based approach trained on 129k multi-source sam- ples; PRIME-Zero Cui et al. (2025), which conducts policy rollouts on 150k NuminaMath queries with implicit process rewards; and Oat-Zero et al. (2025), which removes the standard deviation in GRPO advantage calculation, and is trained on the MATH dataset. For methods combining SFT and RL, we evaluate three approaches: RL w/ SFT loss, which incorporates SFT loss term into the GRPO objective trained on the 46k dataset; LUFFY Yan et al. (2025), a mixed-policy GRPO approach that also utilizes the 46k dataset; and SFT then RL, a two-stage method that applies RL training after an initial SFT phase. Specifically, we evaluate two versions of SFT then RL: SFT then RL (v1) involves taking our trained SFT model and training it with RL for 300 steps on a held-out split of the OpenR 1 1aMth 220k dataset, which contains about 49,000 prompts, similar to previous work work[Cui et al. (2025) Yan al. (2025). For SFT then RL (v2), to fairly compare with Re- LIFT, which uses 8,640 demonstration samples, we first perform SFT on subset of 8,640 samples from OpenR/ -Math-46k--
Then we conduct GRPO training on the full OpenRI -Muth-46k-88111
dataset. For more detailed setup for training these baselines, please refer to Appendix B
4
EXPERIMENTAL RESULTS
4.1 MAIN RESULTS
Table 2: Comparison of resource require- ments of methods.
SOTA performance with Qwen2.5-Math-7B. Ta- ble | presents our main results, comparing ReLIFT against four previous state-of-the-art RLVR meth- ods and replication of SFT and RI, all based on the Qwen2.5-Math-7B model. Across five challeng- ing competition-level reasoning benchmarks and one out-of-distribution benchmark, ReLIFT establishes a new state-of- oftthe-rrt with an overall accuracy of 52.6%, surpassing all baselines. Notably, ReLIFT consistently achieves either the best or second-best performance on every individual benchmark, indi- cating ReLIFT achieves both higher reasoning per- formance and better generalization. More OOD re-
Model
GPU Hours
Demonstrations
SFT
8
46K
RL
40 x 8
0
RI. w/ SFT loss
113.5
46K
SFT then RL v1 SFT then RL v2 LUFFY ReLIFT
57 x 8
46K
63.5 x
8K
73 x 8
46K
52 x
8K
sults can be found in the Appendix G
Comparing against methods combining SFT and RL. When compared with SFT and RL methods trained on the same dataset, ReLIFT outperforms all methods that combine RL and SFT, including RL w/SFT loss, SFT then RL, and LUFFY, as shown in Table ReLIFT also produces more concise responses. Additionally, analysis of the required demonstration data and GPU hours in Table

---

Published as a conference paper at ICLR 2026
Table 1: Overall performance on five math benchmarks and one OOD benchmark based on Dwen2.5-MMath-1 Bold, single underline, and double underline represent the best, second-best, and third-best. ACC represents the average accuracy. LEN represents the average number of tokens.
A1ME-24
MATH--500
Olympind
MML.U-Pro
Model
ACC
LEN
ACC
LEN
ACC
LEN
ACC
LEN
ACC
LEN
ACC
LEN
ACC
LEN
Qwen2.5-Math-77
Qwen-Math
16.4
2125
6.9
2039
45.5
1493
65.1
1171
29.7
1594
24.9
830
31.4
1542
twen-MMatlInnsruett
9.3
4065
8.2
3590
40.5
2905
78.2
1774
36.2
2888
34,0
4274
34.4
3249
Previous RLVR methods
SimpleRI-Zern
27.0
6.8
54.9
76.0
34.7
34.5
39.0
Zee
58.7
PRIME-Zero
17.0
12.8
54.0
81.4
40.3
32.7
39.7
Oat-Zero
33.4
11.9
61.2
78.0
43.4
41.7
45.9
Replication of SFT and RT
SFT
26.9
7344
25.5
7059
59.8
5675
84.8
3503
52.6
5662
44.4
3956
49.0
5533
3287
17.5
1277
2178
2175
Methods combining SFT and RT.
RL w/ SFT loss
26.9
7344
23.1
7059
59.8
5675
84.1
3503
53.6
5662
44.4
3956
48.6
5508
SET then RL (v1)
29.4
5581
21.0
5328
63.1
3908
X7.3
2224
55.7
3772
47.6
2258
50.7
3845
SET then RL (v2)
26.8
6453
23.0
6231
63.8
4586
88.1
2682
54.4
4615
48.9
2638
50.8
4534
LUFFY
27.3
5741
23.0
5267
63.5
3477
85.6
2361
53.8
3758
52.6
2241
50.9
3808
ReLIFT
28.3
5062
22.2
4571
65.1
3540
87.9
2241
57.3
3352
53.9
2248
$2.6
3502
ReLIFT
4000
0.7
Response Length
3500
Reward
3000
Entropy
0.5
2500
0.15
2000
0.10
1500
0.05
0.3
1000
300 Step
400
150
Step
Step
Figure 3: Training Dynamic of rewards, response lengths, and the training entropy during RL and
ReLIFT training.
reveals that ReLIFT requires less training time and demonstration samples while achieving the best performance. These results underscore the effectiveness and efficiency RcLIFT.
4.2 TRAINING DYNAMICS
Figure
preseesssssentss presents the training dynamics of rewards, response lengths, and training entropy during RL and ReLIFT training. The reward achieved by ReLIFT consistently surpasses that of RL, suggesting that ReL.FT is more effective in optimizing the target objective. Regarding response lengths, RL. tends to generate shorter responses as training progresses, which may reflect a diminishing capacity to address challenging questions. In contrast, ReLIFT demonstrates a gradual increase in response length, implying a higher potential to solve difficult problems. The entropy of the RL model steadily declines throughout training, indicating reduced exploration. contrast, the entropy ReLIFT remains relatively high and exhibits persistent fluctuations, consistently surpassing that of RL. This sustained exploration allows ReLIFT to continue discovering novel solutions, thereby facilitating ongoing performance improvements even in the later stages of training.

---

Published as a conference paper at ICLR 2026
Table 3: Overall performance on five math benchmarks and one OOD benchmark based on "wen2.5-Math--1.5 and Qwen2.5-7B. Bold and underline indicate the best and second re- sults, respectively.
AIME-24
AIME-25
MATH--500
Olympiad ACC
MMLU-Pro
Model
ACC
LEN
ACC
LEN
ACC
LEN
ACC
LEN
LEN
ACC
LEN
ACC
LEW
Qwen2.5MMath-L.58
Qwen-Math-l.5W
2.8
1971
1.3
2058
19.5
1533
45.4
1108
18.5
1619
5.2
1982
15.5
1712
Owea-Math-L.SS-Insteee
10.3
4042
7.6
3764
40.0
2809
77.8
1628
35.7
2834
33.7
4300
34.2
3230
71.8
RT
9.8
2425
8.5
2195
44.1
1569
74.2
958
37.2
1695
31.4
1075
34.2
1653
ReLIFT
14.3
3691
10.0
3416
47.2
2207
76.4
1308
39.6
2274
31.7
1724
36.5
2437
Qwen2.5-7B
Qwen-7B
6.2
1632
2.4
1370
31.1
1076
63.8
617
26.5
1068
32.8
945
27.1
1118
Owen-7B-Lnstrue
3325
SFT
15.7
7786
18.6
7533
49.8
6424
80.8
4252
44.3
6234
54.6
4869
44.0
6183
RL
15.5
1784
13.0
1487
50.4
1422
80.0
917
42.2
1438
57.6
954
43.1
1334
ReLIFT
LLaMA-3.1-8B
38.1
0.8
2033
1.5
2032
11.5
2007
28.6
1903
8.6
1997
28.2
1558
13.2
1922
RL
1.8
940
0.0
909
10.8
797
28.2
616
7.3
790
39.4
549
14.6
767
ReLIFT
35.2
4.3 ABLATION STUDY AND ANALYSIS
We conduct ablation study understand what makes ReLIFT effective. Our findings reveal that the scheduling of fine-tuning and the selection of training data are crucial. Specifically, we compare the following three ablation settings: (all, which alternates one fine-tuning step after each RI. step on the same batch, with the number demon- strations matching SFT; (un.ffr))))frrm, which applies fine-tuning at fixed, uniform intervals (ev- ery 8 RL steps), with the number of demonstra- tions matching ReLIFT; ReLIFT(random) which performs interleaved fine-tuning. The buffer is popu- lated based on the number Hardest questions, but instead of using the hardest questions, it randomly filled with non-hardest questions.
60
0000
$2.00
40.30
6000
5220
5000
4000
3000
1000
RWUFT
ReLIFTIal)
INCIFT Juniform)
ReUFT trandoml
Figure 4: Ablation study ReLIFT. The left bar and the right bar represents average accu- racy and length, respectively.
As shown in Figure 4 ReLIFT (all) quickly col- lapses. probably due to conflicting optimization ob- jectives between RL and FT. Similarly, using uniform or random schedule for FT, as in Re- LIFT(uniform) and ReLIFT(random), leads lower accuracy and longer responses. In contrast, ReLIFT strategically applies fine-tuning more frequently in early training stages and focuses the most chal- lenging questions. This approach allows the model to efficiently gain new knowledge without sacrific- ing performance or conciseness. More detailed re- sults are in Appendix [
52.0
$1.5
Accuracy
50.0
3.0e-05
1.5w0tt
1.50.0
tuning
We also study the effect of the entropy coefficient, a,
Figure 5: Accuracy the choice of on the fine-tuning step, using Qwen2.5-Math- As seen Higure d 0.0001 prove optimal,
8

---

Published as a conference paper at ICLR 2026
achieving the highest accuracy of 52.6. Varying in either direction from this value significantly de- creased performance. These results highlight the importance of the entropy loss term for effectively blending SFT with RL. Therefore, use 1 x 10 consistently in our experiments.
4.4 EXTENSION TO MORE MODELS
To evaluate the generalizability of our method, we extend ReLIFT to three additional base mod- els: the smaller Owen2.5-Math- 1.5B Yang al. (2024), the weaker Qwen2.5-7B Team (2024), and Llama3. Dubey et al.] (024)) which has a different architecture. As shown in Table 3 Re- LIFT consistently achieves substantial improvements across all base models and six benchmarks, surpassing both SFT and RI. Our method consistently achieves a top-two ranking while maintain- ing an acceptable response length. These results underscore the broad applicability of ReLIFT in achieving superior performance and generalization across diverse model scales and architectures.
RELATED WORKS
5.1 RL- EMERGENCE OF REASONING ABILITIES
RL has recently emerged as a promising approach for enhancing the reasoning capabilities of LLMs. Early breakthroughs, such as OpenAl o-series Jaech et al. (2024); OpenAl (2025), DeepSeek- RI Guo et al.) (2025), and Kimi k-series Team al. (2025b a). have demonstrated that applying RLVR can lead to significant improvements in complex reasoning tasks. Notably, DeepSeek-RI further shows that applying RLVR directly to base model, without the need for intermediate SFT. Building on these foundations, Light-rl Xie et al. (2025) incorporates curriculum learning to refine LLM's s reasoning skills, while Logic-rl [ii et al. (2025) adopts logic-driven reward functions to im- prove general reasoning ability. Deepscaler Luo Luo et al. (2025) trains LLMS with progressively longer contexts as their performance improves. Based GRPO, DAPO Yu et al. (0255) uses clip-higher to prevent entropy collapse, dynamic sampling for improved efficiency, and token-level policy gradient loss with overlong reward shaping stabilize training.
5.2 TRAINING PARADIGMS
Despite these significant advances, Yue et al. (2025) argue that RLVR does not enable LLMs to acquire genuinely new reasoning abilities beyond those present in their base models, as RLVR- trained models perform worse on pass@ @ass@k than the base models at higher k values, indicating limited reasoning capability expansion. Similarly, /Zhao et al. (2025) demonstrate that RL creates an "echo chamber" effect, amplifying specific behavioral patterns already present in the pre-training data while suppressing others, thereby reinforeing existing behaviors rather than creating new reasoning capabilities. Furthermore, Cheng et al. (2025) finds that RL stifles exploration, causing models to converge on narrow behaviors and leading to performance plateaus on complex reasoning tasks. Research suggests that current RLVR methods do not fully leverage the potential of RL to create genuinely novel reasoning abilities in LLMs. In contrast, directly fine-tuning on demonstration data introduces knowledge LLMSs. However, SFT requires high-quality, detailed demonstration data and tends to encourage memorization of the training data, often resulting in poor generalization to -ito---distribution scenarios Chu al. (2025); Chen al. (2025): Team al. (2025b).
Some approaches attempt to combine SFT and RL. Typically, these methods first use SFT to stabi- lize the LLM's ability and then apply RL unlock more advanced reasoning skills in LLMs Guo et al. (2025); Wen et al. (2025). However, such combined methods remain limited: SFT depends significantly on the quality of demonstration data, and the subsequent RL phase is still confined to the knowledge present in the SFT-trained model, making it difficult for the model to acquire gen- uinely new information. LUFFY Yan al. (2025) seeks to address these limitations by augmenting RL with off-policy reasoning traces, dynamically balancing imitation and exploration through combination of off-policy demonstrations and on-policy rollouts during training. Nevertheless, this approach necessitates the preparation of off- data in advance and relies heavily on probability- sharpening techniques, which may restrict its applicability. In contrast, ReLIFT proposes a simpler and more direct method that integrates RL and SFT to leverage the strengths of both paradigms. Our approach encourages LLMs to explore when possible and integrate new information as needed, thereby pushing RL-driven reasoning beyond the boundaries of existing knowledge.

---

Published as a conference paper at ICLR 2026
6 DISCUSSION AND CONCLUSION
6.1 DISCUSSION
In experiments, use DeepSeek-RI the source for high- quality data, decision made for reproducibility and cost-effectiveness. This does not imply that ReLIFT is distillation method. The core mechanism of ReL.IFT-identiffin "hardest" problems online and interleaving FT-is source-agnostic As mentioned in the paper, the framework is equally applicable to other data sources, such as human expert annotators. Due resource constraints, this study does not include experiments related to human annotation; this task is reserved for future research. Therefore, based solely on the experiments presented here, the approach is considered distillation method. However, the scalability of our method, owing to its low requirement for demonstration data, maintains the feasibility of using human annotation.
Data Source and Core Method:
While our experiments use offline-collcted data due to resource constraints, a true online collection is technically feasible. The key is that the RI policy update step is time-intensive, which is sufficient to "mask" the latency of external data collection. The system can asynchronously dispatch identified hard problems to an API or human queue while the RL update is being computed. As long the data generation rate matches the RL's problem identification rate, online collection would not slow overall training.
Feasibility of Online Collection:
6.2 CONCLUSION
In summary, we systematically analyze the respective strengths of RL and SFT for LLM reasoning, finding that RL is more effective for learning easier questions while SFT is crucial for solving chal- lenging questions. Building on these insights, we introduce ReLIFT, an approach that interleaves RL with online fine-tuning on the hardest questions. ReLIFT achieves state-of-the-an results with significantly less demonstration data, reduced training time, and more concise solutions. Future work will focus on scaling ReLIFT to larger models and developing more effective strategies for coordinating SFT and RL to further advance LLM reasoning and generalization.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grant Nos. 92470121, 62402016, U23B2048, U22B2037), the National Key R&D Program of China (Grant No. 4000)), the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (Grant No. JYB2025XDXM109) Zhongguancun Academy (Grant Nos. C20250204, C20250602), and the High-Performance Computing Platform of Peking University.
REFERENCES
Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, Jeong Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. arXiv preprint arXiv:2504.033380 2025.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or ri? an early investigation into training ri-like reasoning large vasion-language models. arXiv preprint arXiv:2504. 1468, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis. Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec
10

---

Published as a conference paper at ICLR 2026
Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Me- Grew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. 2021.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:250.. 14758, 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le, Sergey Levine, and Yi Ma. Sft memorizes, r1 generalizes: A comparative study of foundation model post-training. arXiv preprini arXiv:2501.17100 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
NoXiv:/803.05457v.. 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprini arXivv2507.06261 2025.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint 0.Xx:22002.01456, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2401 2024.
Hugging Face. Math-verify. Accessed: 2024-06-10.
http:://tthaa.cc////ggggggg//g/h
2024.
Hugging Face. Open rl: A fully open reproduction deepseek-rl, 2025.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Kricken, and Pasquale Minervini. Are we done with mmlu?, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.1294/ 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: challenging benehmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. /4008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874 2021.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner An open source approach scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.22 24290, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1677 2024.
11

---

Published as a conference paper at ICLR 2026
Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, et al. A survey of frontiers in Ilm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.09337, 2025.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su. John Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. /29/7, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Shuangtao Li, Shuaihao Dong, Kexin Luan, Xinhan Di, and Chaofan Ding. Enhancing reasoning through process supervision with monte carlo search. arXiv preprint arXiv:2501.01 004488, 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint 2rXiv:2503.2078: 2025.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol ol-preview with 1.5 b model by scaling rl. Notion Blog, 2025.
Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, et al. Injecting new knowledge into large language models via supervised fine-tuning. arXiv preprint 23X11:2404.00211,, 2024.
OpenAl.
Openai o3 model.
https://opeaal.com/nnnc
Introducing--3 and- 04 minii/ 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for I1m reasoning, arXiv preprint 4rXiv:2410.08146. 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.2053: 2025a.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501. 2025b.
Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.151111 2024.
12

---

Published as a conference paper at ICLR 2026
Y. Wang, X. Ma, G. Zhang, et al. Mmlu-pro. A more robust and challenging multi-task language understanding benchmark. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchnarks Track, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint urXiv:2503. 125500000600 10460, 2025.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 14768, 2025.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv: 00451, 2024.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu Yu Cheng, and Yue Zhang. Learning to reason under off- polico guidance. arXiv preprint arXiv:2504. 200:22504.4444., 2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.1212 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprini 2r03v: 14476, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does inforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arxiv:2504 13837, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025.
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912. 2025.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Ye Ye Yenhhnn and Zheyan Luo. Llamafac- tory: Unified efficient fine-tuning of 100+ language models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 400-410, 2024.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
We use LLMs to polish our writing. They act as a sophisticated assistant, improving the clarity, grammar, and style of our text.
B MAIN EXPERIMENT DETAILS
RL Implementation. For RL, omit the KL divergence loss term following Luo et al. (2025) Yan et al. (2025). Consistent with Dr.GRPO Liu et al. (2025) and LUFFY Yan et al. (2025), we exclude both length normalization and standard error normalization from the GRPO loss in all experiments. The rollout batch size is set to 128, the number of rollout 8, the update batch size is 64, and the learning rate is 1 X 10 6 A temperature of 1.0 is used during rollout generation for exploration. For Qwen2.5-Math-7B. We train 600 steps for all methods except SFT then RI., to be near with the GPU hour of ReLIFT. For other base models, train 500 steps. All experiments conducted on 8 x A800 GPUs.
SFT Implementation. For all our SFT models, utilize the OpenRI-Math-46k--8 dataset. Following hyperparameter search on a sampled small-scale dataset, we evaluate batch sizes of 64, 128, and 256, in conjunction with learning rates of 10-5, and 5 10-5, 5_ The optimal configuration identified includes a batch size of 64 and a learning rate of 5 x 10-5. Employing the L.aMM-Factory framework /Zheng al. 2024), we train our models for epochs.
RL w/ SFT Loss Implementation. We compute the on-policy loss on 7 on-policy samples and SFT loss on 1 off-policy sample per prompt. The remaining setup is consistent with our RL. experiments.
SFT then RL Implementation. We conduct SFT using the same settings outlined in the SFT Implementation and train for 3 epochs accordingly. Subsequently, the RL phase follows the same configuration as described in RL Implementation, and we train the RI phase for 300 steps.
ReLIFT Implementation. During interleaved fine-tuning, we make use of demonstrations collected beforehand. The RI. and fine-tuning configurations for ReLIFT are consistent with the settings previously described except learning rate. For both the RL and interleaved SFT phases, the learning rates are set to x 10-5.
Evaluation. All evaluations are conducted using VLLM Kwon et al. (023), with the tempera- ture set to 0.6 and the maximum number of tokens set to 8192. Results verified using MATH- VERIFY Face (2024). For the baselines, we independently reproduce and verify the results of all methods combining SFT and RI., such as LUFFY Yan et al. (202),, whereas the results for previous RLVR methods are reported as in the LUFFY paper.
Reward Function. To evaluate the impact of our method, adopt simple reward function as below, All training experiments employ the same reward function.
1, if the answer is correct
r=
0, otherwise
Qwen2.5-Math Models. All evaluations are conducted using VLLM Kwon et (023). with temperature setting of 0.6 and a maximum token limit of 8192. Results are validated using MATH- VERIFY Face (2024). For baseline methods, we independently reproduce and verify results for all approaches combining SFT and RL, including LUFFY Yan et al. (2025). The results for other RL models are reported following the findings presented the LUFFY paper.
Llama-3.1-8h5 Because the full OpenR -6kuhh-6k-88888 -89- dataset proved too challenging for Llama- 1-B, we created smaller, more manageable dataset. We selected 11,000 samples from OpenR1 where DeepSeek-RR provided the correct solution and the solution length
under 2,048 tokens. The SFT was then performed using a learning rate x 10-5, 5, as suggested by Liu et al. 20255) The maximum response length is set 2048 tokens for RL training and evaluation.
Motivation Experiment Details. We randomly sample a subset of 8,000 examples from the training dataset described in Section Bas the training set, and select 1,000 examples the validation set. RL and SFT are both conducted on this subset dataset for 120 steps, using the same parameters as in our main experiments.
14

---

Published as a conference paper at ICLR 2026
C CHAT TEMPLATE
To encourage the base model to reason more effectively, following Yan et al. (0025 we adopt complex chat template as shown in Figure our trained models, except LL 1.MA3.11-88 share the same system prompt for training and inference.
Qwen Chat Template
Your is to follow systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and and Solution
In the Thought section, present your reasoning using the format: ">think>lnn (thoughts) </think>>>". Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas.
After "</think~ln, " in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the houghttscctonn
If applicable, include the Answer in (boxed for closed- form results like multiple choices or mathematical solutions.
User: This is the problem: (Question}
ssistant: <think>
Figure Chat template for Qwen-series.
Llama Chat Template
Question: (Question).
Answer: lett think step by step.
Figure 7: Chat template for Llama3. 1-8B.
For (LaMA-3.1-82 we do not use the above system prompt as we find the model cannot follow such instruction. Thus, use a simplified version shown in Figure 7
D DETAILED EXPERIMENT RESULTS
Table Ablation study on ReLIFT. Bold indicate the best result.
AIME-24
AIME-25
AMC
MATH-500
Olympiad ACC
MMLU-Pro
Overall
Model
ACC
LEN
ACC
LEN
ACC
LEN
ACC
LEN
LEN
ACC
LEN
ACC
LEN
ReLIFT
28.3
5062
22.9
4571
65.1
3540
87.4
2241
57.8
3352
53.9
2248
52.6
3502
(eLIFT(all)
7.8
7647
9.0
7648
31.6
6653
54.2
5320
26.4
6600
13.7
6590
23.8
6743
ReLIIF(uniform)
49.1
(eL.IFT(radoo)
22.1
4616
13.9
4676
59.5
5271
84.2
5820
44.7
5187
48.4
6020
45.5
5268
The comprehensive results for ReLIFT(all), ReLIFT(nnifrm)) and (eLdFT(random) are presented Table These findings indicate that merely alternating between RL and fine-tuning insufficient; both the scheduling and the selection of fine-tuning training data are crucial components for success.
E REASONING BEHAIVERS
To analyze the distinct behaviors of the RL, SFT, and ReLIFT models, first curate a pool of keywords corresponding to three key cognitive actions: summarizing, rethinking, and planning. We then quantified the frequency of these keywords within the AIME25 responses generated by
15

---

Published as a conference paper at ICLR 2026
Summary Keywords
Rethinking Keywords
Plan Keywords
(number of key wordswytoken sttkkes
8
7.1
0.6
0.58
0.4
4
0.28
1/
0.8
refiit
reflit
Sif
reflit
Figure 8: Normalized Keyword Counts for RL, SFT, and ReLIFT Models on AIME25
each model. To ensure fair comparison varying response lengths, all keyword counts normalized by the total number of tokens. As shown in Figure 8, our analysis reveals several key differences in how the models approach problem-solving.
Summarizing: The SFT model exhibits notably higher usage of summary keywords, suggesting often prioritizes synthesizing and restating information. The ReLIFT model uses slightly more summary keywords than the RL model, but both are far behind SFT.
Rethinking: In contrast, the ReLIFT model uses significantly fewer keywords related to rethinking or re-evaluation. This finding suggests a higher level of efficiency and confidence in its initial approach, requiring less internal correction or revision during the problem-solvin solving process.
Planning: The most striking difference is in the use of planning keywords. The ReLIFT model demonstrates a substantial increase these keywords, indicating more robust and sophisticated approach to strategic planning. This suggests that ReLIFT is not just more efficient but also more capable developing a logical and powerful wrolemmssolving trajectory.
F
CASE STUDY
AIME25 I Problem 3: The first step to solve this problem is to find three possible combinations (6, 2, 1). (5, 3, 1) and (4, 2,). The RL response fails to include one possible combination, while the SFT response requires excessive analysis obtain the three combinations. In contrast, the ReLIFT response more concise and obtains all three combinations.
AIME25 Problem
Question: The 9 members baseball team went to an ice cream parlor after their game. Each player had singlescoop cone of chocolate, vanilla, or strawberry ice cream. At least one player chose each flavor, and the number of players who chose chocolate was greater than the number of players who chose vanilla, which was greater than the number of players who chose strawberry. Let N be the number of different assignments of flavors to players that meet these conditions. Find the remainder when N divided by 1000.
16

---

Published as a conference paper at ICLR 2026
ReLIFT (Correctness: True; Length: 1819 tokens)
... Let's denote: c as the number of players who chose chocolate. - t as the number of players who chose vanilla. strawberry. The conditions given are: 1. c+ t = 92. (c > v >>> 8) 3. Each flavor is chosen by least one player, s0 c 2 1, u 2 1, and 2 #i## Step 2: Determine Possible Values for U, Since c, U, 2 and c + + = 9. we can start by considering possible values for U, s that satisfy (c > V > s) and all being positive integers. ### Step 3: Find Possible Combinations of c, U, We need to find all the integer solutions to the equation c
the number of players who chose
8 where (c > > 2 1). Let's list possible values for c, 0, 8: First, let's note that must be at least and since (v 8), v must at least 2, and since (c > v). must be at least 3. Also, since + + 9, the maximum value for can be determined by assuming the minimum values for v and s. If s = and and v 3,
2, then then e = 4. So possible values for can be 4, 5, 6. Let's list all possible and (v
6. If 1 and
3, then 5. If s
combinations: If 6, then v +
2 1). Possible pairs
(u,8) 8) = 2, 1). If = 5. then v + = 4 and (v > 8 2 1). Possible pairs
(( 8) = (3, 1)) or (2, 2)). ) but since (o s). only ((v,s) =),,
4, then
5 and (v > 2 1). Possible pairs are (u,s) = 23.
So the possible combinations are: (6, 2, 1), (5, 3, 1) and (4,3,2) Final Answer The final answer 016
RL (Correctness: False; Length: 1269 tokens)
Let's denote: C' as the number of players who chose chocolate, V as the number of players who chose vamilla, - the number players who chose strawberry. We need to satisfy the following conditions: 1. C' + V + S = 9, CiViS 1. First, let's find all possible combinations of (C, V,5) that satisfy these conditions. We'll iterate through possible values to find all valid triples (C, V, and then calculate the number of different assignments for each valid triple. Since CgVi 2 and C+ V + S 9, the possible values for can range from I to 2 (since if S 2 3, then 2 and C' Z 5, which would make S+V+ > 12, which is not possible). Let's list out the possible triples: 1. 5 = 1: C
= and CiVi1. Possible pairs for (C, V) are + V = 7 and C&V22. The only possible pair for (C, V) is (5, 2), but this does not satisfy V&5 since V
(6, 2) and (5, 2. If S
S. So,
= 2 does not yield any valid pairs. Thus, the valid triples (6, 2, 1) and (5, 3, ... Thus, the answer is 756
17

---

Published as a conference paper at ICLR 2026
SFT (Correctness: True; Length: 6636 tokens)
... We have 9 players, each choosing one of three flavors: C, V, S. Each flavor must be chosen by at least one player. So, can't have all choosing C, or only Cs and Vs, etc. Also, the number of Cs > number of Vs > number of Ss. So, need to count the number of ordered triples (c, v, S) where S 9, 2 1, v 2 1, 2 I, and с > > Then, for each such triple, need compute the number of ways to assign the flavors to the players, which would be 9 choose c, then choose which players get chocolate, then from the remaining get strawberry. So, for each valid (c, v, s), the number of assignments is C(9, c) * C(9 c. v) C(9 C- v. s). But since the players are distinct, once we choose the с players for chocolate, the from the remaining for vanilla, etc., that's the standard multinomial coefficient. So, for each valid triple (c, v, s), the number is 91 / (c! v! s!). Then, find all triples (c, V, 8) such that с + s 9, 2 1, 2 1, $ 2 1, and e >> Then, for each such triple, compute the multinomial coefficient 9! (c! v! s!) and sum them all up. Then take that total modulo 1000. So, first, let's list all possible triples (c, s) where
players, choose v get vanilla, and the remaining
over all valid triples. So, first step:
2 1. and V s= Let me start by considering possible values of s. Since is the smallest, and s 1, so s can be 1, 3, 4. Let's check: If = 1, then we need
1, so v 2, and с v. Also, c + =- c === = Since c > v 2 2, and c v 8. So possible (v, c): can be 2, 3, 4, 5, (since has to be greater than v, and
must be at least 2, and с = 8 must be greater than v). Let's check: v=
=6. So (6, 2, 1). Check с > v s:
2 1. Yes. v= = (5, 3, 1). 3 >>1 1. Good. = c= 4. But must be greater than v,
is not > 4. So invalid. == 5: c But then == 3, which is not > v 5. So invalid.
6 2. Similarly, invalid. So for
possible triples are (6,2,1), (5,3.1).
Next, $ = Then 5 So +
=> c+ v= With c > V 2. So Let's check possible
3: 4. (4, 2). 4 3 > 2. Valid. v= 4: = But c= is not > == Invalid. v= 2 Also invalid. So only (4, 2) for = s 3: Then c v+
must be least and с 7
+ 6. With > v>> 3. So must be at least 4, = - v. Let's check: no solutions here. s= 4: Then c 4 c v With c v> 4. But must be least which would make с = 0, which is invalid. So no solutions here. So the possible triples are: For 1: (6.2.1). (5.3,1) For 2: (4.3,2) Total of triples. Wait, me check again. For 1: c v= = 8. c > v 2. (invalid since must be > v) v 5, c (invalid) v 6, c (invalid). So yes, only (6,2,1), (5,3,1) For 2: c 7 с > v 3. So can be 3, 3 = (invalid) 5, c 2 (invalid). So only (4.3,2) == c == 6. c v 4. v= 4, (invalid). So
c
is not > 4 Invalid. v= c Also invalid.
So possible pairs: = 2, C= v 3, c v
4: same above. So total three triples: (6,2,1), (5,3,1), (4.3,2) Thus, the remainder when N is divided by 1000 is
G MORE OOD RESULTS
We evaluate four new OOD benchmarks: ARC-Challenge (ARC-c) Clark et al. (2018), GPQA- Diamond (GPQA) Rein et al. (004)). MMLU-Redux (Redux) Gema et al. (2024), and Hu- manEval Chen al. (2021), in order to comprehensively test the generalization ability of our method. Among them, ARC-Challenge and EPQA-Diamone focus on reasoning ability in the STEM fields, HumanEval focuses coding, and MMLU-Reduu is carefully annotated version of MMLU, covering questions from a wider range domains. The results from these evaluations summarized in the table below.
18

---

Published as a conference paper at ICLR 2026
Table 5: Overall performance on four diverse benchmarks (ARC-c, GPQA, Redux, Humaneval)
based on different models. Bold and underline indicate the best and second-best results, respectively.
ARC-c
GPQA
Redux
Humaneval
Models
Acc
Len
Acc
Len
Acc
Len
Acc
Qwen-Math
0.503
504
0.264
1110
0.474
555
0,518
Qwen-Math-Instruct
0.657
2881
0.228
5071
0.481
3628
0,435
RL w/ SFT Loss
0.635
1814
0.274
5942
0.591
2514
0.604
SFT
0.750
1515
0.265
6547
0.601
2479
0.553
RL
0.808
639
0.401
1881
0.657
863
0,581
LUFFY
0.804
1168
0,441
3285
0.652
1516
0.572
SET then RL(v1)
0.723
879
0.289
3779
0.562
1217
0.548
SFT then RL(v2)
0.728
980
0.339
4490
0.620
1501
0.627
ReLIFT
0.816
1321
0.431
3177
0.670
1511
0.643
ReLIFT performs strongly across these benchmarks, achieving the highest average score and ranking either first or second on every single one. This underscores the robust generalization ability enabled by ReLIFT's integration of RL and fine-tuning, regardless of whether the tasks involve code. STEM, or other domains. Notably, while RI. excels on ARC-Challenge and MMI MMI-U-Redux due its strong OOD paapliiationnouuurfffrrmngg other methods combining and SFT IeL.IFT still manages to surpass it.
H ABLATION STUDY ON BUFFER SIZE AND DIFFICULTY THRESHOLD
We conduct a detailed ablation study focusing two key hyperparameters of ReLIFT: the FT buffer threshold (M), which controls the maximum size of the fine-tuning buffer, and the difficulty threshold (Q). The difficulty threshold Q acts the gating function, selectively determining which those with an accuracy less than or equal to Q (i.e., incorrect/uncertain sam- ples) are retained for SFT. This evaluation performed over 400 training steps the five math benchmarks, with the results summarized in the 6
Table 6: Ablation study buffer size and difficulty threshold
M = 32
M = 64
M = 128
M = 256
0
0.411
0.490
0.478
0.459
1/8
0.405
0.404
0.493
0.471
1/4
0.392
0.403
0.467
0.491
We observe that extreme settings for the key hyperparameters--the difficulty threshold Q and the buffer size M-led to suboptimal results. Specifically, when the buffer size M is overly small (e.g.. M 32) or when relaxed filtering condition Q is combined with a small M (e.g.. Q 1/4 with M = 64), the model's performance significantly degrades (down to 0.392 and 0.403). This clearly indicates that either an small size buffer or an overly relaxed filtering condition results in too frequent SFT updates, failing to provide the necessary stability during RL. The ablation results reveal the existence of several robust parameter configurations that yield peak performance (approximately 0.49 + 0.003):(M 64, Q 0), (M 128, Q 1/8) and (M 256, Q 1/4). All these settings effectively balance data selection and fine-tuning capacity. However, when considering implementation efficiency, the data. Therefore, (M 64, Q = 0) remains our most recommended configuration, achieving near- optimal performance while minimizing the complexity of the gating function.
0 condition requires the minimum amount of demonstration
19

---

Published as a conference paper at ICLR 2026
3.6
04
03
Step
(a) Q=
(b) Q = 0.125
(c) Q 0.25
Figure 9; Training dynamic of rewards during ReLIFT training different Q and M.
The training curves for different combinations of M and Q are shown in Figure It is obvious that extreme settings for Q and M lead to suboptimal results.
20


Reviews:
[{"review_id": "lYcMyPzvtQ", "reviewer": "Reviewer_z9hW", "summary": "Current form of RL for LLMs is insufficient to induce capabilities that exceed the limitations of the base model. This paper targets at how to train reasoning-focused LLMs using RL interleaved with SFT. The paper claims that RL mostly explits what the base model already knows and SFT is better at teaching new and harder behaviors. Though SFT requires a lot of demonstrations and sometimes damage the performance oneasier questions. The paper introduces ReLIFT (Reinforcement Learning Interleaved with Online Fine-Tuning), a training framework that mixes RL and SFT adaptively rather than in fixed stages. The method first do RL on a dataset and collect hardest questions at the same time. When the hard-question buffer is full, it will invoke a SFT on these hard questions, followed by another round of RL training. Experiments show that ReLIFT outperforms pure RL, pure SFT, and prior hybrid/RLVR methods across multiple math benchmarks and an OOD benchmark, achieving higher accuracy with fewer demonstrations and more concise reasoning, suggesting that targeted, online SFT on model's failures is an efficient way to expand a model’s reasoning abilities.", "questions": "* line 300: The paper mentions that temperature is set to 0.6 in all evaluations. However, for OlympiadBench and MATH500, the paper states that \"For OlympiadBench and MATH500, we use pass@1 as the evaluation metric\". I wonder if this is a reliable evaluation on these two benchmarks, as setting temperature=0.6 indicates that the response from the trained model is not deterministic.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper first do analysis on training dynamics of RL vs SFT across different difficulties, showing that RL mainly preserves existing skills while SFT can unlock previously unsolved questions. After that, they propose ReLIFT, which does RL as the primary loop while streaming in SFT steps only on hard questions. The method proposed is gounded on grounded observation, which makes sense.\n* In the experiments, the paper includes careful controlled comparisons of pure RL, pure SFT, and multiple hybrid baselines (e.g., RL+SFT loss, LUFFY, ...) on the subset of OpenR1-Math-220k dataset, with Qwen2.5 model. The performance of the model trained with ReLIFT seems to beat all baselines and prior RLvR methods while using less detailed demonstrations.\n* The paper is well-structured and easy to follow. The motivation of the proposed method is illustrated clearly in Section 2 and the introduction of the method is detailed. The reproduction of the method is very likely.\n* The framework is simple, model-agnostic, and shown to generalize across different model sizes and architectures, it’s likely to influence how future reasoning LLMs combine RL and SFT", "weaknesses": "* The proposed method ReLIFT is built upon the assumption that high-quality CoT already exists so that smaller LLMs can benifit (it's actually a form of distillation from large LLMs like DeepSeek-R1). Although the paper also mentions that such high-quality CoT may come from human annotators in line 202, detailed discussion on \"human annotation\" is missing. Therefore, the scope of this paper seems limited to how to efficiently distill from larger LLMs to improve the performance of smaller LLMs, rather than pushing the performance boundary of exisiting LLMs.\n* All main experiments are on five math benchmarks plus a single OOD benchmark (MMLU-Pro). There’s no evaluation on code / STEM or other tasks, which can't fully support the claim that \"ReLIFT is a powerful and resource-efficient paradigm for developing capable **reasoning** models.\"\n* “Hardest” questions are defined as those with rollout accuracy acc(q)=0 given N samples. Those questions are the only ones that enter the buffer then for SFT. This binary cut can misclassify borderline questions (e.g., model answers corretly only once) and depends heavily on sampling noise. There’s no ablation on using thresholds like acc(q) ≤ p, or on how many “hardness levels” might be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2w0VCSAJfG", "reviewer": "Reviewer_u4oo", "summary": "The paper explores the complementarity between Reinforcement Learning (RL/RLVR) and Supervised Fine-Tuning (SFT) for reasoning LLMs, and proposes ReLIFT—an alternating training strategy. \n\nDuring RL, the method identifies the “hardest” problems (current rollout accuracy acc(q)=0), collects or generates high-quality CoT solutions into a buffer, and periodically performs an SFT step before returning to RL. On five math reasoning benchmarks and one OOD benchmark with Qwen2.5-Math-7B, ReLIFT improves overall accuracy over RLVR and mixed RL+SFT baselines, while producing shorter outputs and requiring less training time and fewer examples.", "questions": "see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper clearly defines the GRPO objective and the alternating SFT loss with entropy regularization (α), and the training curves show how reward, length, and entropy evolve over steps, supporting a mechanism of continued exploration and steady gains.\n\n2. The method is easy to follow thanks to the flow diagram (Figure 2), the difficulty stratification, and the explicit buffer trigger condition \n（Buffer_ft >= M）which together make reproduction straightforward.", "weaknesses": "1. The OOD evaluation relies only on MMLU-Pro and the main experiments focus on math reasoning; please add code, science QA, and multi-step commonsense tasks to test adaptability under different verifiable rewards.\n\n2.  Beyond **acc(q)=0**, please evaluate thresholds based on uncertainty, length anomalies, or self-contradictions in the CoT, and formalize the adaptive SFT trigger as a gating function of reward or entropy; report learning curves under different gating hyperparameters.\n\n3. Although ReLIFT yields shorter outputs on average, please check whether necessary long reasoning is over-penalized on AIME-style problems and provide length–accuracy analyses to identify such cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sk8IQdCJ8q", "reviewer": "Reviewer_H2Ni", "summary": "The paper examines the differences between how LLMs learn to solve reasoning problems across various difficulties when using SFT or RL. Informed by this analysis, the paper proposes selectively using demonstrations for intermittent SFT stages during online RL training when faced with the \"hardest\" problems (i.e., those for which no solution in the GRPO group is correct). The primary results show that this is, for the most part, more effective than appropriate baselines at greater compute efficiency.", "questions": "Have you tried the SFT baselines with entropy regularisation?\n\nIt is not clear if the stable response length for the hardest problems is related to exploration or an artefact of prior training stages. For example, it could be because the response length for these problems is already at the maximum that was seen in training prior to RL and that the demonstrations used for SFT are longer. Could you clarify how response length in the demonstration data compares to generated response lengths from the trained model?\n\nHow do the results change if the group size is doubled?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a method that is grounded in a detailed and revealing analysis about the differing impact of RL and SFT. The core results show that the proposed method yields benefits in-domain (math benchmarks) and on a single out-of-domain benchmark (MMLU-Pro). The analysis of training dynamics and the ablation studies effectively reveal how the proposed method mitigates the limitations of conducting SFT or RL alone, or as distinct training stages.", "weaknesses": "ReLIFT is considerably less effective on LLaMA-3.1-8B than on Qwen models. It would be useful to include the full set of baselines for a model outside of the Qwen model family, so as to ensure the generality of the method. Currently, only SFT or RL alone, in addition to the instruct variant, are used as baselines for the Llama model. \n\nUsing a fixed group size of 8 for all experiments means that it is unclear whether the problem difficulty classes assigned during online RL hold for a larger group size and how this impacts the effectiveness of ReLIFT and baselines. Using at least one more group size (say 16) for the core experiments would be insightful.\n\nIn practice, all demonstrations are collected offline before finetuning, which does not fit one of the core motivations for ReLIFT: only collecting demonstrations where necessary. However, collecting demonstrations on-the-fly would add to the complexity of the training pipeline and introduce latency. Discussion of this limitation would be useful.\n\n(Nit) Using the same underline for the second and third best result reduces clarity.\n\nIt is unclear whether the entropy regularisation term used in ReLIFT is also used for SFT baselines. This is an important baseline variant that is missing if I am not mistaken.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "USJ48YC5N6", "reviewer": "Reviewer_tK9a", "summary": "The paper propose to maintain a cache for the hardest questions, which cannot be solved by rollouts. After the cache is full, we will add a sft stage during the rl process. Empircal results show that it can outperfrom sft then rl method.", "questions": "Do you think this fine-tuning approach is primarily beneficial only when:\n- The task or dataset is particularly difficult (e.g., requiring complex multi-step reasoning), or\n- The base model is relatively weak and struggles with such questions?\n\nIn other words, would the strongest model (e.g., GPT-4-level) still benefit from this approach on datasets?\n\n\nWhat will be the results if use rl w/ sft loss instead of sft in ReLIFT?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to read and well-organized.\n- The idea is simple but effective.\n- Empircal results show the effectiveness of the method.", "weaknesses": "- What is the value of this buffer size, $M$? The paper is also missing an analysis experiment (or ablation study) for this. This is important because Figure 4 indicates that the final results seem to be highly dependent on this parameter. And it seems related to the distribution of the datasets.\n\n- The final experimental results show limited improvement over previous methods, and the performance is inconsistent across different datasets. It will be better to show the training curve when compare to luffy rather than the final results since rl is not very stable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)