Getting Your LLMs Ready for Reinforcement Learning with Lightweight SFT
Authors: Xinran Li
Guangda Huzhang
Siqi Shen
Qing-Guo Chen
Zhao Xu
Weihua Luo
Kaifu Zhang
Jun Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
SFT
Post-train
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning (RL) has emerged as a powerful post-training paradigm for large language models (LLMs), yet its effectiveness varies significantly across base models. While incorporating a pre-RL supervised fine-tuning (SFT) phase can enhance RL training, key questions remain: how long should the SFT cold-start phase last, and is the SFT objective truly aligned with the requirements for effective RL preparation?
In our analysis of cold-start dynamics, we uncover a key limitation: the SFT checkpoint with the highest evaluation performance often fails to maximize RL potential due to distributional forgetting—a phenomenon where the model drifts excessively away from the base model’s distribution even before traditional overfitting occurs. We identify diversity metrics, such as the entropy and self-BLEU, as more reliable early-stopping criteria than the standard performance-based checkpoint selection. Our findings show that SFT checkpoints with peak diversity consistently lead to superior post-RL results. Building on these insights, we introduce Adaptive Early-Stop Loss (AESL), a lightweight and dynamic cold-start method that balances the acquisition of new patterns with the preservation of the base model's distribution. AESL operates at both the token and subsequence levels, providing finer-grained control over the cold-start process. Experimental results on mathematical reasoning benchmarks demonstrate that diversity-based early stopping surpasses traditional performance-based SFT, while AESL further enhances RL preparation. By steering LLMs toward better initialization points for RL, AESL consistently achieves superior final performance compared to existing SFT and cold-start strategies. The
code is publicly available at \url{https://github.com/LXXXXR/AESL}.

Published as a conference paper at ICLR 2026
GETTING YOUR LLMs READY FOR REINFORCEMENT LEARNING WITH LIGHTWEIGHT SFT
Xinran Li
Guangda Huzhang"
Siqi Shen
Qing-Guo Chen'
Zhao Xu²
Weihua Luo² Kaifu Zhang² Jun Zhang
'The Hong Kong University of Science and Technology
2 Allibaba International Digital Commerce
Fujjan Key Laboratory of Sensing and Computing for Smart Cities, School of Informatics, Xiamen University, China
xinran. li@connect .st hk, jy com, siqishen@xmu edu, en, {qingguo. cqg, changgong X2, weihua luowh, kaifu. (aluuukkf@@aiiilla com, ee ee mejzzangeust...
ABSTRACT
Reinforcement learning (RL) has emerged a powerful post-training paradigm for large language models (LLMs), yet its effectiveness varies significantly across base models. While incorporating pre-RL supervised fine-tuning (SFT) phase can enhance RL training, key questions remain: how long should the SFT cold-start phase last, and is the SFT objective truly aligned with the requirements for effective RL preparation? In our analysis of cold-start dynamics, we uncover a key limitation: the SFT checkpoint with the highest evaluation performance often fails to maximize RL potential due to distributional forgetting phenomenon where the model drifts excessively away from the base model's distribution even before traditional overfitting occurs. We identify diversity metrics, such the entropy and self-BLEU, as more reliable early-stopping criteria than the standard performance- baseed checkpoint selection. Our findings show that SFT checkpoints with peak diversity consistently lead to superior post-RL. results. Building on these insights, we introduce Adaptive Early-Stop Loss (AESL), a lightweight and dynamic cold-start method that balances the acquisition of new patterns with the preservation of the base model's distribution. AESL operates at both the token and subsequence levels, providing finer-grained control over the cold-start process. Experimental results on mathematical reasoning benchmarks demon- strate that diversity-based early stopping surpasses traditional performance- based SFT, while AESL further enhances RL preparation. By steering LLMs toward better initialization points for RL, AESL consistently achieves superior final performance compared to existing SFT and cold-start strategies. The code is publicly available at httptt:://ggiit
com/ LXXXXR/AEE AESLL
INTRODUCTION
Large Language Models (LLMs) (Tam et al. 2023 Achiam et al. 2023 Grattafiori et 2024
Yang et al.) 2025) have demonstrated remarkable capabilities across various domains, includ ing natural language processing, question answering, and planning tasks Mirzadeh al. (2025): Valmeekam et al. (2023); [Huan & Chang (002)). Their abundant prior knowledge enables them fields such as mathematical reasoning and code generation (Ahn et al. 2024 Diang et al., 2024). The foundational capabilities and exten- sive knowledge embedded in LLMs unlock a wide range of downstream applications, making them indispensable tools for solving sophisticated real-world problems.
to tackle previously unattainable challenges, particularly
While inherently powerful, LLMs often require post-training to better align their capabilities with downstream tasks. Post-training refines base models by strengthening specific abilities, address- ing gaps in reasoning patterns, domain expertise, or human preferences The et al.] (2025); Kumar
Corrresponding author.

---

Published as a conference paper at ICLR 2026
al. 2025) Shen al. (2023). Among various post-training paradigms, reinforcement learning (RL)
Hu 2025; Shao et al. 2024, Zheng et al. 2025 Guo et al. 2025) has recently emerged particllrlly promising approach, offering significant performance improvements for LLMs across downstream tasks. RL's advantages are compelling: it eliminates the need of large- scale curated demonstrations and holds the potential achieve superhuman performance in tasks where well-defined reward signals are available to guide optimization.
Barto 2021
However, despite its promise, RL's theoretical benefits often fail fully materialize in practice. RL algorithms are notoriously sample-inefficient, requiring extensive exploration to learn complex reasoning patterns from scratch. Moreover, RL's effectiveness heavily depends on the quality and characteristics of the base sodellmmdeels with limited reasoning capabilities may struggle to discover appropriate reasoning patterns and fail to benefit from RI. training. To address these limitations, many post-training pipelines incorporate supervised fine-tuning (SFT) [ambert et al. (2024): Yang et al. (2025); Guo al. (2025) cold-start phase that enables models to aequire essential reasoning patterns, such as long chain-of-thougee (CoT) (Wei et al., 20222 reasoning, before RL training begins. This approach enhances sample efficiency and overall performance by providing stronger foundation for subsequent RL optimization.
Despite the widespread adoption of cold-start phases, fundamental questions remain unanswered. How long should the cold-start phase last to optimally prepare base LLMs for subsequent RL training? More critically, is the standard SFT objective of demonstration imitation well aligned with the goal of RL. preparation? These open questions highlight the need for deeper understanding of effective cold-start design to optimize subsequent RL performance.
In this paper, we address these critical questions by investigating cold-start dynamics and proposing methods to optimize their effectiveness of cold-start SFT for RL preparation. Through an analysis of post-RL performance across various cold-start checkpoints. we reveals misalignment between standard cold-start objectives and RL preparation goals: the best-performing checkpoint after SFT does not necessarily correspond the optimal starting point for RL due distribution forgetting that occurs before traditional overfitting. To address this issue, we introduce improved criteria and methods for cold-start design, ensuring better alignment with RL training requirements. Specifically, our contributions are follows:
We demonstrate that diversity metrics, such as entropy and self-BLEU, serve as superior early-stopping criteria compared evaluation performance. Selecting checkpoints based on diversity consistently yields better RL performance than relying solely on evaluation metrics.
Building on these insights, we propose Adaptive Early-Stop Loss (AESL), a novel cold-start method that dynamically balances new pattern acquisition with base distribution preservation both token and subsequence levels, providing fine-grained control over the preparation process.
Through extensive experiments across different base models (Qwen2.5-7B Instruct (Yang et al., 2024a) and Qwen2.5-Mathe demonstrate AESL's superior performance, establishing it as an effective solution for preparing LLMs for successful RI. training.
(Yang et al.) 2024b)) and various cold-start data settings,
PRELIMINARIES
The goal of the post-training phase for LLMs is to refine base models by enhancing their knowledge and capabilities in specific domains, such as instruction following and reasoning (Kumar et al.) 2025). Modern LLM post-training predominantly employs two complementary paradigms: SFT and RL. SFT relies on high-quality demonstration datasets to align model outputs with desired responses, while RL requires only prompt sets and reward signals from verifiers to optimize response generation through policy learning. The RL paradigm has gained significant attention due to its potential to achieve super-human intelligence.
2.1 RL FOR LLM POST-NRAINING
In the reinforcement learning with verifiable reward (RLVR) framework (Lambert al. 2024). the objective is to optimize the policy (i.c., the LLM) T lq, $ze) to maximize the expected reward from verifier R(se, q), where 8t represents the current token output, 8gt represent the first /
2

---

Published as a conference paper at ICLR 2026
generated token and denotes the input question. In this work, we adopt the widely-used GRPO algorithm (Shao et al. 2024) for RL post-training, which optimizes the following objective:
(0) E. Pa (ole)
To To(sfla 22 Toma (8) st(z)
To(s//g,822
min
clip
At -BDKL [et
Tea (s) 8/.)
where PRL is the question set used for RL, is the group size in GRPO, and denotes a complete rollout sequence. The term Tew represents the policy before the update, while Tee denotes the reference policy. The advantage estimate is normalized within the group as A r'-mean(r) where r R(st, q) is the verifier that provides reward signals and r is the reward for the i-th rollout.
ad(r)
2.2 RL COLD-START PHASE VIA SFT
Although RL is highly effective at optimizing task-specific performance, its success strongly influ- enced by the capabilities of the base model (as discussed in Section 3.1 Additionally, RL training sample-inefficien making it challenging to learn complex reasoning patterns, such as long-Coo from scratch. To address these challenges, many post-training pipelines (Guo al. 2025 Lambert et al.] 2024) include cold-start phase using SFT. This phase injects essential reasoning patterns, such long-Co'T, into the base model using small amount of demonstration data, enabling RL. to achieve better sample efficiency and performance. Typically, the cold-start phase employs the cross-entropy (CE) loss as the training objective:
(i) = -Bg.s" Dam Tog s<t))
(2)
where DSFT the demonstration dataset. The loss aligns the model's predictions with the provided demonstrations, enabling it to learn desired reasoning patterns before progressing to RL-based post-training.
METHODOLOGY
In this section, examine the post-training process holistically and propose enhanced criterion for early stopping and an improved loss function for the cold-start phase. These enhancements aim improve final performance after subsequent RL training. First, in Section 3.1 analyze how the cold-start phase affects the effectiveness of L.M post-training. Our analysis reveals that the best-performing checkpoint after cold-start, measured by evaluation metrics, often fails to prepare the model optimally for RL training due to distribution forgetting from the base models. We empirically demonstrate that diversity turning points during cold-start training correspond to superior RL potential, motivating diversity-basee early stopping Building on this insight, Section 3.22 introduces our Adaptive Early-Stop Loss (AESL) for the cold-start phase, which adaptively balances preservation of the original distribution with adaptation to new demonstration patterns on token-by-token basis. Together, these contributions offer improved flexibility and superior performance compared to traditional CE-based approaches.
more effective criterion.
3.1 MOTIVATION: UNDERSTANDING THE COLD-START PHASE DYNAMICS
The cold-start phase is crucial for steering base models toward promising starting point for RL training, particularly when models lack domain-specific knowledge or reasoning capabilities (e.g., long-CoT patterns). To investigate its role, first evaluate the RL performance of two base mod- els: Qwen2.5-Math-77 (Yang et al., 2024b) and Qwen2.5-78 -BIInstruct (Yang et al. 2024a), and compared their post-RL performance.

---

Published as a conference paper at ICLR 2026
Before RL
Performance
40
38.07 36.00
38.07
40.12
Performance
38
30
36
20
34
10
30
0
@wetn255Mathh76
Qwen2.5-78 5-78-instruct
0
100
Cold-start Steps
200
300
400
(a) Qwen2.5-Math achieves significant direct per- (b) Performance before and after RL training across formance gains through RL, while Qwen2.5-7B- different cold-start training steps. The best post-cold- Instruct requires cold-start preparation for effective start checkpoint does not correspond to optimal post- subsequent RL training.
RL performance.
Figure 1: Motivating observations showing the necessity and optimization challenges of cold-start training. Performance represents averages across multiple mathematical reasoning benchmarks (de- tailed Section B. Complete results are provided in Section C.T
0.825
0.800
Entropy
0.775
0.750
0.725
0.700
0.2
0.675
0.1
0,650
100
100
200
Cold-start Steps
Steps
(a) Entropy dynamics during cold-start training. (b) Self-BLEU dynamics during cold-start training.
Higher values indicate increased response diversity. Lower values indicate increased response diversity.
Figure 2: Diversity measurements throughout the cold-start training process corresponding to Fig- ure 1b revealing the relationship between diversity measurement and RL potential.
The Necessity of Cold-Start As shown in Figure la Qwen2.5-Math-7B achieves significant per- formance gains directly after RL training. In contrast, Fren2.55-B-Inntrrue shows minimal im- provement under direct RL training. This disparity is further reflected in slower growth of average response length (detailed Section C.T) underscoring the need for cold-start SFT phase ject Ong-CooT patterns into the base model. Following established conventions, we implement lightweight cold-start phase (using less than 1/10 of the data volume of the subsequent RI phase) and observe substantially enhanced overall performance gains. This confirms that strategic cold-start SFT can effectively steer base models toward better RL starting points. However, this also raises critical question: is optimizing for the best cold-start evaluation performance aligned with achieving the optimal post-RL performance?
Misalignment Between Cold-start and RL Objectives Our investigation reveals that the objec- tive of the cold-start phase does necessarily align with the purpose to prepare base model for subsequent RL training, particularly when demonstration data is limited. We examine this by vary- ing training budgets for standard cold-start phases and subsequently applying RL training. Figure presents the performance for both post-cold-start and post-RL. models: initial performance decay followed by recovery during cold-start training, representing a shift-and-readaatttionn process. Cru- cially, we observe that while evaluation performance continues improving during the readaptation phase, corresponding post-RL performance begins declining. This indicates that the RL potential deteriorates before overfitting to the cold-start dataset, suggesting fundamental misalignment be- tween cold-start objectives (steering models toward better RL starting points) and standard CE loss objectives (maximizing demonstration dataset imitation).
4

---

Published as a conference paper at ICLR 2026
Diversity Criteria for Cold-start Early-stopping: To understand this performance degrada- tion phenomenon, we examine diversity measurements throughout the cold-start training process (Figure 2). This pattern reflects important distribution dynamics during cold-start training. In the early stages, the model balances acquiring new reasoning patterns from the cold-start dataset with retaining knowledge from its original distribution, as illustrated in Figure [3 However, as training progresses, the model begins to over-adapt to the new dataset, leading to distribution forgetting and reduced RL potential. The diversity peak represents an optimal balance point-a "sweet spot" (step 100 in this example) where the model successfully acquires new reasoning patterns while sufficiently retaining distribution from base model for effective RL training. The enhanced entropy at this point likely stems from the model maintaining dual distribution characteristics from both the original base model and the new dataset patterns. Our analysis yields two key insights:
From an analytical perspective, deterioration of RL potential occurs before SFT overfitting, re- vealing fundamental misalignment between cold-start objectives (preparing models for RI.) and CE loss objectives (maximizing demonstration imitation).
From a practical perspective, selecting the best-performing post-cold-start checkpoint may be suboptimal for subsequent RL training. Effective cold-start requires balancing new pattern quisition with original distribution preservation, with diversity measurements serving as valuable indicators of this trade-off.
3.2 ADAPTIVE EARLY-STOPPING LOSS
Based on our experimental findings, reframe the cold-start objective from complete demonstration imita- tion achieving optimal trade-off between learning new patterns and preserving distribution from the original base model, as depicted in Figure The simplest enhancement to traditional cold-start approaches implementing diversity-base early stopping. Instead of selecting the best-performing checkpoint after SFT cold-start, select the checkpoint corresponding to the diversity measurement turning point, which indicates the onset of base model distribution forgetting. Based on observations in Figure [bb this approach should improve subsequent RL post-training performance, which we validate comprehensively in Section
InPiatication
Ovemitiing Forgetting
Responso
Datasst
Content2
Figure 3: Conceptual illustration of the cold-start training process. The model distribution gradually shifts from the base model toward the cold-start dataset distribution. Overly excessive training leads to overfitting and forgetting of the base model's original distribution, potentially reducing its subsequent RL potential. This highlights the need to balance learning new reasoning patterns from the dataset with pre- serving the base model's 's original distribution. Our proposed AESL adaptively addresses this trade-off based on context.
While vanilla early stopping proves effective, lacks flexibility by applying uniform stopping crite- ria across the entire dataset. As illustrated in Figure B overfitting and distribution forgetting occur different rates for different tokens and contexts due to varying "distance" between the base model and cold-start dataset distributions. This heterogeneity necessitates more flexible mechanism for managing the inseeratiion-ddapaane
trade-off. To address these limitations, we propose the Adap- tive Early-Stop Stop Loss (AESL), which provides fine- fine-grained trade-off control at the token and subse- quence level rather than dataset-level early stopping. The loss function is weighted version of the original CE loss:
Aade-stop (0) = En [p(g, st TO) log To lg. s<c)l,
(3)
where the adaptive weighting is defined as:
y(8 q, 8c1))
p(g, 8 )o) = 1 softmax
(4)
-scaling
14 El= log (o(5)) 9g ses)
exp(x:)
with y representing the output logits before softmax normalization, softmax[zz, = Σ, expla,) and
lscating being hyperparameter.
5

---

Published as a conference paper at ICLR 2026
Table Post-training performance (?) with different cold-start methods. For AIME and AMC, we report avg @ 64 performance due to smaller test set sizes. For other benchmarks, pass 1 is reported. Avg. denotes the macro-average across benchmarks.
AIME24
AIME25
AMC23
MATH.
Min.
Olym.
Avg.
Qwen2.5-7B-Instruct
Base
11.93
8.39
53.16
78.2
36.76
40.0
38.07
+RL
13.7
5.73
54.45
76.4
38.24
40.0
38.09
+SFT (CE)
12.29
15.31
44.22
74.2
39.34
35.26
36.77
+RL
15.99
15.52
53.36
78.4
37.13
40.74
40.19
+SFT (CE-ES)
9.74
12.5
43.32
71.2
31.25
31.7
33.28
+RL
18.23
15.94
54.37
80.4
37.5
42.81
41.54
+SFT (GEM)
13.02
14.37
45,47
74.2
34.56
33,93
35.93
+RL
16.2
15.83
53.83
80.6
36.4
40.3
40.53
+FT (PSFT)
13.33
14.69
44.73
73.6
36.03
34.52
36.15
+RL
15.73
16.15
52.15
80.6
40.07
39.11
40.64
+FT (AESL)
11.41
13.65
44.26
73.8
36.4
32.44
35.33
+RL
18.18
16.88
56.48
81.8
37.13
43.11
42.26
Qwen2.55 5-athh-5--
Base
16.77
9.17
55.12
64.4
32.72
34.52
35.45
+RL
22.92
13.33
63.98
82.8
39.34
45.63
44.67
+SFT (CE-(ES))
20.26
19.11
59.61
83.6
40.07
41.93
44.1
+RL
24.11
21.3
69.38
86.6
41.18
49.04
48.6
+SFT (GEM)
17.24
20.83
58.44
81.4
41.18
43.85
43.82
+RL
26.15
21.51
68.24
87.4
41.54
51.7
49.42
SFT (PSFT)
19.32
21.46
60.08
82.4
41.18
42.37
44.47
+RL.
24.27
23.12
65.74
87.0
40.44
47.56
48.02
+SFT (AESL)
17.29
18.75
58.16
83.6
34.93
40.3
42.17
+RL
25.0
21.98
71.25
87.0
42.28
52.74
50.04
MATH. denotes MATH-500, Min. denotes Minerva Math, and Olym. denotes OlympiadBench. Gray shading indicates methods for easier readability. Best performance is bolded, and second-best is underlined.
For Qwen2.5-Maal 1-a2255-Mhh-77- CE and CE-ES checkpoints coincide, so results are reported together.
At the token level, the weighting function gradually reduces the loss contribution for tokens where the ground truth already corresponds to high probability under the current policy. This mecha- nism slows learning when the model already assigns high probabilities to correct tokens, thereby preserving original knowledge while still allowing adaptation to new patterns. AESL also incorpo- rates subsequence-
considerations, recognizing that effective RL preparation requires the abil- ity to generate diverse and plausible reasoning paths. Since sequence diversity measured by entropy H H 44) can be decomposed as: H (<+lg)) Σ.,,, (s:
(q) (x) + ((x=-1lg) (detailed in Section A). tokens following high probabbobyyylityy prefixes contribute more to overall diversity. Build- ing this insight, we scale the token-level early stopping with the average log- probabbbyil of the prefix context (denominator in Equation (4)). This encourages the model to maintain base distri- bution when the prefix already aligns closely with the dataset distribution, balancing adaptation to demonstrations with preservation of distribution of the base models.
EXPERIMENTS
this section, we integrate the Adaptive Early-Stop Loss (AESL) cold-start method into standard post-training pipelines and evaluate its effectiveness in preparing LLMs for RL post-training. We first compare AESL against baseline cold-start methods, demonstrating its superior performance across multiple base models. We then analyze the impact of dataset configurations and hyperparam- eters, validating the robustness of AESL diverse experimental. More experiments are included in Section

---

Published as a conference paper at ICLR 2026
4.1 SETUP
Base Models We evaluate approach using two distinct base models: Qwen2.5-7B-Innse In-strucctt and Qwen2.5-Math-78 Both models do not present the ability to reason through long CoT pattern (as demonstrated in Table 13), making the cold-start phase essential for injecting long COT reasoning patterns. These models represent different starting points for RL post-training: Qwen2.5-7B-Instme excels instruction following but may not be optimally positioned for RLVR (as demonstrated in Figure Ia), while Swen2.5-Math-777 having been pre-trained on mathematical datasets, generates more diverse responses and serves a stronger foundation for RLVR. While Owen2.5-Matte Math-7B requires less intervention during cold-start, we include it to gauge AESL's effectiveness in different initial conditions.
Dataset and Algorithm Following established practices (Fu et al. 2025 Zhang et al. 2025), we employ the OpenrI-Maa -Math-46k- 99166-88121008 Hugggng Face ) Yan et al. 2025) dataset for post-training This dataset comprises 46k long COT demonstrations generated by the DeepSeek-RI model. For SFT cold-start, we uniformly subsample 3k examples, ensuring each question corresponds to one correct demonstration verified using Math- Verify For RL post-training, utilize the complete 46k question set without demonstrations, employing the GRPO algorithm (Shao et al. 2024).
Math-Verify!
Baselines We compare AESL against five baseline approaches: (i) Direct RL: applying RL directly to base models without cold-start preparation; (ii) CE: conducting SFT cold-start with CE loss and selecting the best-performing checkpoint for subsequent RL; (iii) CE-ES (CE with early-stopping): conducting SFT cold-start with CE loss and selecting the checkpoint at the diversity turning point for subsequent RL (motivated by findings Section 3.1]; (iv) GEM (Li et al. 2025); and (v) PSFT (Zhu et al. 2025).
Further details about experimental setups and implementation specifies are provided in Section
4.2 RESULTS
Main Results The comparative performance of AESL and baseline methods is summarized in Table [ Overall, AESL achieves supe- rior performance for both base models, con- sistently outperforming baselines in subsequent RL training. This highlights AESL's ability to effectively steer LLMs into better starting point for RLVR. As discussed in Section 3.2 A. this performance boost by care- fully balancing the need to the base model's core distribution while simultaneously learning new reasoning patterns. As shown in Table 2 AESL produces the highest entropy and lowest self- BLEU scores, indicating increased diversity in model outputs after cold-start. This aligns with the findings in Section [.1] where greater diversity translates to better RL potential. Moreover, BLEU scores between pre- and post-cold-ste with the base model's original output distribution (0.140 vs. 0.135 for CE). This indicates that AESL-trained models retain stronger base knowledge, enabling them to more effectively sample from the base distribution during RL training. Importantly, the CE-ES method also outperforms CE in providing a better starting point for RLVR, despite having inferior performance immediately after the cold-start phase. This observation, coupled with AESL's success, supports the argument in Section 3.2 that the cold-start phase should not solely focus on intimating the demonstration dataset or maximizing evaluation scores but should also consider preserving the base model's existing ca- pabilities, particularly when the dataset size for cold-start is limited. This is further supported by the increased entropy and reduced Belf-BLEU scores achieved by AESL, as shown in Table Extended discussion is included in Section
Table 2: Response diversity measurement after cold-start with Qwen2.5-7B-Instee Higher entropy and lower self-BLEU indicate greater diversity.
Method Entropy(1)
Self-BLEU())
CE
0.326
0.710
CE-ES
0.53
0.696
AESL 0.553
0.694
outputs indicate that AESL maintains a closer alignment
Results with Different Cold-start Datasets To assess the robustness of AESL, we evaluate its performance under varying cold-start dataset sizes (1k, 3k, and 6k samples) and question

---

Published as a conference paper at ICLR 2026
Table 3: Performance(1)) comparison across different cold-start dataset sizes using Qwen2.5- IB-Inntruct the base model.
AIME24
AIME25
AMC23
MATH.
Min.
Olym.
Avg.
Base
11.93
8.39
53.16
78.2
36.76
40.0
38.07
+RL.
13.7
5.73
54.45
76.4
38.24
40.0
38.09
Ik cold-start dataset
+SFT (CE-(ES)
11.67
13.44
44.92
72.8
37.5
33.33
35.61
+RL
16.2
14.74
52.62
80.8
39.71
40.74
40.8
SFT (AESL)
12.4
12.24
46.13
74.0
37.13
34.52
36.07
+RL
17.14
15.36
54.02
76.8
41.18
42.37
41.14
6k cold-start dataset
+SFT (CE)
12.92
15.99
45.74
75.0
36.4
36.44
37.08
+RL
14.32
16.04
54.26
77.4
39.71
40.44
40.36
SFT (CE-ES)
11.61
14.22
42.97
71.6
34.56
32.15
34.52
+RL
16.56
16.72
53.79
80.2
41.54
42.37
41.86
+SFT (AESL)
12.55
15.73
44.84
73.6
34.19
33.63
35.76
+RL
16.67
16.82
56.41
81.6
43.01
42.22
42.79
For Ik dataset size, CE and CE-ES checkpoints coincide, results reported together.
Table 4: Performance(t) comparison across different cold-start dataset difficulty splits using Qwen2.5-7B-Instree as the base model.
AIME24
AIME25
AMC23
MATH.
Min.
Olym.
Avg.
Base
11.93
8.39
53.16 54.45
78.2
36.76
40.0
38,07
+RL
13.7
5.73
76.4
38.24
40.0
38.09
3k dataset with hard questions (filtered with the base model)
+SFT (CE)
13.49
14.95
46.48
74.0
36.4
36.15
36.91
+RL
16.93
16.35
53.91
79.2
38.24
39.26
40.65
+SFT (CE-ES)
12.81
13.91
46.48
74.0
36.76
34.22
36.36
+RL
17.03
15.78
54.14
79.4
38.24
40.0
40.77
+SFT (AESL)
13.7
16.04
46.56
75.2
33.46
33.48
36.41
+RL
17.55
17.03
54.14
80.4
40.07
40.89
41.68
3k dataset with simple questions (filtered with the base model)
+SFT (CE)
12.19
13.33
44.69
72.4
35.66
34.81
35.51
+RL
14.43
15.42
53.91
80.6
40.81
40.74
40.99
SFT (CE-ES)
11.15
12.03
44.65
71.4
33.46
33.19
34.31
+RL
14.9
15.78
51.88
77.0
37.13
39.41
39.35
+FT (AESL)
10.78
12.03
45.47
73.2
33.09
33.48
34.68
+RL
16.77
16.46
55.35
80.8
38.97
45.04
42.23
difficulty levels (simple and hard splits). The results, shown in Table 3 and Table 4 demonstrate that AESL consistently outperforms traditional CE-based cold-start methods across all dataset sizes and difficulty levels. These findings highlight AESL's versatility and effectiveness in diverse post-training scenarios, even under varying data availability constraints. Additional discussions are provided in Section
Results with Different Model Families To further validate AESL's effectiveness, we extended the method to post-train the 1-B--.trtttt model (Grattafiori et al.) 2024). The results, detailed in Table confirm AESL's ability enhance post-training performance across different base models. This demonstrates that the proposed approach is not only effective for Qwen-based models also generalizes well to other architectures.
8

---

Published as a conference paper at ICLR 2026
Table 5: Post-training performance (1) with different cold-start methods using Llama-3.1-8B-
Instruct as the base model.
AIME24
AIME25
AMC23
MATH.
Min.
Olym.
Avg.
Base
5.26
0.68
19.14
45.6
23.16 16
16.89
18.46
+RL
4.22
1.09
25.78
50.6
27.57
15.41
20.78
+SFT (CE)
1.82
2.66
24.57
51.6
25.0
18.81
20.74
+RL
4.
2.45
24.57
58.0
24.26
21.33
22.46
+SFT (CE-ES)
1.88
0.68
22.7
50.6
22.79
17.78
19.4
+RL
5,36
1.93
28.12
59.8
29.04
22.22
24.41
+STT (AESL)
2.6
1.61
20.43
47.8
20.22
15.85
18.08
+RL.
6.56
3.8
31.21
60.4
28.68
22.37
25.5
Table 6: Ablation studies for AESL cold-start using Qwen2.5-7B-Insrrutt the base model.
Results indicate post-RI. performance.
AIME24
AIME25
AMC23
MATH.
Min.
Olym.
Avg.
Base
13.7
5.73
54.45
76.4
38.24
40.0
38.09
AESL
18.18
16.88
56.48
81.8
37.13
43.11
42.26
ASEL w/ const, d.
16.51
15.0
52.62
78.2
40.07
39.7
40.35
ASEL w/ rank
14.58
14.95
54.18
78.6
39.34
41.93
40.6
ASEL w/ linear w.
16.09
15.62
52.54
79.6
41.18
40.74
40.96
Ablations and Hyperparameter Analysis We conduct ablation studies to evaluate the impact of key components AESL, with the results presented in Table 66 Specifically, compare the full AESL method with four ablations: 1) ASEL w/ const. d.: In this variant, the denominator in Equation
replaced with constant, removing the dynamic adjustment based on prefix predic- tion confidence. 2) ASEL w/ rank: In this variant, we mask the token loss when the ground truth token has the highest predicted probability (Top-1), effectively ignoring such tokens. 3) ASEL w/ linear w:: In this version, the token-level SFT loss is weighted inversely proportional to the prefix prediction confidence. The results indicate performance drops across all ablations, underscoring the importance of explicitly accounting for prefix prediction confidence, implemented in AESL. This finding aligns with the motivation discussed in Section A demonstrating that the absence of prefix-aware adjustments in the ablations leads to less effective cold-start.
We then examine the impact of the temperature scaling hyperparameter tscaling. which controls the balance between learning new reasoning patterns and preserving distribution from base model. revealss that extreme values degrade performance: excessively small (scaing prevents effective pattern learning, while excessively large values cause base distribution forgetting and reduced diversity. However, within the optimal range (3-5), AESL consistently delivers performance improvements over all baseline methods, demonstrating robust performance across reasonable hyperparameter settings.
RELATED WORKS
The pretraining stage of LLMs primarily focuses on constructing general foundation of knowledge and capabilities, whereas the post-training stage adapts these models to specific tasks and require- ments (Tie et al. 2025). Two widely used methodologies for post- post-training are: SFT Guo et al. (2023): Lambert et al. (224)) and RL (Shao et al., 2024] Hu 2025 Yu et al.] 2025 Zheng et al. 2025 Hu et al. 2025). which are employed to inject domain knowledge or enhance specific abilities in L.Mss. Recent studies have explored the strengths and limitations of these two paradigms. For instance, Sun et al. 20025) and MMennighoff et al. (2025) analyzed SFT datasets, highlighting the importance of data curation and scaling in improving model performance. In contrast, Yue et al. (002)) argued that RLVR does not introduce new reasoning patterns but instead constrained by the base model's underlying capabilities. Bridging these observations, Chu et al.) (2025) demonstrated that while RL improves generalization, SFT tends to memorize training data but helps stabilize output formats for L.MMs.

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameter analysis for Escaling in AESL using Qwen2.5-7B-Instrruc as the base model.
AIME24
AIME25
AMC23
MATH.
Min.
Olym.
Avg.
Base
11.93
8.39
53.16
78.2
36.76
40.0
38.07
+RL
13.7
5.73
54.45
76.4
38.24
40.0
38.09
+SFT (Escaling
9.43
12.4
43.05
70.6
31.99
32.44
33.32
+RL
17.45
15.21
52.7
79.8
38.24
43.11
41.09
SFT (Eacaling 3)
12.34
14.27
44.14
72.8
36.76
33.63
35,66
+RL
18.23
15.57
55.78
80.4
38.97
44.89
42.31
SFT (Ascaling 5)
11.41
13.65
44.26
73.38
36.4
32.44
35.33
+RL
12.18
16.88
56.48
81.8
37.13
43.11
42.26
+SFT (Ascaling
13.28
13.54
45.98
72.8
36.03
31.85
35.58
+RL
18.44
14.84
55.16
79.4
37.87
40.89
41.1
Building on this foundational understanding SFT and has proposed methods to combine the two paradigms more effectively. One promising avenue involves enhancing SFT through evolving data strategies, where data progressively guides the base model during training. For example, Light-R1 (Wen al. 2025) employed curriculum learning to gradually increase the difficulty of SFT question sets, thereby improving the learning landscape. Similarly, REFT (Luong et al.] 2024) leveraged multiple sampling of model outputs, selecting the best trace for SFT instead of relying on single demonstration paths. Another line of research integrates SFT with RL in combined training stage, allowing demonstrations
post-training. recent research
better guide RL
exploration. LUFFY Yan et al. (2025) approached this from a data perspective, mixing off-policy demonstrations with on on-policy rollouts to improve RL performance. SRFT (Fu al. (0225) and CHORD Zhang et al. 2025) tackled the problem through modified objectives, reframing SFT as an auxiliary task within the RI. framework.
Our work, alongside GEM (ii et al. 2025) and the concurrent work PSFT (Zhu et al. 2025 belongs third category approaches that aim to refine the SFT stage by shifting the training ob- jective beyond simple dataset imitation. GEM emphasizes promoting diversity, while PSFT focuses on out-of-distribution generalization using abundant SFT datasets. contrast, our proposed AESL targets lightweight cold-start SFT setting (using less than 1/10th the data size of RL training). AESL addresses the challenge of preparing the base model for subsequent RLVR by striking a better balance between learning reasoning patterns and preserving the base model's prior knowledge.
CONCLUSIONS
In this work, addressed the critical challenge of preparing LLMs for RL training through effec- tive cold-start SFT. Our analysis uncovered a fundamental misalignment between traditional SFT objectives and RL preparation goals: the best-performing cold-start checkpoint, evaluation performance, fails to maximize RL potential. This failure arises due to distribution for- getting from the base models, which occurs before traditional overfitting. We demonstrated that diversity metrics, such as entropy and self self-BLEU, are more reliable early-stopping criteria, with peak diversity checkpoints consistently yielding superior post-RL results. Building on this insight, we introduced Adaptive Early-Stop Loss (AESL), novel cold-start method that dynamically bal- ances new pattern acquisition with the preservation of the base model's distribution both the token and subsequence levels. Experimental results across mathematical reasoning benchmarks demon- strate that AESL consistently outperforms traditional cold-start methods and competitive baselines, achieving superior final performance in the lightweight SFT setting.
measured by
While our work addresses cold-start optimization from an objective perspective, several important aspects warrant further investigation. From evaluation standpoint, developing methods to predict the upper bound subsequent RL training could enable more informed cold-start design decisions. From data curation perspective, understanding how optimally select and structure cold-start datasets to maximize RL preparation effectiveness remains an open challenge. These directions represent promising avenues for future research in LLM post-training.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide comprehensive details on prompting strategies, datasets, hyperparameters used for train- ing and evaluation, as well as the infrastructure of our experimental setups in Section B The code is publicly available at httttps:///ggiaeie
ACKNOWLEDGMENT
This work was supported by the Hong Kong Research Grants Council under the Areas of Ex- cellence scheme grant AoE/E-601/22-R and NSFC/RGC Collaborative Research Scheme grant CRS HKUST60//22. We thank the anonymous reviewers for their valuable feedback and sugges- tions.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. Technical report, OpenAl, 2023.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157 2024.
Andrew G Barto. Reinforcement learning: An introduction. by richard's sutton. SIAM Rev, 6(2):
423. 2021.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study foundation model post-training. arXiv preprint arXiv: 2025.
Yugian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Donghin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506. 19767, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. Technical report, Meta AI, 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang. Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wa Chang. Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL https:ts:
wtt/ /2066. 04178
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Jian Hu. Reinforcet+: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262 2025.
Jian Hu, Xibin Wu. Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An casy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405. 111433, 2024.
Jingcheng Hu. Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. reen-reasoner-zerr zero: An open source approach to scaling up reinforcement learning on the base model, arXiv preprint 2003:22503.24290. 2025.

---

Published as a conference paper at ICLR 2026
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212. 10403, 2022.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https:
//github. htt.://..oooonnggggaaa//eee/e/ x
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.005/3 2024.
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Lim post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21132, 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training, arXiv preprint arXiv:2411. 15124, 2024.
Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised fine-tuning of large language models. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang. Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. In Conference on Language Modeling (COLM), 2025.
Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Rea- soning with reinforced fine-tuning arXiv preprint (tt::2401.089661 2024.
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun MA, and Wenhu Chen. General-Reasone Advancing LLM reasoning across all domains. In The Thirty-ninth Anmual Conference Neu- ral Information Processing Systems, 2025. URL httph:t///// http:///pppeeeeeeeeeeeeee net. www.eeeef/rtriiiiee
pBFVo11880 18Xa
Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representa- fions, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto. sl: Simple test-time scaling. In Workshop on Reasoning and Planning for Large Language Models, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint arXiv:2309. 15025, 2023.
Yiyou Sun, Georgia Zhou, Hao Wang, Dacheng Li, Nouha Dziri, and Dawn Song. Climbing the lad- der of reasoning: What Ilms can-and still can't t-solve after sft? arXiv preprint 1rX1v 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Cran-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:23/2. /805, 2023.
Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models. arXiv e-prints, pp. arXiv-2503, 2025.
12

---

Published as a conference paper at ICLR 2026
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Informa- fion Processing Systems, 36:75993-76005, 2023.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxín Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforeement learning for Ilm reasoning. arXiv preprint urXiv:2506.01939, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv: 10460, 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off- ooflcpolcy guidance. arXiv preprint arXiv:2504. 14945, 2025.
Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang. Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5 5-math technical report: Toward mathematical expert model via self-improvement, arXiv preprint 2tti:2200.122222 2024b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 25f.::2550.144700 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint ar arXiv:2504.133 13837, 2025.
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508. /408, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao. Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507. 18071, 2025.
Wenhong Zhu, Ruobing Xie, Rui Wang, Xingwu Sun, Di Wang, and Pengfei Liu. Proximal super- vised tineituning. arXiv preprint 27Xiv:2508.17784, 2025.
13

---

Published as a conference paper at ICLR 2026
A DETAILS ON DIVERSITY ANALYSIS
Following previous work (Fu ct al. 2025). we formulate the LLM's token generation process as Markov Decision Process (MDP). The state 8t is defined as the concatenation of all tokens gener- ated so far, making the transition deterministic in this MDP. As discussed in Section 3.1 diversity serves indicator of the balance between learning new reasoning patterns and preserving base distribution. To analyze this, we examine the entropy at the sequence level and its relationship to token-level entropy, which corresponds to next-token prediction using cross-entropy (CE) loss:
H (s:lg)
Σ
n (st log logoo(ssllg (s+lg)
(5)
Σ n (8 ((()(()-)))((ss. log (81/822 (
(6)
Σ n ((s Σ 2((((1t-1)((s-110) log (
Σ (0) H (st-199)
(8)
In online RL, where the data distribution is induced by the policy æ, preserving token-level diversity equivalent to preserving sequence-level diversity. During the SFT cold-start stage, entropy under different contexts contributes differently to overall sequence diversity. Therefore, we leverage this insight adaptively adjust the balance between learning new reasoning patterns and preserving distribution from base model, based on prefix prediction accuracy.
B
DETAILS ON EXPERIMENTAL SETUPS
B.1 PROMPTING
Following established conventions Guo et al. 2025 we append the following context math problems to prompt CoT reasoning:
Please reason step by step, and put your final answer within \bboxe({).
Additionally, we enforce that the model starts its response with <think>> to ensure reasoning is explicitly triggered during both training and evaluation.
B.2 TRAINING
We use the OpenRHLF framework parameters listed in Table
Hu et al. 2024 to conduct all the experiments with hyper- 8 and Table For AESL- hyperparameter,
(scaling 5. For baseline GEM Li et al. 2025) and PSF 7hu al. 2025). use the respective official codebase with recommended hy hyperparameters. For RL reward function, we use the outcome based reward with for correct answers and for incorrect ones.
Given the maximum sequence length for Qwen2.5-Math-7B is 4096, we increase the RoPE theta from 10,000 40,000 and expand the window size 16,384 support longer outputs.
B.3 EVALUATION
In this work, we use the evaluation framework proposed by Liu et al. 2025). with test-time param- eters listed in Table
httass/44thbbb.co////hooo/11111/
14

---

Published as a conference paper at ICLR 2026
Table 8: Common hyperparameters used for AESL and baselines during SFT cold-start.
Hyperparameter
Value
Learning rate
5 X 10-6
Learning rate scheduler
cosine with minimal Ir
$1, .2 Learning rate warm up ratio Weight decay Batch size
0.9, 0.95
0.1
0.1
64
Number of epochs
10
Table 9: Common hyperparameters used for AESL and baselines during RL phase.
Hyperparameter
Value
Learning rate
1 X 10-6
Learning rate scheduler
cosine with minimal Ir
$1, 32 Learning rate warm up ratio
0.9, 0.95
0.1
KL. coefficient Batch size
0.001
128
Number of samples per prompt
Number episode
Temperature
1.0
Rollout cut-off length
8192
B.4 DATASET
Cold-start Phase For SFT cold-start, we subsample from the OpenrI-Math-4000 (Hugging
Face 2025) dataset. Specifically:
Experiments in Table I and Table We uniformly subsample 1k. 3k, and 6k splits from the verifiable questions, each paired with one corresponding demonstration. The same subsampled datasets are used to conduct SFT cold-start experiments with proposed AESL. and baseline methods.
Experiments in Table 4 We begin by uniformly subsampling 6k split from the original dataset. Next, the base model Jowen2.5-7B-Instruce is used to rank the questions by difficulty, de- termined by the reversed average accuracy of 8 rollouts per question. Using this ranking, construct two splits: (i) A hard split consisting of the top 3k most difficult questions (i.e., those with the lowest accuracy). (ii) A simple split, comprising the remaining questions. The SFT cold-start experiments with our proposed AESL and baseline methods are then conducted on these two splits.
Evaluation For evaluation, the datasets used are listed in Table 11 Given the large evaluation variance induced by the small sizes of the AIME24, AIME25, and AMC23 datasets, we report the avg@64 64 metric for these datasets and @ for others.
B.5 INFRASTRUCTURE
The experiments conducted using NVIDIA H100 Tensor Core GPUs.
15

---

Published as a conference paper at ICLR 2026
Table 10: Test-time parameters used for evaluation.
Hyperparameter
Value
Output length
8192
Temperature 0.6
Top-p 0.95
Table 11: Evaluation benchmarks.
Dataset
Number of Questions
AIME24
30
AIME252
30
AMC233
30
MATH-500
500
Minerva Math
272
OlympiadBench
675
https: https: httes://iaccaaaeeeerrraaatttttar
2
https:/ httpp///iggguboeeeee//////etttar
https: / http:////neiiilieeee/////eeettea
https:/ http:///hoottticcccod/deeeeeeett
https: http:///ccciicccccc/////tee/eeaa
C DETAILED EXPERIMENTAL RESULTS
C.I ELABORATED RESULTS OF THE MOTIVATION EXAMPLE
Detailed Evaluation Results We present the breakdown results from Figure across different
evaluation datasets in T2
Table 12: Post-training performance (f) with different cold-start training budgets with standard
CE loss.
AIME24
AIME25
AMC23
MATH.
Min.
Olym.
Avg.
Base
11.93
8.39
53.16
78.2
36.76
40.0
38.07
+RL
13.7
5.73
54.45
76.4
38.24
40.0
38.09
+SFT (100 step)
9.74
12.5
43.32
71.2
31.25
31.7
33.28
+RL
18.23
15.94
54.37
80.4
37.5
42.81
41.54
+SFT (200 step)
11.82
14.11
45.04
72.8
33.09
33.19
35.01
+RL
16.72
15.57
55.12
78.8
38.97
42.07
41.21
+FT (300 step)
12.76
15.1
44.96
72.4
34.56
33.63
35.57
+RL
18.12
15.99
53.24
80.2
36.76
40.0
40.72
+SFT (I epoch)
12.29
15.31
44.22
74.2
39.34
35.26
36.77
+RL
15.99
15.52
53.36
78.4
37.13
40.74
40.19
Response Length Analysis We provide the average response lengths before and after RI. for Swen2.5-7B-Instrue and Qwen2.5-Math-7B in Table B3
Overall, the average response length before RL for Qwen2.5-Math-77 is slightly longer than that of Wrren.5-7B-Innstreee Additionally, the post-RL growth in average length for Swve2.55-Mth--77 much higher. These results indicate that Qwen2.5-Math-77 demonstrates a superior ability to scale response length and discover Cong-CoT patterns autonomously. whereas Dwen2.5-7B-Inntmmn fails to do so through direct RL. This highlights the necessity of a pre-RL cold-start SFT phase for 1--n2.5-7B-Insrruu to inject long-CoT patterns for better RI. scaling.
16

---

Published as a conference paper at ICLR 2026
Table 13: Averaged response length before and after RL.
AIME24
ATME25
AMC23
MATH.
Min.
Olym.
Avg.
Qwen2.5-7BBInntruu
1224
1039
917
625
688
938
905
+RL.
1072
1061
1046
712
721
1180
965
Qwen2.5-Math-77
1487
1517
1016
808
691
1074
1099
+RL
2689
2317
1445
891
935
1550
1638
C.2 OTHER PERFORMANCE-BAAER EVALUATION METRICS
In addition to accuracy, as discussed in Figure Tb we evaluate the pass@8 metric during the cold- start phase assess suitability as an carly-stopping criterion. To ensure the robustness of the estimated pass @8 metric, we use 16 rollouts per question, which provides more reliable mea- sure of model performance by reducing the variance of the metric across questions. The results, shown in Figure 4 reveal that pass 8 follows a similar trend to evaluation accuracy, reflecting the characteristic
andrreadaptation process observed during cold-start training.
However, despite its alignment with evaluation accuracy, @ fails to reliably predict RL po- tential during the lightweight cold-start phase. This indicates that pass@8, @8, like accuracy, is an inadequate choice for guiding early stopping in this context, as it does not effectively capture the diversity dynamics necessary for effective RL preparation.
PassBa
54
50
48
100
Steps
Figure 4: The Pass metric throughout the cold-start training process, corresponding to Figure lb The trend mirrors evaluation accuracy, demonstrating the rerd-aaaddrraa process.
C.3 EXTENDED ANALYSIS ON EXPERIMENTS
To further investigate the mechanism of AESL, analyze the word clouds generated after the cold- start phase using CE loss and AESL. From Figure 5 we observe that both methods learn the new reasoning pattern effectively,
evidenced by the increased frequency of branching words (Wang eet al. 2025) (c.g., "wait" and "maybe" aligned with the dataset. However, the CE loss-based cold-start exhibits more severe over-memorization of specific dataset elements, such as repeated high-frequency numbers (e.g.. "12"). In contrast, AESL mitigates this issue by striking a balance between learning the new reasoning pattern and preserving the base model's original distribution.
In addition to the word cloud analysis. the results in Table 3 and Table demonstrate that AESL consistently outperforms traditional CE-based cold-start methods across varying dataset sizes and difficulty levels. By comparing the results across different cold-start dataset sizes in Table 3 it is evident that scaling the dataset size during the cold-start phase improves performance. Furthermore, comparing the results in Table 4 and Table reveals that using mixture of difficulty levels provides superior RL preparation compared to relying solely on either the simple or hard splits of the dataset.
These findings suggest promising direction for future research: optimizing the curation of cold- start datasets to better prepare LLMs for subsequent RL training. Such dataset curation, combined with improved loss functions like AESL, could further enhance the base model's capacity balance learning new reasoning patterns while preserving its original distribution, ultimately improving its readiness for downstream RI. training.
17

---

Published as a conference paper at ICLR 2026
need
JUM
numbers
Ino
Puositive
guad
me
values total
dmp Leee
maybe
must Frmm
Chint
ang
to
Limo
So Le
squati
tiinn
W
Ma
points
total
Point
umm
PITO
19 VO VEUEED
10
Letsthus
aingl
right
equa tio
tternatively o
Fumoe
compute Theck
Farst
1eG
need
case
sum must Sides!
Cold start dataset.
(b)
Base model.
C
naybetwa proble ee tor PEUrErrrCO CICOS
c ChEck
muu
wa
carch
SUM
ELOPAE
12.
number
mo letsum=
ry
T
estffrrm
alternatively
War
sides
Lets S
must
iöh
tota point
(c) Model after cold-stan start with CE loss.
(d) Model after cold-start with AESL.
Figure 5: Word frequency analysis showing vocabulary patterns across different methods.
C.4 COMPUTATIONAL OVERHEAD
While AESL introduces improved preparation of LLMs prior to the RL phase, its more complex form compared to standard CE loss inevitably introduces additional overhead. In our implemen- tation, we trade off memory complexity for time complexity, leading to approximately 20% more training time compared the standard CE loss.
However, it is important to note that, this work focuses on the cold-start phase, and the proposed loss specifically designed for this stage. The cold-start phase is typically far less resource- -interceinteesive compared to the subsequent RL phase. For example, in our experimental settings, the RL phase requires approximately 10x more wall-clock time than the cold-start phase when using the same computational resources. This is primarily due to the larger dataset size and slower convergence during RL. As result, the additional time complexity introduced by AESL in the cold-start phase accounts for only 2% the total post-training time. We believe this trade-off is worthwhile given the improved performance it provides.
D EXTENDED EXPERIMENTS
To validate our insights beyond mathematical tasks, we conducted additional experiments general reasoning benchmarks.
D.1
SETUP
Base Model and Algorithm We inherit the environmental settings described in Section fand use Dwen2.5-7B-Insttenn as the base model alongside the GRPO algorithm for RLVR. Math- is employed as the verifier.
Datasets For SFT cold-start training, we subsample data from the OpenTTooughs-111 14k (Guha ct al. 2025) dataset to provide general reasoning demonstrations and combine with subsampled data from Openrl 1 (Hugging Face, 2025) for format alignment. Specifically, we filtered out coding tasks as our initial experiments revealed that their inclusion (due to their non- boxed output format) significantly hindered the model's ability to generate boxed answers during general reasoning tasks. From these filtered results, we uniformly subsampled 3k examples and mixed them with 3k examples uniformly subsampled from Openr Maatr---a8-446-8811
18

---

Published as a conference paper at ICLR 2026
For RL post-training, subsampled 20k examples from the general reasoning dataset WebInstruct- verified (Ma al. (023). which contains general reasoning questions paired with verifiable answers.
For evaluation, the datasets used are summarized in Table 14
Table 14: Evaluation benchmarks for general reasoning.
Dataset
Number of Questions
ARC-C!
1172
GPQA-Dimond MMLU-Pro
198
12032
D.2 RESULTS
The results are presented in Table 15 We observe that the proposed AESL method consistently outperforms the baseline method using CE-loss. These findings confirm the effectiveness of AESL beyond the mathematical domain and emphasize the importance of dynamically balancing new pat- tern acquisition with the preservation of the base model's original distribution during the post-train cold-start phase.
Table 15: Post-training performance (f) with different cold-start methods.
Avg. denotes the macro-average across benchmarks.
ARC-C
GPQA-Dimond MMLU-Pro
Avg.
Base
87.97
19.19
53.15
47.44
+SFT (CE-(ES))
52.22
16.67
42.48
37.12
+RL
87,88
19.19
48.51
51,86
+SFT (AESL)
68.27
17.17
43.17
42.84
+RL
90.1
32.32
54.21
58.88
CE and CE-ES checkpoints coincide, results reported together.
E THE USE OF LLMS
LLMss are used to polish the writing in this paper.
19


Reviews:
[{"review_id": "F7oAFOKBvE", "reviewer": "Reviewer_WyUj", "summary": "Reinforcement learning fine-tuning of LLMs typically involves a SFT phase that primes the model for more efficient RL training. In this context, this paper studies the question of how to select the right SFT checkpoint to initialize RL training from. The authors claim that diversity metrics (entropy, self-BLEU) are better correlated with final RL performance than standard CE loss, and propose an adaptive early-stop loss (AESL) for SFT that consists of weighted CE where the weight is designed to scale decreasingly with both the likelihood of the target token and the likelihood of the prefix sequence.", "questions": "1. Cold start seems like the wrong term to use here - warm-start is more appropriate. I know Deepseek-R1 paper also uses this term improperly, but it should be corrected here.\n2. Table 5 should include vanilla SFT as well. Along with varying t_scaling, I would also like to see a sweep with the average prefix log prob term removed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper studies the important yet under-studied question of how SFT checkpoint selection influences final RL performance. This is especially critical because all modern LLM training pipelines involve SFT and RL phases for nearly all tasks, including reasoning and human alignment. The paper is clearly written and the proposed method is easy to understand.", "weaknesses": "I found the empirical motivation and evidence to be a bit lacking. In section 3, the authors claim that diversity has a clear correlation with final performance, but only present results for two (related) models and a single SFT dataset. More broadly showing that this is the case on more models (especially non-Qwen ones), different datasets, and different tasks (e.g. RLHF) would immensely strengthen the claims in the paper. \n\nI am also not strongly convinced by the argument presented in Figure 3, and would also like to see an ablation of the denominator in the loss weight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E0xlmCOXau", "reviewer": "Reviewer_iMAH", "summary": "This paper explores the best strategy for fine-tuning a base LLM checkpoint in order to achieve the best performance after RL fine-tuning. Currently, when RL fine-tuning is used to improve a reasoning LLM's ability to solve math or coding problems, a short supervised fine-tuning (SFT) is done on a base checkpoint. One reason for this, as mentioned in the paper, is to provide the base LLM with useful reasoning patterns (that the base model does not have, or is quite weak on), which are further enforced via RL.\n\nA key finding of the paper is that the commonly used metrics such as accuracy on downstream task during SFT does not correspond to the best performance after RL fine-tuning. In fact, the paper shows that lower accuracies during SFT (somewhat surprisingly) leads to the best performance after RL fine-tuning. The paper then proposes to use diversity metrics such as entropy or self-BLEU scores to pick the best checkpoints during the SFT stage, in order to maximize performance after RL. \n\nThe paper also proposes a new method of doing SFT before RL fine-tuning that, by using an adaptive weighting on the loss function, maintains diversity of the SFT model while also learning the necessary reasoning patterns for RL. Experiments show that the method improves over existing SFT + RL baselines on math benchmarks, for the Qwen class of models.", "questions": "Appendix A, equation (7)-(8), what are the \\pi(s_t|s_{t-1})'s, are they deltas functions?\n\nPage 5, line 260, what does the index j index over?\n\nTable 1, why are there no error intervals? How many seeds were used?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Empirically showing that commonly used metrics for SFT are not optimal for RL fine-tuning. This is an interesting finding that is somewhat counter to existing conventional wisdom. \n\n2. Proposed new methods for SFT fine-tuning target. The paper proposes to use diversity metrics such as entropy or self-BLEU to select checkpoints during SFT before RL fine-tuning. This is a well-motivated idea (maintain diversity to avoid model collapse and allow more exploration during RL) and seems to be supported by empirical evidence.\n\n3. Proposed a change to the standard cross-entropy loss during SFT in order to optimize for diversity metrics while allowing the model to learn new useful reasoning patterns.\n\n4. Experimental evidence for the effectiveness of the proposed method, and ablations.", "weaknesses": "1. Lacks analysis of the complexity of the proposed loss. How much additional complexity does the weighting require? Are there significant memory requirements to handle the logits for each training data sequence?\n\n2. Lack of diversity in experiments. Only the Qwen class of models is used in experiments. Why are there no other open-source, small-sized models? The evaluations are also only done on math benchmarks. It is difficult to be convinced of the generality of the proposed method when the breadth of results are quite limited.\n\n3. Lack of details of baselines. What are GEM and PSFT? You especially mention that GEM \"emphasizes promoting diversity\" and that PSFT is \"concurrent work\". How similar are they to the method proposed in this paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jb8qZ1Msgu", "reviewer": "Reviewer_H9W3", "summary": "The paper finds SFT checkpoints with best evaluation performance fail for RL due to distributional forgetting. Diversity metrics are better early-stopping criteria. Furthermore, it proposes AESL, balancing new pattern learning and base distribution preservation, outperforming baselines in math reasoning tasks.", "questions": "(1) Could you provide experimental evidence to verify whether AESL’s mechanism of balancing new pattern learning and base distribution preservation generalizes to other domains where RL post-training for LLMs is critical?\n(2) Could you conduct layer-wise parameter analysis or data-specific ablation studies to discuss the root mechanisms of distributional forgetting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) It pinpointed the core misalignment between traditional SFT and RL preparation, uncovering \"distributional forgetting\"\n(2) Developed AESL with token/subsequence-level adaptive weighting.\n(3) Conducted in-depth mechanism analysis and reported multiple metrics, enhancing result interpretability and credibility.", "weaknesses": "(1) While the paper identifies \"distributional forgetting\" as a key bottleneck, it lacks a deep investigation into its underlying causes.\n(2) It remains unproven whether AESL’s effectiveness translates to other domains (e.g., code generation, scientific writing).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2n3tTrf2ea", "reviewer": "Reviewer_Wffo", "summary": "In this paper, the authors show that when doing SFT before RL, the checkpoint with the best validation performance often isn't the best starting point for RL. The authors find this happens because models drift too far from the base distribution before traditional overfitting occurs. They propose two solutions: (1) using diversity metrics (entropy, self-BLEU) for checkpoint selection instead of validation performance, and (2) Adaptive Early-Stop Loss that balances learning new patterns while preserving the base model's distribution.", "questions": "1. I am curious about under what conditions does AESL underperform standard approaches? Are there cases where preserving base distribution hurts?\n2. How does AESL compare to simply doing less SFT (fewer epochs)? Is the sophisticated weighting necessary or would early stopping at lower loss achieve similar results?\n3. Is this approach applicable when use a model which hasn't been RL'd before? Qwen-Math already has been RL'd before. For example, Olmo-2 provides checkpoints right after pre-training.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The observation that best SFT performance does not translate to best RL readiness is valuable and practically very relevant.\n2. The paper is well-motivated and easy to follow.", "weaknesses": "1. I feel that in this paper, severely limited scope threatens generalizability of the results. For example, only mathematical reasoning tasks tested that too only on Qwen model family (both 7B variants are closely related). There's no evidence this generalizes to code, instruction-following, creative writing, etc. or is applicable to a different model family like Llama 3 8B.\n\n2. The AESL weighting function in equation 4 appears ad-hoc. Why softmax? Why this specific ratio? Why average log-probability in the denominator? The entropy decomposition (appendix A) shows a decomposition exists but doesn't strongly motivate this particular functional form. Moreover, there's no comparison to simpler alternatives (linear weighting by confidence, threshold-based stopping loss, etc.) An ablation comparing 3-4 different weighting schemes would greatly strengthen the paper.\n\n3. [Minor] There's no significance testing despite small margins (many improvements are 1-2 points in Tables 1-4).\n\n4. AESL requires additional forward passes for computing weights. How much does this add to training time? Is the improvement worth the computational overhead?\n\n\nOverall, in its current form, this paper feels more like a useful engineering trick for a specific setting (math reasoning with Qwen) rather than a fundamental contribution to post-training research. I'd encourage the authors to either (a) substantially expand the scope to other domains to demonstrate generality, or (b) provide much deeper analysis of the specific math/Qwen setting to get broader insights.\nRight now, the contribution in this paper feels incremental for ICLR but with significant additional work, this could become a solid accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)