Reinforcement Mid-Training
Authors: Yijun Tian
Shaoyu Chen
Zhichao Xu
Yawei Wang
Jinhe Bi
Peng Han
Wei Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Mid-training
Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We propose reinforcement mid-training, an intermediate phase between pre- and post-training to enhance performance and alignment, with a plug-and-play training method.

Abstract: The development of state-of-the-art large language models is commonly understood as a two-stage process involving pre-training and post-training. We point out the need for an additional intermediate stage called reinforcement mid-training with potential for strong performance gains. In this paper, we formally define the problem and identify three key challenges: (1) inefficient training due to excessive reasoning steps, (2) disregard of the imbalanced token entropy distribution, and (3) underutilization of token information. To address these challenges, we propose RMT, a framework for efficient, adaptive, and unified reinforcement mid-training with various innovative components. In particular, we first introduce a dynamic token budget mechanism that constrains unnecessary reasoning steps and mitigates model overthinking. Next, we design a curriculum-based adaptive sampling method that fosters a progressive learning trajectory from easy to hard tokens. Finally, we present a dual training strategy that combines reinforcement learning with next-token prediction, ensuring targeted learning on key tokens and full exploitation of all token information. Extensive experiments demonstrate the superiority of RMT over state-of-the-art methods, achieving up to +64.91% performance improvement with only 21% of the reasoning length in language modeling. We also show that checkpoints obtained after reinforcement mid-training can benefit the subsequent post-training, yielding up to +18.76% improvement in the mathematical domain.

REINFORCEMENT MID- AID-TRAINING
Yijun Tian L,, Shaoyu Chen Zhichao Xu', Yawei Wang", Jinhe Bi Peng Han', Wei Wang". 1."
Xi' Jiaotong University Shanghai Jiao Tong University University of Utah
4 The George Washington University Ludwig Maximilian University of Munich
meetyijun gmail.com
wei.wang@ wwt.euuu..uuueeu.c.n
Training Code:
ABSTRACT
The development of state-of-th ofttt--o--heeartt large language models is commonly understood as a two-stage process involving pre-training and post-training. We point out the need for an additional intermediate stage called reinforcement mid-training with potential for strong performance gains. In this paper, we formally define the problem and identify three key challenges: (1) inefficient training due to excessive reasoning steps, (2) disregard of the imbalanced token entropy distribution, and (3) underutilization of token information. To address these challenges, we propose RMT, framework for efficient, adaptive, and unified reinforcement mid-training with various innovative components. In particular, we first introduce a dynamic token budget mechanism that constrains unnecessary reasoning steps and mitigates model overthinking. Next, we design a curriculum that fosters progressive learning trajectory from easy to hard tokens. Finally, we present a dual training strategy that combines reinforcement learning with next- token prediction, ensuring targeted learning key tokens and full exploitation of all token information. Extensive experiments demonstrate the superiority of RMT over stte-of-the-an methods, achieving up +64.91 % performance improvement with only 21 % of the reasoning length in language modeling. We also show that checkpoints obtained after reinforcement mid-trainin -od-training can benefit the subsequent post-training, yielding up to +18.76% % improvement in the mathematical domain.
adaptive sampling method
INTRODUCTION
The development of state-of-the-at -hrt large language models is commonly understood as a two-stage process (Minaee et al., 2024, Kumar et al.] 2025 Tian et al.) (025). The first stage, pre- pre-training, involves training a model on vast, unlabeled web-scale data using objectives such as next-token prediction. This stage equips the model with broad world knowledge and linguistic competence (Devlin et al. 2019a] Raffel et al. 2020: Brown et al. 2020). The second stage, post-training, aims to align the model with human objectives downstream tasks, often using specialized, high-quality datasets. At this stage, the goal is to instill capabilities such as instruction following, tool use, agentic reasoning, and preference alignment, typically through methods like supervised fine-tuning (Wang) et al. 2022 Chung et al., 2024) and reinforcement learning (Ouyang et al.) 2022 Lai et al. 2025).
However, this two-stage view overlooks crucial, emerging intermediate phase: mid-training, which leverages the large-scale unlabeled pre-training data with more targeted objectives than general pre-training, systematically enhancing complex capabilities like mathematical reasoning better prepare the model for subsequent post-training (Wang et al. 2025b] OLMo et al. 2025 Wake et al. 2024). Unlike post-training that usually depends on domain-specitn signals, mid-training relies pre-training data that does not require extra human annotations or verified rewards. The foundational concept of applying reinforcement learning to the mid-training stage was pioneered in the Reinforcement Pre- rre-iiinnng approach (Dong et al. (025). Despite its name, this method technically functions in a mid-training stage. The base model they used has
data with labels or reward
Equal Contribution *Corresponding Author

---

Accuracy vs Response Length
Token Entropy Distribution
70
222
The 80th parcentile: 1.26
60
RMT-Q3
Frequency (log scale)
Accuracy( (%)
RMT-R1
2
RPT
Qwen3-14B
30
w/o reasoning
214
20
Qwen3-145
212
10
w/ reasoning
0
210
300
1200
1500
0.0
2.5
Response Length
Token Entropy
(a)
(b)
(a) Our proposed models RMT-Q3 and RMT-R1 achieves superior performance with significant less reasoning length. (b) The majority of the tokens are of low-entropy.
Figure 1:
already demonstrated established instruction-following and reasoning capabilities, positioning clearly beyond the pre-training stage. However, existing work faces three critical challenges:
Inefficient Reinforcement Learning with Overthinking: The thinking process is unconstrained and Icads the model generate excessively long reasoning chains, causing additional computation overhead during training and inference. For example, in Figure I (a), existing method requires significantly more tokens for reasoning while cannot guarantees satisfactory performance.
Overlook of Imbalanced Token Entropy Distribution: Tokens vary in entropy. reflecting various levels of uncertainty and learning difficulty. Existing approaches sample challenging tokens indiscriminately, which can overload the model with highly difficult tokens during the early stages training, before has sufficient capacity.
Underutilization of Token Information: As shown in Figure (b), the majority of tokens in the data exhibit low entropy. Existing method only support the training on high-entropy tokens, neglecting the vast majority low-entropy tokens. For unlabeled pre-training data, where every token contributes to language understanding, this exclusion leads to substantial information loss and missed learning opportunities.
In this paper, first formally define the problem of Reinforcement Mid- Mid-Training. To solve this problem and overcome the challenges, we propose RMT, a Reinforced Sid-Trrining framework that is efficient, adaptive, and unified. RMT contains several innovative designs: (1) To address the overthinking problem, we introduce a dynamic token budget mechanism that adaptively constrains the length of the reasoning process, encouraging the model become taking into account the imbalanced token entropy distribution, we present curriculum-bast adaptive sampling method that guides the model on a progressive learning trajectory from easy hard tokens. This ensures stable learning while gradually increasing the difficulty to master complex reasoning. (3) To incorporate every token information, we propose a dual training strategy that synergistically combines telen-selective reinforcement learning with loken-inclusive next-token prediction. This ensures that the model benefits from targeted, reward-driven learning on challenging tokens while still capturing the full information from all other tokens. Language modeling experiments show that RMT can achieve up to +64.91 performance improvement with only 21 % of the reasoning length. We further demonstrate the checkpoints obtained from reinforcement mid-training later post- pest-training stage, with up to 118.76% performance improvement. Moreover, our method's plug-and sogg-and-playy nature makes it effective with different backbone model choices and reinforcement learning algorithms. To summarize, our major contributions in this paper are as follows:
concise and efficient. (2) By
benefit
We formally define the problem of Reinforcement Mid-Training and point out three critical challenges: inefficient overthinking, imbalanced token entropy distribution, and the necessity of utilizing all token information.
To address the challenges, we propose RMT, plug-and and--lay framework that efficient, adap- tive, and unified. RMT contains several innovative designs, including a dynamic token budget mechanism, curriculum-hase adaptive sampling method, and dual training strategy.
2

---

Extensive experiments both language modeling and continual post-training demonstrate the superiority of RMT over state-of-the- art methods in both effectiveness and efficiency, establishing a stronger foundation for post-training stage.
2 RELATED WORK
LLM Pre- Tre-iTainngg. Language models are predominantly trained with unsupervised pre-training, a process that equips the model with both linguistic fluency and broad world knowledge. Early works include BERT (Devlin et al. 2019b). T5 (Raffel et al. 2020), and GPT-2 (Radford et al., 2019). Modern LLMs like GPT-3 (Brown et al. 2020) establish the dominant paradigm of causal language modeling via next-token prediction. Scaling law research (Kaplan et al.) 2020 Hoffmann et al. 2022] further elucidated how compute, data, and parameters interact to enhance model pre-training
performance, laying the groundwork for recent advanced models such as LLaMA ( (Touvron et al.
2023). GPT-4 (Achiam et al. 2023), and Gemini (Gemini 2023 Comanici et al., 2025).
LLM Post- Dost-Training. Post-Training refers to the techniques and methodologies employed after model has undergone pre-training, aiming to refine and adapt the model for specific tasks or user requirements (Tie et al., 2025 Du et al. 2025). From methodology perspective, existing post-
training methods mainly leverage supervised fine-tuning (SFT) (Wang et al. 2022 Wei et al. 2022
Chung al.) 2024) and reinforcement learning (RL) (Ziegler al. 2019 Rafailov et al. 2023
Wang et al. [025a Li et al. 2026). Recently, RL has gained significant attention from industry and academia due its effectiveness. While RLHF (Ouyang et al. 2022 Bai et al.] 2022) with PPO algorithm (Schulman et al. 2017) remains the dominant framework, recent works have been proposed to tackle the critical limitations in stability and computational efficiency of PPO. Notable advances
include GRPO (Shao et al. 2024). DAPO (Yu et al. 2025). VAPO (Yue et al., 2025),, and GFPO (Shrivastava al. (025). Curriculum learning (Hammoud al., [025] Chen al. (025)) and the combination of SFT and RL have also been introduced to further expanded the RL paradigm.
LLM Mid-Training: Compared to the well-studied paradigms of pre-training and post-training. mid-training has only recently emerged as a distinct stage in the LLM lifecycle, positioned between the two. Following the definition of (Wang et al. 2025b), mid- mid-training can be seen as an intermediate phase in both computational and data demands, designed to preparing models for post-training. While prior efforts (Wake et al. 20224] OLMo et al.) 2025 Wang et al. (025b) have explored these directions, the role of reinforcement learning (RL) in this stage remains underdeveloped. The sole attempt far, RPT (Dong et al.) 2025) does not resolve the central challenge of bringing RL into mid- mid-training. i.c., its prohibitive computational cost over massive unlabeled corpora. In this work, we formalize the problem of Reinforcement Mid- dra-nirannng and introduce an efficient framework that enables RL to be applied effectively at this stage.
3 REINFORCEMENT MID-TRAININ
In this section, we formally define Reinforcement Mid-Training and present the problem definition.
What is Reinforcement Mid-Training? The model is optimized using reinforcement learning on large-scale unlabeled pre-training data, with the goal of enhancing subsequent post-training perfor- mance on downstream tasks. In this framework, the model is encouraged to allocate reinforcement learning updates primarily to key tokens that are the most influential for reasoning, while relying on standard next-token prediction for the remaining tokens. The central challenge to reduce the computational cost of reinforcement learning while ensuring concise and efficient reasoning.
Problem Definition. Given sequence of tokens S { Te..., } of length we partition the tokens into two disjoint subsets: 4nz. C S denotes the tokens selected for reinforcement learning and &NT = S \ &RL represents the tokens for standard next-token prediction, with [ >>> (Pu.)- Given token Tm E PRL with position < L for reinforcement learning, the policy model nо takes the preceding tokens {T<m} C S as input and predicts the next token Tm by sampling response Om (Cms 3m)) where Cm denotes chain-of-thoud matching reward r is introduced by comparing the exact match of prediction ym and the ground truth Tm- A reinforcement learning algorithm is applied optimize the model with the rewards obtained. In the meantime, for every token Tn E & we perform the next-token prediction by maximizing the likelihood log To (T Ten).
reasoning and ym is the predicted token. A
3

---

Example Question: Sik students sit in a and leave at random times, What the probability that one student must another to exit the row?
Example Answer: The probability $p$ that no student will have to pass by another student to exit the row is the probability that the first student is of the students the end, the student to leave
of the ends the (53 Icdot (racf2H4) lodat Mrae so the desired probability is $1 33-p=\/fac(4334455
students, (p-) lcdat
Entropy Distribttion
Masking
Token Selective RL Token-tnclusive NTP
max(Pra Be D
99RO
Dynamic
IN IOOM
Policy Model
LNTP
iomn
NTP Input
Whight
xxt?
Length
Token Budget
XXX?
Policy Model
e'e'
Reward
Input
Mechanism
XXXX?
m''
Matching
Question
legea)
Regular Taken
Curriculum
stappe
Adaptive Sampled
Reference
Model
hnh''
Reward
Easy
[ce]
Medium
[mm]
[hhh]
Sampling
RL Input
The overall framework. We first identify the difficulty of each token using entropy as measure of uncertainty. Then, token budget is assigned to dynamically control the generation length. Building on this, we introduce curriculum-No from easier tokens during early stages and challenging tokens as its capabilities strengthen. Next, we perform reinforcement learning on sampled tokens with length rewards and verifiable matching rewards to enforce training efficiency and accuracy. At the same time, the majority low-entropy tokens are incorporated through next-token prediction with a unified training objective.
Figure 2:
sampling to encourage the model learning more
METHOD
In this section, we formally present RMT to address the challenges described in Introduction. RMT contains three innovative components, including dynamic token budget mechanism, curriculum- based adaptive sampling strategy, and a unified training objective token-selecting reinforcement learning and token-inclusive next-token prediction. Figure illustrates the framework of RMT.
4.1 DYNAMIC TOKEN BUDGET MECHANISM
To mitigate the overthinking problem and enable efficient reasoning, we introduce a token budget dynamically constrains reasoning length while maintaining prediction quality, through a tailored decay factor coupled with length-based reward signals. In particular, let T denote the total number of training steps and E {0, 1,... ....., ,) represent the current training step, we define training step-dependent token budget B, that decays exponentially with training progress:
B, = max (Bmins) Bo [Bo -/TT)
(I)
where Buin 2 1 is the minimum allowable budget to ensure non-trivial reasoning capacity is pre- served, Bo is the initial budget, and 7 E (0, 1) is the decay factor controlling the rate of budget reduction. At training step f, inject budget-aware instruction into the input prompt, specifically requesting the model to utilize exactly B, tokens within the designated <think></ faik></think rea- soning segment. To further encourage the model adhere the target budget B, while maintaining flexibility, we design a length reward function (éo B,) with a triangular profile centered at the target budget. Let é denote the actual number of tokens generated in the reasoning trace. The length reward function is defined as follows:
T Tmux
B.
if < 8 Bt, if B, < 2B,
Tlen (e; B:)
Tmax
28, -
B.
0,
if > 2B,
where Fmax denotes the maximum reward and the obtained length reward Mie is used in later training stage. This reward structure exhibits several desirable properties: (1) it achieves its maximum value Tmax when { = B,, encouraging precise budget adherence, (2) it linearly penalizes both insufficient reasoning (l B,) and excessive verbosity (B, l < 2B,), and (3) it provides reward
4

---

for severely over-budget reasoning (L 2B,), effectively discouraging computational waste. The integration of this dynamic budget mechanism enables the model to adaptively reduce reasoning overhead as its capabilities mature, leading to more efficient training dynamics and improved resource utilization without sacrificing performance.
4.2 CURRICULUM-BANS BASED ADAPTIVE SAMPLING
To effectively exploit the inherent imbalance in token entropy distribution and support adaptive model training, we design a curriculum- sampling method that prioritizes tokens according to their difficulty levels. This enforces the model to learn from easier tokens in early stages, while gradually shift the attention to challenging tokens as the model's ' capabilities strengthen. Specifically, we categorize tokens into various difficulty levels based on their entropy values, categorized as easy, medi um, and hard. During training, tokens are sampled from these categories with dynamically evolving probabilities. Formally, let denotes the total number of steps. We define two curriculum transition points & and la for progressive adjustment, where 0 < 1 l2 T. The corresponding sampling probabilities pd and p over difficulty categories are defined as:
pd = (P'áay S Priedium Pinnd)
Dd (P'äsy p Peddumm Puard)
where we constraint Edpd = across different difficulty level d to ensure valid distributions. To enable adaptive learning, the sampling probability p is changing according to curriculum schedule, which is implemented via piecewise linear interpolation that smoothly transitions the sampling probabilities over difficulty levels:
P $1
ift t1,
p'
(1
p
a + r
if 1 S 122 where f-h 12-h1
ift t2
where t is the current training step and is the decaying factor. The adaptive sampling process operates in two stages. We first sample a difficulty level d ~ Categorical(p") according to the current curriculum probability. Then, we select uniformly among available candidates that difficulty level to form a token set for later training.
This entropy-drive driven murriculum-based sampling strategy separates the training into three distinet phases: (1) Early training (t < t1) emphasizes easy and medi um difficulty tokens to establish stable convergence simpler reasoning patterns.
(2) Transition phase (4, < (2) provides smooth interpolation that gradually increases exposure to medi um and hard entropy tokens as reasoning capabilities strengthen. (3) Late training (t 2 t²) maintains stable sampling probabilities that
leveraging challenging high-entropy ha tokens to boost the model performance.
4.3 BEXEN-SELECIVVE RL AND TOKEN- INCLUSIVE NTP
To address the token information underutilization problem and ensure the contribution all tokens, we present unified training objective that synergistically combines token-selective reinforcement learning with token-inclusive next-token prediction. These two objectives perform different functions. Reinforcement learning are performed small set of selected tokens with different levels of difficulties, while next-token prediction is leveraged to enable the model learn from the majority of low-entropy tokens.
Token-Selective Reinforcement Learning. Given the selected token set C, a token 7 E C, and the token position pos in the sentence, ask the model to reason about the next token conditioned on the prefix context T<pos- Specifically, the policy model takes {T<pos> } as input and predicts the next token Tpos by sampling G responses (oros 1 Each response pos = (crpos yjoss consists of a chain-or Shain--ff-thouhh reasoning sequence cjoss and final prediction sequence Vyoos We employ verifiable matching reward rjos for i-th response and position pos by comparing the exact match of prediction vjos and ground truth next token Tpos We then form per-sample composite reward yd by adding r and the dynamic length reward ric obtained from Eq. 2 The procedure is as follows:
(1,
if Vjos Tpass
pos = (0. otherwise,
w) rpos + rien
(5)
5

---

where [0, 1] is the trade-off weight balancing both rewards, ensuring both accuracy and concise- ness. To obtain a stable learning signal, we employ GRPO as our reinforcement learning algorithm and compute group-relative, whitened advantage from {r') Pi=1
1 G
Σ
a == (id))))
A'
a
where i is the mean reward across all G samples, o is the standard deviation of the rewards, and 5 > 0 is small constant to prevent division by zero. The normalized advantage A' measures how much the i-th sample's reward deviates from the group average, thereby guiding the policy update toward relatively better-pett petter-performing outputs. Finally, the reinforcement learning objective is defined as the expectation of the advantage-weige weighted angllike of the sampled outputs:
G
(f) = Eyane
Σ i=1
A' log T (<<po)) - KL Teat)
(7)
where A' is the group-relative advantage from Eq. 6 To To(oppss (<ppo)) is the policy model's probability of generating response opos given the input sequence T<poss B controls the strength of the KL- divergence penalty, and Tee is the reference model.
Token-Inclusive Next-Token Prediction
For the majority of tokens with low-entropy that not exploit for reinforcement learning, we include them for the standard next-token prediction training to acquire comprehensive information. In particular, have been used in reinforcement learning to avoid duplicate training. Then, we employ supervised fine-tuning with teacher foreing for next-token prediction. The procedure as follows.
first apply masking on the selected tokens that
Mpos 11 [Tpos & C), LNTP = Σ Myos log po (Tpos Te
(8)
where mas is binary mask specifying whether the token position pos contributes to the NTP loss, the function [Tpps & C] is the indicator function, equal to 1 if Tpos is not in C and 0 otherwise, (0) is the next-token prediction loss, and S is the training sequence. To integrate both supervised and reinforcement learning, we combine the token-inclusive next-token prediction loss with the reinforcement loss obtained in Eq. 7
L (Q) + A LNIP(O),
where is trade-off weight for balancing LNIP and CRL. This design creates complementary optimization dynamics that leverage the strengths of both paradigms: reinforcement learning improves model non-trivial reasoning decisions via uncertain high-entropy tokens, while next-token prediction handles language understanding with broad coverage of vocabulary.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets and Baselines. For language modeling, we follow the settings of RPT to employ Omni- MATH dataset and identify token entropy, resulting in collection of 4051 training data and 200 evaluation data. For baselines, we compare against strong reasoning models R RI-Distill-Qwen- 14B (denoted as 1--Distill-44B) Qwen3-14B, and the state-of-the-art method RPT. We conduct the experiments in both settings of next token prediction and next token reasoning, with accuracy as the evaluation metric. For continual tost-training, we evaluate and post-train the language modeling checkpoints on Skywork dataset (He et al. (025). which comprises a diverse collection of mathe- matical problems spanning multiple levels of difficulty. By uniformly stratifying and sampling data with difficulties ranging from 1 to 13, we include 4096 data points the training set and 600 for evaluation, ensuring balanced coverage across difficulty levels.
Implementation Details. For the proposed model, we set the batch size to 128, epochs to 10, learning rate to 1e-6, initial budget B, to 800, minimal budget Bynn to 1, decay factor y to 0.2, transition points f1 and to to 30% and 70% of the total training steps. the number of rollout G to 8, and trade-off weight À to O.I. We set max response length to 2048 for language modeling and 4096 for continual post-training. We run all experiments on sixteen NVIDIA Tesla H100 GPUs with 80GB RAM.

---

5.2 LANGUAGE MODELING
A language modeling task involves training a model to predict the next token in a sequence given its preceding context. We present results in Table [ reporting performance across tokens of varying difficulty levels, including easy, medium, and hard. We consider two settings for baselines, including next token prediction (NTP) with standard autoregressive approach and next token reasoning (NTR), where a chain-of chain-of-thought is generated before the prediction. We denote the proposed models as RMT-RI that uses RI-Distill- 14B as the base model, and RMT-Q3 that depends on Qwen3-14B,
According to the table, NTP methods yield decent results, confirming the effectiveness of pre-trained models and establishing a reliable benchmark. In contrast, NTR methods perform poorly across all difficulty levels, reflecting the absence of explicit reasoning during pre-training stage and the resulting inability to predict tokens with explicit reasoning chains. The current state-of-the-arr method RPT demonstrates the potential of reinforcement learning, but does not substantially outperform the base model. By comparison, our method achieves state-of-the-an performance, surpassing the strongest baseline RPT by average improvement of +30.74% with RMT-RI and +64.91 % with RMT-Q3. Moreover, results show that stronger base model amplify these gains: RMT-Q3 consistently outperforms RMT-R1 across all difficulty levels. This demonstrates the superiority of our method in harnessing reasoning capabilities in predicting tokens and language modeling.
Table 1: Performance comparison of language modeling. The best and second-best results are highlighted in bold and underlined, respectively.
Method
Easy
Medium
Hard
Average
R1-Distill114B (NTP)
42,04
31.71
19.64
31.13
Qwen3-14B 14B (NTP)
47
34.61
25.01
35.84
RI- RI-Distill-1445 14B (NTR)
4.76
2.43
2.09
3,09
Qwen3-14B (NTR)
8.47
5.08
4.51
6.02
RPT
48.67
35.84
23.03
35.85
RMT-RI
62,50
43.96
34.14
46.87
Improvement
+28.42%
+22.66%
+ +48.24%
+30,74%
RMT-Q3
76.92
55.79
44.64
59.12
Improvement
+58.04%
+55.66%
+93.83%
+64.91%
5.3 CONTINUAL POST- TRAINING
To verify the transferability of our approach, we leverage the checkpoints obtained after reinforcement mid-training in Section 5.2 and further post-train them on distinct dataset Skywork. For fair comparison, adopt GRPO the training algorithm. The results are shown in Table
Table 2: Performance comparison before and after post-training.
Model
Before
After
RI-Distill-14B
23,00
51.00
Before post-training, all baseline models exhibit compa- rable performance, with Qwen3-14P lagging significantly behind the others. However, after undergoing continual post-training, all models show substantial performance gains, with accuracy rising from the low 20s to nearly 50 or higher. Crucially, the proposed method outperforms all baselines and delivers the highest gains across both phases. Among all methods, RMT-Q3 achieves remarkable jump from 25.17 to 64.33 accuracy, showing that checkpoints obtained after reinforcement mid- traininnn can be effectively post- post-trained to achieve strong downstream performance. Compared to the strongest baseline, RMT-QQ3 shows an improvement of +8.63 % over RPT before post-training, and +18.76 % over Qwen3- after post-training. These results clearly demonstrate the superior transferability and generalization ability of our approach towards post-training stage.
Qwen3-14B
12.84
54.17
RPT
23.17
48.50
RMT-R1
24.08
54.67
RMT-Q3
25.17
64.33
Improvement
+8.63%
+18.76%
5.4 RESPONSE LENGTH AND GENERATION TIME ANALYSIS
To evaluate the computational efficiency of our approach, we analyze both response length and generation time across the language modeling and continual post-training stages. Here define
7

---

response length the average number of tokens generated per response, and generation time the average time required to generate all responses within a batch.
Superior efficiency during language modeling. As shown in Table both RMT-R I and RMT-Q3 generate significantly shorter responses compared all baselines with 188 and 186 tokens, respectively. This reduces response length to just 21 % of RPT (872 tokens) and 12% of Qwen3-14B (1577 tokens). Figure (a) further illustrates this efficiency gain: as training progresses, both response length and generation time for our method decrease substantially. Importantly. this dramatic reduc- tion in computation cost is achieved without sacrificing quality, as our methods maintain superior performance according to Table
Table 3: Response length comparison.
Method
Length
RI-Distill-14B
1305
Qwen3-14B
1577
RPT
872
RMT-RI
188
RMT-Q3 186
Dynamic adaptation during continual post-training. According to Figure 3 (b), our method generates substantially shorter reasoning chains compared to RPT step However, as post- training progresses and the model encounters more complex downstream tasks, it adaptively increases response length to accommodate deeper reasoning requirements. Notably, even at this increased length, our model remains more efficient than RPT while achieving superior performance in Table 3 The generation time curves mirror this adaptive behavior, consistently demonstrating computational efficiency throughout the training process. These results show that method effectively prevents verhose reasoning without sacrificing accuracy.
Language Modeling Efficiency
Continual Post-Training Efficiency
1494
116
3765
405
1922
Sexdim (19)) Conentin Tire
Seavire (01). Corestin
360
Response Length
1151
Oun (INT)) Hosoras Lengh
Comeatiin firm
8
Generaliin Time (8)
Reaponse Length
2022
Dan -emasdon fiee
Dar y011)- Woppee Lengh
316
Generatiol renerttioon Time
979
2501
807
60
2079
228
635
4B
468
22
1230
282
815
-00
250
300
363
50
100
150
200
250
Step
Step
(a)
(b)
Figure 3: Efficiency comparison in language modeling (a) and continual post- post-training (b).
5.5 ABLATION STUDY
Since RMT contains various designs (i.e., dynamic token budget mechanism (DTB), curriculum-bais based adaptive sampling (CAS), and dual training strat- egy with both RL and NTP objectives), we con- duct ablation studies to analyze the contributions of different components by removing each of them independently (see Table Specifically, removing the NTP loss significantly affects the performance, highlighting the critical role of next-token predic- tion in language modeling and the necessity of in- corporating comprehensive token information. In addition, the decreasing performances of removing DTB and CAS demonstrate their effectiveness in enhancing the model, while simultaneously improv- ing efficiency. Overall, RMT consistently achieves the best results across all settings, demonstrating that each component contributes meaningfully and that their integration provides a synergistic effect, leading to superior performance.
Table 4: Ablation study.
Method
Easy
Medium
Hard
Average
w/o DTB
60.35
32.79
27.59
40.24
w/o CAS
59.32
42.66
27.68
43.22
w/o NTP
40.00
34.26
27.27
33.84
RMT-RI
62.50
43.96
34.14
46.87
w/o DTB
60.87
50.48
34.69
48.68
w/o CAS
62.71
50.28
30.51
47,83
w/o NTP
52.17
45.63
29.41
42,40
RMT-Q3
76.92
55.79
44.64
59.12
5.6 CASE STUDY
For more intuitive understanding and comparison, randomly select example from OmniMath and present the model generation in next token 'probability". Its reasoning is concise (165 tokens), grammatically correct, and remains
Table According to the Table, RMT-Q3 correctly predicts the
8

---

tightly aligned with the question. The model effectively links the incomplete clause the noun "probability" and avoids unnecessary detours. This focused reasoning connects the context to the intended continuation and produces the correct answer. On the other hand, RPT generates much longer and less coherent reasoning (1,376 tokens). Its output repeatedly cycles through candidate options and revise itself multiple times. This back-and-forth deviates from the intended semantics, ultimately producing the incorrect prediction "same". Overall, our method is both accurate and concise. It makes the correct decision with compact and focused reasoning chain, while RPT is verbose, inconsistent in its guesses, and returns the incorrect answer.
Table Case study. The proposed method produces concise and coherent reasoning traces, whereas existing method often generate verbose and inconsistent reasoning. characterized by frequent hesita- tions and shifts. Key words and sequences that guide the reasoning process are highlighted in gray.
Complete the given text by predicting the next token, list multiple potential tokens and select the most probable one as the final prediction. The text is: "Question: Six students sit a row and leave at random times. What is the probability that one student must pass another to exit the row? Answer: The probability p that no student will have to pass by another student to exit the row is the Expected Answer: Probability
Input:
RMT-Q3
Hmm, so need to predict the next token after the given text. First. I'll look the structure of the last sentence: ... The most probable next word is "probability" since the context is about probability, and the sentence structure suggests it should continue the idea introduced before. Looking at the tokens, the next word should follow grammatically. Given the context, "probability" fits perfectly here So, the next token is most likely "probability" as it follows the context and grammatical structure. Answer: Probability
Reasoning:
Length: 165
RPT
Reasoning: Okay, so need to predict the next token in the given text. Let's see, the current text is ... The question is about The sentence is incomplete. The structure after "is the" might he followed by word like "probability". "chance". "number". or maybe "case". Alternatively, perhaps it's "probability p... is the..." maybe the word that comes after "is the" would be something like "probability". but wait, the sentence already starts with "The probability p...", so maybe it's fraction or something. Wait no. maybe the word after is "probability" itself, but that seems redundant. Alternatively, perhaps the sentence is saying it's s the "number", so maybe the next word is "number". Or maybe it's the reciprocal or some function. Wait, let me think. The question is ... So maybe the next word after "is the" would be "same" as some known probability, but perhaps in the text it's phrased as "the probability... the reciprocal..." ". but maybe not. Alternatively, perhaps the word after is "case", as "the probability... is the case", but that might not make sense. Alternatively, maybe the word is "same" as another scenario. Wait, maybe in the original text, it's phrased as ... Alternatively, perhaps the text is "the probability p... is the number of ways...". maybe the word after is "number". Alternatively, maybe it's "the probability p is the reciprocal...", so maybe "reciprocal". But But looking back. ... But without more context, perhaps the most probable word after "is the" is "same", because maybe the probability is the same as some other scenario. So, based on this, I think the most probable taken after "is the" is "same". Wait, but perhaps "same" isn't the correct word. Alternativll., maybe it's "case", "value", "chance", etc. Hmm. Alternatively, But given that the question is asking for the probability p that student will have to pass another, perhaps the text continues as "the probability p... is the same as..." so "same" would come next.
Answer: same
Length: 1376
CONCLUSION
In this paper, we formally introduce and define the problem of Reinforcement Mid-Training an intermediate stage that strengthens large language models before downstream post-training. We identify three key challenges in this setting: excessive computational overhead from overthinking, learning ineffectiveness caused by imbalanced token entropy, and underutilization of token informa- tion. To address these challenges, we propose RMT, a reinforcement mid-training framework that is eoeeeeeeeepeemeee
efficient, adaptive, and unified. Extensive experiments show that RMT improves both performance and efficiency, producing models better prepared for subsequent post-training stage. This work positions Reinforcement Mid- Trad-ingg as a valuable new stage in the LLM lifecycle and opens up promising directions for future research.
9

---

7
ACKNOWLEDGMENT
This work was supported in part by the National Natural Science Foundation of China, under Grant 62441227.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kermion, Andy Jones, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08003,, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang. Alexandre Pich e, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for llm reasoning. arXiv preprint 1rxxi:2505.14977 2025.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXivv2507.06261, 2025.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019a.
Jacob Devlin, Ming-Wos Hei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019b.
Qingxiu Dong, Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, and Furu Wei. Reinforcement pre-training, arXiv preprint 2XXi:2500600807 2025.
Hongzhe Du, Weikai Li, Min Cai, Karim Saraipour, Zimin Zhang, Himabindu Lakkaraju, Yizhou Sun, and Shichang Zhang. How post-training reshapes LLMs: A mechanistic view knowledge, truthfulness, refusal, and confidence. COLM, 2025.
Gemini. Gemini: a family highly capable multimodal models, arXiv preprint arXiv:2312. 2023.
Hasan Abed AI Kader Hammoud, Kumail Alhamoud, Abed Hammoud, Elie Bou-Zeid, Marzyeh Ghassemi, and Bernard Ghanem. Train long, think short: Curriculum learning for efficient reasoning, arXiv preprint arXiv:2508, 08940, 2025.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22311 22..0....211, 2025.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2001 (0000), 2020.
10

---

Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Haat Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models. arXiv preprint ar arXiv:2502.21321, 2025.
Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, et al. A survey of post-training scaling in large language models. In ACI, 2025.
Jiazheng Li, Yawei Wang, David Yan, Yijun Tian, Zhichao Xu, Huan Song, Panpan Xu, and Lin Lee Cheong. Salt: Step-level advantage assignment for long-horizon agents via trajectory graph. EACL 2026, 2026.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint wxXv:2442.061999 196, 2024.
Team OL,o, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00653 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurIPS, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint trXiv:1707.06344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, et al. Deepseek- math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning. arXiv preprint arXiv:2508.099 09726, 2025.
Yijun Tian, Xingjian Diao, Ming Cheng, Chunhui Zhang, Jiang Gui, Soroush Vosoughi, Xiangliang Zhang, Nitesh Chawla, and Shichao Pei. On the design choices of next level llms. TechRxiv, 2025.
Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, et al. A survey on post-training of large language models. arXiv preprint 2555::2503.00072, 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth 'ee Lacroix, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv.2302.13971, 2023.
Alan Wake, Bei Chen, CX Lv, Chao Li. Chengen Huang, Chenglin Cai, et al. Yi-lightning technical report. arXiv preprint arXiv:2412.01253 2024.
Jiongxiao Wang. Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, and Lin Lee Cheong. Reinforcement learning for self self-improving agent with skill library. arXiv preprint arXiv:2512. 17/02, 2025a.

---

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, et al. supralnnaturalnnntttttinnns Generalization via declarative instructions on 1600+ nlp tasks. In EMNLP, 2022.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXi::2506.20512 2025b.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, et al. Finetuned language models are zero-shot learners. In ICLR, 2022.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo. Yu Yue, et al. Dapo: An open-source IIm reinforcement learning system scale. arXiv preprint arXiv:2503. (442::2503.4477,, 2025.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang. TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05122 2025.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint ArXiv: 08593, 2019.
12


Reviews:
[{"review_id": "IgoC0ciBzn", "reviewer": "Reviewer_D7Vg", "summary": "This paper introduces RMT, a framework for efficient, adaptive, and unified reinforcement mid-training. It proposes several strategies to enhance both training efficiency and performance. These include: 1) a dynamic token budget mechanism, which prevents the generation of excessively long chains of thought that can impede training efficiency; 2) a curriculum learning strategy that determines token difficulty based on entropy, allowing the model to gradually progress from simpler to more complex tokens during training; and 3) a combined training approach that utilizes both cross-entropy and Reinforcement Learning (RL) loss. The experimental results demonstrate that RMT improves next-token prediction capabilities and generalizes effectively to subsequent post-training with RL.", "questions": "Could the authors elaborate on the initial stages of training? Was a \"warm-up\" phase or a specific set of prompts used to enable the model to learn how to reason for the next token from a pre-trained model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method shows strong performance, significantly enhancing the model's ability for next-token prediction and demonstrating good generalization to the post-training RL phase.\n2. The paper is well-written, and the proposed method is straightforward to understand.", "weaknesses": "1. I was unable to find some specific experimental details. For instance, what proportion of tokens were subjected to reinforcement learning training versus the standard cross-entropy objective? \n\n2. I am curious about the added value of reinforcement mid-training in a scenario where the post-training RL phase is exceptionally thorough and prolonged. Would the improvements from RMT still be significant if the model undergoes a very comprehensive post-training RL alignment?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6yxS4yeagp", "reviewer": "Reviewer_Vakx", "summary": "This paper proposes **RMT**, a “reinforcement mid-training” stage placed between pre-training and post-training. RMT targets three issues—**overthinking**, **imbalanced token-entropy**, and **wasted token information**—via: (1) a **dynamic token budget** with a triangular **length reward** to keep chain-of-thought concise, (2) an **entropy-based curriculum** that shifts sampling from easy→hard tokens over time, and (3) a **dual objective** that runs **RL on selected high-entropy tokens** while using **next-token prediction (NTP)** on the rest. On OmniMATH language-modeling, RMT reportedly improves accuracy by **up to +64.91%** while using **≈21%** of the reasoning length vs. baselines, and the mid-trained checkpoints further boost later post-training by **up to +18.76%** on Skywork.", "questions": "Have you tried **sequence-level** or **verifier-based** rewards (beyond exact token match) and, if so, how do they change the efficiency/accuracy trade-off?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly defines **reinforcement mid-training** and motivates why an intermediate stage can help later post-training. \n- The **dynamic token budget** with a **triangular length reward** is an intuitive way to curb overthinking without eliminating reasoning. \n- The **entropy-based curriculum** is straightforward to implement and aligns with progressive difficulty.", "weaknesses": "- The **evaluation scope is narrow**, focusing on math-reasoning (OmniMATH/Skywork) with only **200 evaluation samples** in the LM setup, which limits statistical confidence and domain generality. \n- The **reward is exact next-token match** (Eq. 5), which may bias learning toward surface-form correctness rather than semantic adequacy; the paper does not compare against richer RL signals (e.g., verifier or sequence-level rewards). \n- Several **strong dynamic RL baselines** are missing in experiments (e.g., recent efficient RL variants or learned curricula), even though related work mentions them; this makes the SOTA claim harder to verify.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tTmdEWgXDz", "reviewer": "Reviewer_gdXM", "summary": "This paper introduces Reinforcement Mid-Training (RMT), a distinct stage in the LLM development positioned between pre-training and post-training. The proposed RMT framework addresses key challenges of inefficiency, imbalanced data difficulty, and information underutilization by integrating a dynamic token budget, a curriculum-based adaptive sampling strategy, and a dual training objective. The work demonstrates improvements in both performance and computational efficiency on mathematical reasoning tasks, establishing a foundation for subsequent post-training.", "questions": "- How would the performance of the curriculum learning component change if difficulty were defined by a more semantically aware metric (e.g., reasoning step complexity) instead of token entropy?\n- Could the dynamic token budget be made adaptive to the input's complexity rather than decaying with the training step, for instance, by allocating a larger initial budget for demonstrably harder problems?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper clearly identifies three practical challenges (i.e., overthinking, entropy imbalance, token underutilization) that are highly relevant to current research.\n- The proposed RMT framework is a well-designed solution where each component (DTB, CAS, dual training) directly addresses one of the identified challenges.\n- The empirical results are convincing, showing substantial gains not only in accuracy but also in efficiency.", "weaknesses": "- My main concerns lie in the paper's foundational assumptions about *difficulty* and *efficiency*. The framework relies heavily on token entropy as a proxy for reasoning difficulty, which is a significant simplification. High uncertainty does not always correlate with high cognitive complexity. Similarly, the dynamic token budget enforces a *shorter is better* heuristic, which may not hold for problems that intrinsically require elaborate, long-chain reasoning. These simplified proxies could limit the method's robustness and generalizability, potentially causing it to penalize valid, complex reasoning paths in favor of concise but possibly suboptimal solutions.\n- The experimental validation is predominantly confined to the domain of mathematical reasoning. This narrow focus makes it difficult to assess the framework's effectiveness on other complex reasoning tasks like code generation, legal analysis, or creative problem-solving.\n- The framework introduces a considerable number of new hyperparameters (e.g., budget decay, curriculum transition points, trade-off weights), but there lacks a sensitivity analysis, leaving its practical robustness and ease of implementation in question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)