T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
Authors: Zhenyu Hou
Xin Lv
Rui Lu
Jiajie Zhang
Yujiang Li
Zijun Yao
Juanzi Li
Jie Tang
Yuxiao Dong
Venue: ICML 2025 poster (2025)
Keywords: Language Model Reasoning
Reinforcement Learning
Primary area: deep_learning->large_language_models
TL;DR: We present  a reasoning model T1 to scale reinforcement learning by encouraging exploration and understand inference scaling.

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. However, existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. While reinforcement learning (RL) holds promise for enabling self-exploration, recent attempts yield modest improvements in complex reasoning. In this paper, we present T1 to scale RL by encouraging exploration and understand inference scaling. We first initialize the LLM using synthesized chain-of-thought data that integrates trial-and-error and self-verification. To scale RL training, we promote increased sampling diversity through over-sampling. We demonstrate that T1 with open LLMs as its base exhibits inference scaling behavior and achieves superior performance on challenging math reasoning benchmarks. More importantly, we present a simple strategy to examine inference scaling, where increased inference budgets directly lead to T1’s better performance without any additional verification. The model weights and training data are publicly available at https://github.com/THUDM/T1.

T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
Zhenyu Hou
Xin Lv
Rui Lu
Jiajie Zhang'
Yujiang Li
Zijun Yao
Juanzi Li
Jie Tang'
Yuxiao Dong
Abstract
50
Large language models (LLMs) have demon- strated remarkable capabilities in complex reason- ing tasks. However, existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. While reinforcement learning (RL) holds promise for enabling self- exploration, recent attempts yield modest im- provements in complex reasoning. In this paper, we present Tl scale RL by encouraging ex- ploration and understand inference scaling. We first initialize the LLM using synthesized chain- of-thought data that integrates trial-and-error and self- ceff-verificttion. To scale RL training, we pro- mote increased sampling diversity through over- sampling. We demonstrate that Tl with open LLMM as its base exhibits inference scaling be- havior and achieves superior performance on chal- lenging math reasoning benchmarks. More im- portantly, present simple strategy to exam- ine inference scaling, where increased inference budgets directly lead to T1's better performance without any additional verification. The model weights and training data are publicly available https:/ / /github...../ Com/THUDM/TT
45
50%
Accuracy (%)
40
More Inference Compute
35
30
More Training compute
25
20
1024
2048
8192 max_gen_lengee gen__engthhheeeeeeeeeeeeeegge (tokens)
4096
16384
Figure 1: Training and inference scaling of T1 (Qwen2.5- 32B) AIMR2024 with different max generation budgets (z-axis) and different RL training steps (SFT: 0% RL steps). Tl achieves better performance with increasing generation length. More RL training steps bring more significant infer- ence scaling. Details in Section 4.
tize the imitation learning stage, with efforts dedicated to generating reasoning paths through prompting (Yu et al., 2024; Mitra et al., 2024; Yue et al., 2024) or rejection sam- pling (Yuan et al., 2023), followed by training LLMs to replicate the selected reasoning processes.
Despite these advancements, reinforcement learning (RL) which can enable LLMs to self-explore and learn from feedback has demonstrated greater potential than imitation learning (OpenAI, 2024) for unlocking inference scaling, i.c., test-time scaling of LLMs. However, its development within the research community remains limited date. Pre- vious studies (Shao et al., 2024; Wang et al., 2024; Hou et al., 2024) suggest that RL yields relatively modest per- formance improvements in complex reasoning and lacks scalability compared to its earlier training stages.
1. Introduction
Large language models (LLMs) (Achiam al., 2023; Team et al., 2023; Dubey et al., 2024) have recently exhibited remarkable capabilities in addressing complex reasoning tasks (Shao et al., 2024; Lozhkov et al., 2024; Zhu et al., 2024; Zhou et al., 2024). The chain-of-thought (CoT) paradigm (Wei et al., 2022) has been instrumental in en- hancing L.L reasoning, emphasizing the importance of constructing and refining reasoning paths (Zelikman al., 2022; Gulcehre al., 2023). Recent approaches priori-
Regarding test-time scaling, existing methods typically rely on repeated sampling (Brown et al., 2024), where multi- ple outputs are generated from given policy model and auxiliary verifiers (Snell al., 2024) are used select the best response. However, these approaches do not update to the policy model itself, thus failing to fundamentally improve the reasoning ability of LLMs. Repeatedly sam- pling short responses with verifiers also falls short of the expected inference scaling behavior (OpenAI, 2024). Ide- ally, deeper thinking and longer generation are expected
'Tsinghua University ZhipuAl. Correspondence to: Yuxiao Dong
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
to directly lead to better performance without relying on external signals. Consequently, improving L.M reason- ing through RL scaling and inference scaling remains underexplored challenge.
More importantly, Ti exhibits promising trends in both training and inference scaling. The model weights and the data for SFT and RL training are publicly available at https:/ /tts:///gtthbl httpps///iitabb..coo////////// cOm/THUDM/T1.
In this work, we explore the key factors for scaling RL and enabling test-time scaling behavior in LLM reasoning. We introduce TI, which exhibits superior reasoning capabilitiess trained via RL. We demonstrate its promising inference scaling behavior in reasoning tasks, shown in Figure 1. The core idea of T1 is to encourage extensive exploration in RL to scale its training while applying appropriate penalties to maintain training stability.
2. Building T1 with RL Scaling
2.1. Preliminary
Supervised Fine-Tuning (SFT), In the initial phase of alignment, the pre-trained model is fine-tuned to replicate high-quality demonstration data (e.g.. dialogue, summariza- tion). This process, commonly referred to as SFT, serves as a foundational step for aligning the model's outputs with human like performance.
First, we finetune the LLM using synthesized CoT data with trial-and-error and velffverifi,ation, which helps substan- tially expand the exploration space before RL training. This strategy makes it different from previous works (An al., 2023; Yuan et al., 2023; Zhang et al., 2024) that typically focus on the correct steps but usually overlook the overall thinking process.
Reinforcement Learning from Human Feedback (RLHF). To further align the fine-tuned model me with human preferences, Ouyang al. (2022) proposes the use of RL to maximize reward signal while regularizing the model ne. that is, to optimize the objective:
Next, to scale RL, we promote greater sampling diversity during RL training by oversampling responses for each prompt with high temperature. In addition, adopt token-level entropy bonus and an on-policy KL normaliza- tion strategy to encourage varied token generation. These also help mitigate excessive regularization from the refer- ence. Strict penalties are imposed on repetitive or nonsensi- cal outputs to prevent collapse and stabilize RL training.
To y
x)) x
(Ta) = Eee yyrst r(æ, y) -Blog
Ter (y
Here, r(.) represents the reward function, which evaluates the quality or correctness of each response. takes a prompt x and its corresponding response y as input and produces a scalar reward. The term Teet refers to the reference model, typically the SFT model.
Third, building on this scaled RL training, we propose simple way to measure and understand inference scaling. The idea is to explicitly separate the generation of interme- diate reasoning steps from the final answer. This allows us to manually control the inference budget by truncating the reasoning process and study how inference cost affects LLMs' reasoning performance. Figure illustrates the train- ing and inference scaling behavior of the policy model on AIME2024. With the same amount of inference budget, the performance improves consistently as RL training scales. T1 also demonstrates stronger inference scaling trends as training steps increase. In contrast, the TI-SFT and early- stage policy models (e.g.. 30% RL steps) show marginal improvements even with the max inference budgets.
The general RLHF pipeline proceeds follows: given a prompt x, the policy model тo generates K different re- sponses, denoted as (y1,..., 9K)). The reward function then assigns a scalar reward to each pair (æ, y/). Subse- quently, the policy model Ty is updated via reinforcement learning maximize the objective defined in Eq.
2.2. Scaling Reinforcement Learning for Reasoning
We present T1 to scaling RL with the goal of advancing the reasoning capability of LLMs. The core idea behind T1 is to promote exploration during RL training. To achieve this, we propose to expand the search space of the LLM and encourage diverse reasoning trajectories while maintaining the training stability with proper penalties to stably scale the RL training.
We build T1 on top of open models such as Qwen (Yang et al., 2024a) and GLM (GLM et al., 2024). These non-ol style models are equipped with long thinking through T1's 's RL scaling, without relying on directly-distilles long Chain- of-Thought data. We evaluate the models on college- and competition-lewe math reasoning benchmarks. Experiments show that the T1 models achieve superior performance across all benchmarks. For example, TI with Qwen-32B as its base can outperform the recent Qwen (-Q-32B-Preeien model on MATH500, AIME2024, and Omni- Mani-MATH-500.
The first step of Tl initialize the SFT model with rich reasoning patterns, such as trial-and-error and verification- elements often overlooked in prior works as they tend to emphasize (only) the correct steps while neglecting the un- derlying thought process. The second step is to develop strategies to effectively scale RL training that further con- tributes to the improvement of LLM reasoning. Figure 2 illustrates the overall framework of T1.
2

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Initialize policy
Reinforeement Learning
Scaling response sampling with high temperature. Scal- ing sampling aims capture broad spectrum of reason- ing paths by generating an increased number of responses per prompt during RL training. As indicated in previous work (Li et al., 2024a), the policy model has possessed strong ability in its inherent sampling space, and it is crucial to search extensively for the valuable reasoning path that can enable effective learning for the policy model.
Procupl.
diverse resporses Ke64
Reward
KL
No garhage?
morm
Policy Model
Attemut Reftect
Y1
dz-d
E 0
T
sample
+
:
Answer
yK
dx-d
x -1
yN cx
veliiverily
Rewarding strategy (kat)
Attempts
Solution
(x. (x))))))))
Formally, given prompt æ, we sample K responses and obtain D = [(ar, y)), yz) y2)...., (æt, yk)]. We find that a larger K works better in practice and thus adopt K = 64 in general. Previous works (Touvron et al., 2023; Kazemnejad et al., 2024; Hou et al., 2024) commonly adopt K I 8 except for K = 32 in Qwen2.5-Man (Yang et al., 2024b).
Figure 2: The overall pipeline of RL and T1.
2.2.1. I. INITIALIZING POLICY w/ CoT
The chain-of-thouge (CoT) (Wei et al., 2022) often defines the reasoning paths of LLMs (OpenAI, 2024), influencing the response search space and RL scaling. To encourage exploration and broaden the search space, we first initialize the policy model with diverse reasoning patterns in the form of CoT (e.g.. reflection and verification) in the SFT stage. In doing so, the SFT model is expected to produce responses that incorporate self- corfecorretton from flawed attempts as well as thoroughly-verifieed approaches. The policy model is then used for subsequent RL training. Note that previous works (Zelikman et al., 2022; An et al., 2023) that optimize CoT often undervalue trial-and-error processes, focusing primarily on correct reasoning steps.
To further improve response diversity, we utilize high tem- perature during sampling. A higher temperature flattens the probability distribution, increasing the likelihood of sampling less probable tokens, and thus encouraging the exploration of diverse token sequences. This strategy facili- tates the generation of various reasoning paths and avoids falling into a fixed pattern. Our experiments demonstrate that high sampling temperatures can help training stability and improve the performance gainss in RL training.
1
For optimization, we use the leave-one-ore strategy used in RLOO (Ahmadian et al., 2024) to normalize the rewards:
Specifically, we begin by generating multiple responses from different LLMs as attempts (y1..... yN) for given prompt æ and judge their correctness based on its ground- truth label. Then we prompt an LLM to thoroughly examine each attempt to obtain the critic c; including 1) identifying the nature of the errors for incorrect attempts and reflecting on their underlying causes, and 2) performing a verifica- tion process for correct ones to confirm the validity of the conclusions derived.
K
1
TJ f
(2)
where r, denotes the reward from reward models or the correctness in {0, 1}. Consequently, the policy model can learn from a richer set of experiences, leading to improved generalization and performance on reasoning.
Auxiliary entropy bonus. To encourage LLMs to generate diverse tokens and avoid deterministic response patterns, we incorporate an entropy bonus into the RL loss function. The modified loss function defined as:
To obtain the reasoning path, we further prompt an LLM to incorporate these refined attempts {ær, ye. J both corrected misconceptions and validated reasoning- into a single output. This is used to illuminate the trial-and-error process that can lead from flawed initial attempts to the final correct solution. Furthermore, we find that some constructed CoT' could simply enumerate different approaches and fi- nally present the correct solution. To overcome this issue, we ask an LLM to rewrite the CoT based on the abstracted pattern to obtain the final solution for SFT training.
L = LRL -aH(((--1)))
|v/
= -LR + a
Σ
Σ WEV
To 2. 11) log Ae <aww U1j-1)
(3)
where LRL represents the standard RL loss, a is weighting coefficient, H (T((aa)) is the token-level entropy given the prompt x. V is the vocabulary. The entropy term measures the uncertainty in the token generation process, incentiviz- ing the model to explore tokens of lower probability.
2.2.2. ENCOURAGING EXPLORATION IN RL TRAINING
To scale RL training, introduce strategies encourage exploration. For generation, we adopt a hard sampling strat- egy to promote the policy model to explore as many trajec- tories as possible to reach the correct path. For optimization, we integrate the response entropy bonus as an auxiliary loss and also ease the KL. regularization to facilitate scaling.
On-policy KL normalization. In RL training, the Kullback-Leiller (KL) divergence is used to force the pol- icy model to remain close to the reference model and thus
3

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
prevent forgetting and reward hacking (Ouyang et al., 2022). However, fixed reference anchor could hinder the reward optimization and thus prevent RL scaling. We adopt two strategies to overcome this problem.
identified through language detection and text quality as- sessment. This prevents the generation of incoherent unreadable output. During training, we observe that the policy model can produce fluent but semantically irregular text, leading to a significant increase response entropy. To address this, we implement both rule-based detection of low-quality text and perplexity filtering to en- hance training stability.
First, similar to the reward normalization which scales the rewards to have zero mean, we enforce KL normalization by subtracting the average KL divergence within responses from prompt æ, effectively keeping the final normalized reward centered at zero. Denoting the KL for prompt- lM:l response pair (ær, y/) as de = 3j malized KL is:
log molg,, the nor- mat (x)
This penalty effectively prevents the training process from collapsing and steers the model away from generating out- puts that could undermine the reasoning capabilities.
K d J74
d, = d.
(4)
k
3. Evaluating T1
Additionally, we apply the Exponential Moving Average (EMA) dynamically update the reference model, thus avoiding it lagging too behind the policy model:
We build T1 by using open models, including GLM-4- 9B (GLM et al., 2024), Qwen2.5-14B, and Qwen2.5-
32B (Yang et al., 2024a). We evaluate the performance on widely-used math reasoning benchmarks-AIMM.. Omni- MATH (Gao et al., 2024), MATH (Hendrycks et al., 2021), and GPQA (Rein et al., 2023). Accuracy (Pass@) 1) is used as the primary evaluation metrie. The details of the experi- mental setup is listed in Appendix A.
(er(t) = at - 1) + (1- a)0(t)
where is the decay rate. The EMA provides smoothed es- timate of the policy parameters, serving as stable reference. With these two strategies, policy updates are incremental and controlled, preventing too large shifts that could desta- bilize the model training.
Table I: Experiment results on challenging reasoning bench- marks, We report the Accuracy($5) (%) for all datasets.
2.2.3. PENALIZING UNEXPECTED PATTERNS
Omni-MA
MATH500 AIME
GPQA
Encouraging exploration is essential for discovering effec- tive reasoning strategies. However, it is also crucial to dis- courage unexpected and undesirable response patterns. This helps prevent training collapse and keeps the model from deviating unintended directions. To achieve this, we im- plement straightforward penalty mechanism by assigning negative reward of 11 responses exhibiting common issues such repetition, overlong text, and garbage text, e.g., mixed multilingual content garbled characters.
TH--50
GPT-4o
76.6
9.3
26.8
53.6
Claudc-3.5
78.3
16.0
65.0
Llama-3.3-70B-Instte
73.9
24.2
27.9
50.5
Owee2.5-Matth7BBnnnmee
82.7
16.7
29.7
36.9
ol- e1-preview
85.5
44.6
72.3
OwQ-32B-preview
90.6
50.0
46.6
58.2
GLM-4-9B
50.1
1.7
12.9
30.9
TI-SFT (GLM-4-9B)
60.2
4.1
20.0
37.2
T1 (GIM-4-99B)
65.8
9.2
24.4
38.1
Specifically, the reward function r' is adjusted as follows:
Dwee2.5-114BInsst Instruct
78.9
13.7
30.1
45.5
T1-SFT (Qwen2.5-14B)
77.2
10.3
28.5
42.3
if bad pattern is detected in otherwise.
T1 (Qwen2.5-144) 14BB
87.4
30.5
38.6
48.3
Qwen2.5-33 5-322 5-32B-Instruct
82.8
13.6
33.1
49.5
TI-SFT (Qwen2.5-32B)
83.4
24.9
34,6
49.5
where is the original reward and y is the generated re- sponse. This formulation penalizes responses that exhibit undesirable patterns, guiding the LLM avoid the follow- ing behaviors during training:
Tl (Qwen2.5-32B)
92.4
50.6
49.6
56.1
3.1. Performance Results
Table shows the overall results on major math benchmarks. It is observed that the T1-SFT (without RL scaling) models show promising performance advantages over their original counterparts, respectively. Further with scaling RL, Tl helps achieve significant performance gains over both SFT and original baselines. Specifically, TI (Qwen2.5-32B) trained with RL achieves over 10% improvement on Omni-
Repetition and Overlong Text: Responses containing repetitive n-grams or exceeding predefined maximum length will receive a - reward. This discourages the generation of meaningless verbose and repeated outputs.
Garbage Text: Responses containing mixed languages or garbled characters are also penalized with a -1 reward,

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Generation Length (Token
3500
K-4
87.5
K-4
0.7
K-4
K=16
Km64
Accuracy(95)
85.0
K-64
0.6
Km64
3000
Reward
82.5
0.5
2500
80.0
0.4
0.3
77.5
1500
0.2
100
300
100
200
0.000
0.001
0.002
0.003 0.004
Training steps
Training steps
KL
(a) Response length
(b) Performance on MATH500
(c) KL-reward relation
Figure 3: Training and evaluation with different number of responses (K) sampled in training on TI (Qwen2.5-14B).
MATH-500 and MATH500 and over 20% improvement on AIME over its TI-SFT version, showcasing the substantial contribution of RL in reasoning capabilities.
90
SFT Baseline
80
K-4
K-16
70
K=64
Furthermore, in math-related benchmarks, Tl shows ad- vantages over baselines. For instance, on MATH500, TI (Qwen2.5-32B) achieves a score of 92.4, outperforming the previous best result (90.6). On AIME, the model also achieves super competitive results compared baselines. These results highlight the superiority of our approach and demonstrate its effectiveness in boosting the ability to han- dle complex mathematical reasoning tasks.
Accuracy
50
40
30
20
MAHH-T05OO AIME
GPQA
Figure 4:
Evaluation results of T1 (Qwen2.5-14B) using different numbers of sampled responses (K) in RL training.
While our method is primarily optimized for math-related tasks, it also exhibits out-of-domain (OOD) performance improvements on the GPQA benchmark, where no task- specific optimization is applied the training. But we also observe remarkable performance improvement in these two benchmarks, indicating that the learned reasoning capability can be generalized across different tasks.
Table 2: Effects of sampling parameters in RL training on Ti (Qwen2.5-14m 14B). We use top-p=0.95 for all experiments.
Temperature min-p
MATH500
AIME Omni-MATH MATH-500
0.9
32.0
1.1
0
84.6
29.0
37.8
3.2. The Effect of Encouraging Exploration
1.2
0
86.4
29.3
38.6
1.3
0
84.6
24.3
36.4
Sampling more responses encourages exploration. Fig- ure 3 shows the effect of sampling different number of responses (i.e., K) during RL training. We have the fol- lowing observations: First, sampling more responses during RL training significantly boosts exploration, leading to substantial increase in response length, as shown Figure 3 (a). Second, for fixed number of prompts, more pled responses accelerate performance improvements, with models achieving better accuracy tasks like MATH500 in fewer training steps, as evident in Figure 3 (b). Additionally, models trained with more responses yield higher rewards the same KL divergence and exhibit slower KL growth, as seen Figure (c). This indicates better trade-off between KL and reward.
1.2
0.05
78.8
11.5
31.6
consists of mathematical content and includes almost no science-related data (i.e., physics, chemistry, or biology). TI demonstrates notable performance improvements GPQA. The impact of response sampling particularly pronounced when increasing the number of samples to 64, producing substantial improvement (over 6%), while sampling only 4 responses shows little to no benefit (around 3%).
High temperature sampling benefits RL training. A higher sampling temperature can encourage the policy model generate more diverse responses, but RL training often sensitive the chosen temperature. Table 2 presents the results of using different sampling temperatures. We observe that higher temperatures contribute to more stable training while training with a temperature S 1.0 often col- lapses after just a few steps. This behavior might occur because, at lower temperatures, if the model starts produc-
Figure 4 illustrates the consistent improvements with sam- pling more responses in RL training. It is also observed that sampling additional responses significantly enhances generalization. Although our training data predominantly
5

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Our work (T1): Single Response as Long Thinking
4. Understanding Inference Scaling in T1
Step-1
Step-2
Step-10
With TI, we aim to improve our understanding of inference scaling, which describes how LLMs achieve performance gains from increased compute during inference (Snell et al.,
thinking Yı,
thinking yt2
thinking
2024; OpenAI, 2024). Unlike previous works (Snell et al., 2024; Brown et al., 2024; Kumar et al., 2024), which mainly focus on scaling through repeated sampling, this study in- vestigates a different approach. We explore how a single longer generation (long thinking) affects the correctness of responses and L.MMs reasoning ability.
yt
Summay Model Φ
Answer-2
Answer-10
Previous: Select from Multiple Responses
Response-1
Response-2 2
Response-12 100
Select the Best One as Answer-2
How to measure inference scaling? By scaling RI., TI enables a new perspective for examining inference scaling, Naturally, when we solve complex problem, the reasoning process often involves a sequence of thinking steps. Even if an intermediate step is wrong, it still plays a key role in figuring out the solution. Through reflection, the wrong step can help reevaluate the approach, refine the reasoning pro- cess, and lead to the correct path. That said, each previous step in the (long) sequence is crucial, as the final solution is built upon existing thinking.
Select the Best One as Answer-10
Figure 5: Inference scaling in T1 (Top) and repeated sam- pling in previous works (Down).
ing undesirable patterns-such repetitive or nonsensical outputs-it it becomes increasingly difficult for it to sample coherent response, preventing recovery from such issues.
Furthermore, training with a moderately high temperature (e.g., 1.2) yields better performance, but excessively high temperatures also negatively impacts outcomes. Although previous studies (Nguyen et al., 2024) have proposed using min-p sampling to mitigate the risks associated with higher temperatures, our experiments find that this approach can make the training process more prone to collapse, especially exacerbating repetition during generation. Therefore, we only use top-p with high temperature in our training.
Inspired by this assumption, we outline a simple strategy to analyze the single (long) response generated by T1, as illustrated by Figure 5 (Top). The idea is to truncate each re- sponse from the beginning to different lengths, although re- sponse length can be controlled through specialized prompt- ing or post-RL fine-tuning. This is used to simulate varying scales of inference cost. Specifically, we truncate each T1 response into various lengths of tokens yı,, with ( repre- sents the length of the truncated response. For each of the truncated response Ytl,, we propose to use a summarization model ((yx.) + Ad; to generate the final answer A4,- In the analysis below, we simplify by setting I; as ix 10% of the original length of y. and by using the base model of TI as the corresponding summarization model. Since the sum- marization model can produce complete answers regardless of whether the thinking process has fully concluded, we evaluate Ad, to assess performance on benchmarks.
Table 3: Effects of penalty reward during RL training on T1 (Qwen2.5-14BB) It shows the overlong ratio and accuracy on MATH500. "OverLongRatio" denotes the ratio of generated responses exceeding the configured maximum length.
Penalty
step40
step80
step120
s6ep160
0%
2.6%
1.6%
0.7%
OverLLonRaato 0%
4.1%
16.3%
This simple strategy enables us analyze the relationship between thinking/reasoning length and performance, that is, the effect of inference scaling. Figure 6 shows the re- sult on AIME, Omni-MATH-500, and MATH500, as x-axis representing the average token count of yı, for different responses with respect to each 7,. On each benchmark, per- formance improves consistently as the average number of thinking tokens increases, reflecting the positive impact of longer thinking on LLMs' reasoning performance. Take AIME in Figure 6 for example, the summarization model's accuracy steadily rises from 24% to 50% as more and more thinking tokens are generated. These results indicate that scaling the inference process of LLMs by increasing the number of tokens generated leads to significant improve- ments in reasoning performance. The consistency of this
78.6
80.1
81.2
81.2
Accuracy(%)
79.0
79.2
76.4
Effects of penalty. Table 3 demonstrates the impact of penalizing unexpected patterns during training. Without penalty, we observe that the generation length becomes unstable and grows explosively after approximately 100 steps, leading to substantial deterioration in performance. In contrast, the model quickly stabilizes when the penalty is applied, although there is a minor increase in the overlong ratio during the early training stages. The penalized model maintains appropriate generation lengths and produces text with minimal repetition and noise, leading to consistent performance improvement.
6

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
50
47.7
10.7
50,030.0
49.0
92
Accuracy186
45
14.7
(ccuracy((%)
47.5
40.2
Accuracy/9%)
51.2
45.0
90
50.2 50.3
35
38.0
42.5
43.8
88
30
37.6
40.0
AIME
50.9
MATH500
25
54.3
86
16.
2000
4000
6000
2000
4000
6000
1000
1500
2000
2500 3000
3500
Average tokens
Average tokens
Average tokens
Figure 6: Inference scaling performance of T1 (Qwen2.5-32B) with truncated thinking in AIME,
and
MATH500. r-axis: the number of thinking tokens used for the summarization model & to generate the final answer A.
AIME
Omni-MATH-500
MATH500
50
staps
92
40%
60%
Accuracy (96)
40%
Accuracy (96)
45
40%
990
100%
102% strps
40
Accurat aay
88
50%
30
40
86
200% J0223
84
20 024
2048
4096
8192 16384
1024
2048 max_ger wew_geennngggheeeeeeeeee length (tokens)
4096
8192 16384
1024
2048
4096
8192
16384
max max_gen__aae___eeagaeeeeaeaa Len_leggth (tokens)
max_gen gen tokens
Figure 7: Training and inference scaling of TI (Qwen2.5-32)) 323B at different RL training steps under different maximum generation budgets. r-axis: the maximum allowed length of thinking, and thinking that exceeds the max-length is truncated.
inference scaling effect across different benchmarks sug- gests that longer inference budgets directly contribute to better performance. This also demonstrates the effective- ness of our simple strategy to measure inference scaling.
different patterns: mistake, Try different approach, Verification, and others. This helps study the reasoning be- havior in longer responses. Figure (b) shows the count of different reasoning patterns corresponding to the purple line in Figure 8 (a). Longer responses contain more di- verse reasoning patterns, suggesting the model engaged in trial-and-error problem-solving. An initial drop in reason- ing attempts likely indicates that the model first learned to eliminate redundant steps caused by the SFT. Overall, these results demonstrate that RL training enhances the model's ability leverage increased inference compute for more effective reasoning.
Inference scaling is closely related to RL training scaling, Based on the above strategy, we further study the relation- ship between RL training and inference scaling. Figure illustrates the inference scaling behavior T1 with different RL training compute. The policy model consistently demon- strates improved performance across all three datasets training compute increases (from bottom to top), under the same maximum generation budget. However, both the TI- SFT model and under-trained RL policy models (e.g., 30% steps in RL) show minimal gains when scaling up inference costs (from left to right). More trained RL models (e.g., 60% steps) achieve significant performance gains. This sug- gests that more RL training could activate and enhance the inference scaling property. In addition, we can observe that challenging tasks benefit more from inference scaling. For example, with 100% RL training steps (purple lines), T1 gains a 66% relative improvement (from 30% to 50%) on AIME, 30% (from 38% to 49%) on Omni-math-5000 and 6% (from 86% to 92%) on MATH500.
Case study. As described in Section 1.2.1, the Tl model is capable of exploring different approaches and correcting reasoning errors. We aim to identify the key reasoning steps-those in which the model discovers the correct idea to solve problem-and investigate whether these steps share common characteristies.
Since our model can rectify its mistakes, it is not appropriate to judge the correctness of a reasoning step solely based on whether it leads to the correct final result. To address this, we employ a strategy inspired by truncated inference scaling analysis: step is labeled ditional reasoning step transforms the model's summarized answer from incorrect to correct. Using this approach, we identify 126 instances from the
key step when ad-
We further study the inference behavior during RL training, as shown in Figure 8. First, the policy model is encouraged to produce increasingly longer responses with the presented exploration strategies during training, as seen in Figure 8 (a). Next, we use an LLM to classify each reasoning step into
dataset and 108 instances from the AIME dataset where such transitions occur. We then conduct a frequent word analysis on these
7

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Generation Length (Tokenss)
8-
Frequent words in key steps
Explore w/o Explore
Correct mistake
3000
Try different approach
0.7
Verification
0.6
2500
Counts
0.4
2000
4
0.3
0.2
1500
100
150
100
200
0.0
lers
walt
consider
perha try
suggest
reed
simplity
however
atremate
Training steps
Training steps
(a) Length w/ and w/o encouraging explo- ration during RL training.
(b) The number of different reasoning pat- temss during RL training with exploration.
(c) Word frequency distribution in key rea- soning steps. key step refers to the stop that, if omitted, yiclds an incorrect answer, but when included, leads to the correct result.
Figure 8: A study of the inference behavior of TI (Qwen2.5-14B) during RL training.
key steps, with the results illustrated in Figure 8 (c). No- tably, words such wait, perhaps, and alternate frequently appear in these steps. These terms often signal moments of rethinking or the exploration of alternative approaches. The observation further demonstrates that reflection is a key capability in improving the reasoning ability.
Scaling language models. Scaling is one of the key fac- tors leading to the success of powerful LLMS and provides crucial insights into the continuous improvement. Kaplan et al. (2020); Hoffmann et al. (2022); Du et al. (2024) study the scaling laws for pretraining and demonstrate that scaling model size and training tokens can both lead to predictable improvements. Recently, reinforcement learning as well as test-time scaling for LLMs to boost reasoning capabilities has attracted much attention since the emerge of OpenAI ol (OpenAI, 2024), but are still under-explored in the open community. Gao et al. (2023); Cobbe et al. (2021) explore the scaling laws in reward modeling under a synthetic set- ting and Rafailov et al. (2024a) studies the scaling of direct policy optimization (Rafailov et al., 2024b). Hou et al. (2024) investigates the impact of scaling and shows that tra- ditional methods are not scalable and are far from effective as shown in ol in boosting the reasoning abilities of LLMs. Beyond scaling RL training. inference scaling is also cru- cial yet under-explored. Existing works (Brown et al., 2024; Snell et al., 2024) measure the inference cost by repeated sampling, which heavily relies on external supervision as a verifier and not as scalable as proposed in ol.
Figure in Appendix A illustrates an example of the rea- soning process of T1. Initially, the model attempts to solve the problem until it reaches an intermediate step. It is not confident in the reasoning and then employs a different ap- proach and arrives at the same result. Finally, the model performs an additional verification step to ensure that no potential cases are overlooked. At this point, is confident that it has arrived at the correct answer.
5. Related Work
Language model reasoning. Recent advances in language models in complex reasoning tasks have been remarkable and have shown substantial improvements (Lewkowycz et al., 2022: Shao et al., 2024; Lightman et al.). One line of work involves pretraining large language models (LLMs) on extensive reasoning-relater datasets, such as mathematies and code, which has significantly improved their founda- tional understanding of reasoning tasks (Paster et al., 2023; Shao et al., 2024; Lozhkov et al., 2024). Another line of work focuses on post-trane training strategies. Some works (Yuan et al., 2023; Yue et al.; Zelikman et al., 2022; Li et al., 2024b) focus on synthesizing reasoning-related question- answer pairs and incorporating additional verifiers or critics to improve data quality Additionally, fine-tuning models through reinforcement learning (Ouyang et al., 2022; Bai et al., 2022) enables the model to learn from feedback and self-guided actions. This iterative process allows models to critique and refine their answers, thus improving their mroblem-solving abilities(Shao et al., 2024; Wang et al., 2024; Kazemnejad et al., 2024). In this work, we build the third line of work by scaling RL techniques.
6. Conclusion
In this paper, we present Tl for enhancing large language models' reasoming capabilities through scaled reinforcement learning. By promoting extensive exploration during RL. training while maintaining stability through strategic penal- ties and oversampling, Tl achieves strong reasoning perfor- mance and demonstrates promising test-time scaling behav- ior. We introduce novel approach to measuring inference scaling by analyzing the relationship between reasoning steps and model performance, revealing that increased RL training improves both reasoning accuracy and inference scaling trends. Experimental results demonstrate that TI shows excellent performance and outperforms existing mod- els on challenging reasoning benchmarks.
8

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Impact Statement
Du, Z., Zeng, A., Dong, Y., and Tang, J. Understanding emergent abilities of language models from the loss per- spective. arXiv preprint 24Xi::220031579%, 2024.
This paper advances the field machine learning by improv- ing language model reasoning capabilities through reinforce- ment learning and inference scaling. Our work contributes to the development of AI systems with enhanced reason- ing abilities. We foresee that our work will have positive impacts on research in language model development and applications. The ethical aspects and societal implications of our work align with those commonly associated with advancing the field of machine learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.217585 2024.
Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q. Li, L., Ma, C., Chen, L.,, Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024.
Acknowledgement
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
This work has been supported by the Natural Science Foundation of China (NSFC) 62495063, Tsinghua Univer- sity (Department of Computer Science and Technology) Siemens Ltd., China Joint Research Center for Industrial Intelligence and Internet of Things (JCIIT), and the New Cornerstone Science Foundation through the XPLORER PRIZE.
GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G., Zhao, H., Lai, H., et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
130b
Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint 238.:2308..8998 2023.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint AxXi:2303308874,, 2023.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, Measuring math- ematical problem solving with the math dataset. NeurlPS, 2021.
Ahmadian, A., Cremer, C., Gallé, M., Fadace, M., Kreutzer, J.. Üstün, A., and Hooker, S. Back to basics: Revisit- ing reinforce style optimization for learning from human feedback in llms. In ACL, 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2003.15556, 2022.
An, S., Ma. Z., Lin, Z., Zheng, N., Lou, J.-G., and Chen, W. Learning from mistakes makes Ilm better reasoner. arXiv preprint arXiv:2310. 2023.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.0586.. 2022.
Hou, Z., Du, P., NIu, Y., Du, Z., Zeng, A., Liu, X., Huang, M., Wang, H., Tang, J., and Dong, Does rlhf scale? exploring the impacts from data, model, and method. arXiv preprint arXiv:2412.06000 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. 2020.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, arXiv preprint wxXiv:2407.21788 2024.
arXiv preprint arXiv:2001.088.
Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., and Roux, N. L. Vineppo: Unlocking potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01669, 2024.
Cobbe, K., Kosaraju, V. Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 10.1118, 2021.

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., lqbal, S., Bishop, C.,, Roelofs, R., et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024.
Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algo- rithms. arXiv preprint arXiv:2406.02000 02900, 2024a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 2024b.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C.,, Schlag, I., Gutman-Son Solo, T., et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang. R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-pree proof q&a benchmark. arXiv preprint arXiv:23 2002,, 2023.
Li, C., Wang, W., Hu, J., Wei, Y., Zheng, N., Hu, H., Zhang, Z., and Peng. H. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.044 044006 2024a.
Shao, Z., Wang. P., Zhu, Q. Xu, R., Song, J., Zhang, M., Li, Y. Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Li, J., Beeching, E., Tunstall, L.,, Lipkin, B., Solet- skyi, R., Huang, S. C., Rasul, K., Yu, L., Jiang, A., Shen, Z. Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G., and Polu, S. Numinamath dataset. https::/ggithe wwtp:pojecttneeeteeeoooooeeeo aimo-progrese
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.033 2024.
wwtpppogrressspiieeeeeeeee perize/blen haallooommmnnnnnn recirr/ pdf, numina_datasee pdff 2024b. GitHub repository.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. / 1805, 2023.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference Learning Representations.
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy- J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. Starcoder and the stack v2: The next generation. arXiv preprint arXiv:2402, 19173, 2024.
Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint 24Xv::2402.14830. 2024.
Wang. P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math- Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9426 9439, 2024.
Nguyen, M., Baker, A., Neo, C., Roush, A., Kirsch, A., and Shwartz-Ziv Ziv, R. Turning up the heat: Min-p sampling for creative and coherent Ilm outputs. arXiv preprint arXiv:2407.01082, 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of- thought prompting clicits reasoning in large language models. In Proceedings of the 36th International Con- ference on Neural Information Processing Systems, pp. 24824-24837, 2022.
OpenAL. llms.
Learning to reason with com/ index/ learning weaeinngtt-rrssoooe with wiithhhlmme 11s,, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instruc- tions with human feedback. In Proceedings of the 36th International Conference on Neural Information Process- ing Systems, pp. 27730-27744, 2022.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F,, Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024a.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint 2xXiv:2409.1/222 2024b.
Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. Open- webmath: An open dataset of high-quality mathematical web text. arXiv preprint mxxiv:2310.067:0 2023.
10

---

TI: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Yu, L., Jiang, W., Shi, H., Jincheng, Y., Liu, Z., Zhang. Y., Kwok, J., Li, Z.,, Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large lan- guage models. In The Twelfth International Conference on Learning Representations, 2024.
Yuan, Z.,, Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv preprint arXiv:2308.0 01825, 2023.
Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations.
Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2024.
Zelikman, E., Wu, Y., and Goodman, N. D. Star: Self-taught reasoner. arXiv preprini arXiv:2203. 14465, 2022.
Zhang, D.. Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., and Tang, Rest-mets*; Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.038. 2024.
Zheng, L., Yin, L., Xie, Z., Huang. J., Sun, C., Yu, C., Cao. S., Kozyrakis, C., Stoica, I., Gonzalez, E., al. Efficiently programming large language models using sglang. https://par.ns http:///ww...oooeeeee/////////// 2023.
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learn- ing Representations, 2024.
Zhu, Q. Guo, D., Shao, Z. Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., al. Deepseek-coder- v2: Breaking the barrier of closed-source models in code intelligence, arXiv preprint arXiv:2406.1 1931, 2024.

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
A. Appendix
A.1. Setup
Data construction. The prompts used the training data all come from publicly available datasets, including MATH- train (Hendrycks et al., 2021), and NuminaMath (Li al., 2024b). We split around 12k for the SFT stage and the others for RL training. To prepare the data for reinforcement learning, convert the original instances into (Question, Label) pairs through the following two steps: The first step is answer extraction. Since NuminaMath provides combined rationale and final answer, we employ an LLM to extract the answer the label and shorten it as much as possible. A concise label facilitates more accurate correctness checks by the LLM during later training stages, For MATH-train, we directly use the question and official answer provided in the dataset. The second step is data filtering. For both NuminaMath and MATH-train, we further filter the data to ensure reliability and high quality for RL training. In particular, we remove inaccurate (noisy) labels and overly simple problems. For each question, we generate responses and retain only those instances whose pass rate lies in the interval (0, 6) (where d 0.3 in our experiments). Finally, we got 30k data that can be used for RL training.
An example from A AIME-2022 2024 Question- 15 solved by '1
Let B be set of rectangular boxes with surface area volume Let be radius of smallest sphere that can contain each of the rectangular boxes that are elements of B. The value of ra can be written as P. where and are relatively prime positive integers. Find p
q
o, h, c. From volume constraint: abe = 23, Our goal to minimize a² + under these constraints.
(Problem Analysis) Let dimerssios
aroa constraint: 2(ab + + aej
(Symmetry Attempt) Perhaps the box is close to cube. Let's try as a simpler case. This gives:
27 o²
2/0² 2nc) mnn + 2nc 27 mmb
20
Combined with volume construint a²c = 23, (simplifiaatoon) we get:
a² 27a + 46
[Numerical Trial) Testing 2, (serifioation) this gives = b = 2 and
(Another Approach) Les's consider another approach using Lagrange multipliers minimize a² 1/ subject construints:
Ma,b. pi a² 8 + o Maa de oe- 27) p(abe 23)
--- (hhoayy colculation, of partial derications). Sa & optinal condition.
23 4
[Caleulation) For 2, c=
520
657
16
16
[Veriftcation) Since 637 = 3 x 73 and 61 = 2", these numbers are relatively prime. Therefore, p 657 and 64, giving p + 721.
(Try other values) Wait, let's make sure 'e vonsidened all possibilities. Let's try 1. ..- (derise oquation and calevlate A), but 1 does not satisfy 2
0.122. Therefore, cannot
/23
[Final Auswer)
Answer: 721
Figure 9: An example of T''s response to Question-15 15 in AIME2024. T1 explores multiple solution approaches and employs various verification strategies to ensure the correctness of the final answer.
Training details. For SFT, train the models for three epochs using learning rate 1e-5 with cosine decay scheduling. For RI. training, we sample 64 responses for each prompt and perform policy gradient descent for every 32 prompts. We train the model with 1. 1.5c-6 learning rate and KL set to 2c-4. For the reward function, we use the ground truth, i.c., the correctness of the response, as the metric, assigning a reward of 1 for correct answers and 0 for incorrect ones. Although using a trained reward model generally considered a superior approach due to its case of optimization, we find that using the correctness of response as the reward also performs well for reasoning tasks and helps mitigate issues such as data
12

---

TI: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
distribution shifts and reward hacking. If not specified, the max generation length for training and inference is set 10,240 for GLM---9B and Qwen2.5-14B models and 16,384 for Qwen2.5-32B models.
Evaluation. We evaluate the performance of the models on the test sets of reasoning-relatee datasets, including AIME2024', Omni-MATH (Gao et al., 2024), MATH (Hendrycks et al., 2021), and GPQA (Rein et al., 2023) Accuracy (Pass @ 1) is used as the primary evaluation metric. We report the results of baselines from previous works available. For MATH, we assess performance on subset of the MATH-test set, referred to as MATH500, following the predefined split in Lightman et al.. For Omni-MATH, we sample a smaller evaluation subset by sampling 500 examples called Omni-MATH-500 for efficient yet comprehensive evaluation. GPQA consists of graduate-level problems in biology, physics, and chemistry. For AIME, the official questions released for the year 2024, which consists of 30 problems. We evaluate each model 32 times AIME to get stable results and report the average performance. For all datasets, we use the greedy sampling strategy with SGLANG (Zheng et al., 2023) as the inference engine.
A full example from AIME-2024 Question-15 solved by T1
Question
Let be the set of rectangular boxes with surface area 54 and volume 23 Let be the radius of the smallest sphere that can contain each of the rectangular boxes that are elements of B, The value of r² can be written as P where p and are relatively prime positive integers. Find p + Response from T1
Let's tackle this problem step by step. We are given set of rectangular boxes with surface area 54 and volume 23. Let the dimensions of the box be b, and c. The surface area and volume constraints give us:
aab + be + ac) = 54 = ab + bc + ac = 27
abc 23
Our goal is to find the radius r of the smallest sphere that contain each of these boxes. The smallest sphere that can contain rectangular box is one that contains all the vertices of the box. This means the diameter of the sphere must be at least the space diagonal of the box, which is:
d = 2=+
Therefore, the radius must satisfy:
Va² + 8 2
2r 2 V 02
62
c²
r
2
To find the smallest possible r, we need to minimize a² +2+b b + e² under the given constraints.
First, let's express a 12 + e² in terms of + b c, ab be + ac, and abc. Recall the identity:
(a+b+c) o 12 o² + 2(ab + be + ac)
Rewriting this, we have:
2 + 12 + 22 (a + ++ - 2(ab + be ac)
We know ab + be + ac 27 from the surface area constraint. Let + b Then:
2 + 22 27
54
Now, we need to express s in terms of b, and Since abc = 23, let's consider the symmetric sums. We might try to find directly, but it's not immediately obvious. Perhaps we can use the Arithmetic Mean-Geometric Mean inequality AM 2 GM to find a lower bound for s. Wait, let's apply AM 2 GM to a, and c:
13

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
a + +bb+0 2 abc
1/23 s 3323
3
This gives us lower bound for 8, but we need the exact value to minimize a² + + c². Maybe we can express terms a, b, and e using the given constraints.
Let's consider a, and e as the roots of the cubic equation:
t st² +7t 23
According to Vieta's formulas:
a a+ a+b+c=s 8, ab + bc ac 27, abc 23
Now, we need to find such that this cubic equation has real positive roots. Perhaps we can find the value of that minimizes a² 62 c². From earlier, have:
a² + + = - 54
To minimize a² + we need to minimize s² (since -54 is constant). Therefore, we need to find the minimum value of s such that the cubic equation has real positive roots. Let's consider the cubic function:
((1) 1 st2 271 23
For the cubic to have three real positive roots, certain conditions must be met. However, this approach might be too complicated. Perhaps we can consider specific values of a, b, and e that satisfy the constraints.
Let's try assuming the box is a cube. a = b = c, then:
2(ab bc + ac) 54 = 2(3a²) 54
6a 54 a²
But then the volume is:
abc = a² = 3 = 27
However, the volume given as 23, not 27. So the box cannot be a cube. Perhaps the box close to a cube. Let's let Then the constraints become: Surface area:
27 a²
2(a² 2ac) =
2a² 4ac 54 = a² 2ac 27 = c=
2a
Volume:
27 a²
27 - o²
a²c= 23 = a²
23 =
23
2a
Multiplying both sides by
a (7
a²)
46
27a
0²
46
27a + 46 -6
Now, need to solve the cubic equation a² 27a + 46 0. Let's try to find rational roots using the Rational Root Theorem. The possible rational roots are factors of 46: #1, +23, 446. Trying = 2:
23 2-227 27
+ 46 54 46
So is root. Therefore, factor the cubie:
14

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
(a - 2) + 2a 23) =0
Setting each factor zero:
1. 2 = 2. a² 2a 23 =
Solving the quadratic equation:
a - (2)² ((1)( 23)
-2 + /
92
-2
96
-2 4v6 = -1 2v6
2
Since dimensions must be positive, a= -1 2v6. However, let's first consider If a = 2, then:
2.
27 a²
27
4
23
c=
2a
4
4
Now, let's verify the volume:
23
23
a²c =
= 4
X
=
23
4
4
T satisfy both the surface area and volume constraints. Now, let's calculate a² + b² + c²:
So a c 23
23
529
529
128
529
657
a² b ² 2 2
4
8
+
4
16
16
16
16
16
Therefore, the space diagonal squared is:
657
a²
+ 13 +
So the radius of the sphere is:
657
V 12 2
657
r
2
2
Now, let's simplify 665:
657 = 9 73
V657
v9x 73 3/73
Therefore:
2
3/73
3v
73
9 x 73
657
2
64
64
Now, let's consider whether this the minimal , Could other dimensions yield a smaller 22 Perhaps consider the general case using the identity derived earlier:
a² 2 2 54
To minimize a² b22 + c², we need to minimize 2 which means minimizing 8. From the AM 2 GM inequality, we have:
+ 3 ab 3/23
So the minimum possible value of is 3 2/3 Let's calculate 82 in this case:
2 = 2 = (2/3)) 232/3
15

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Then:
- 9 23 54
a² + + + = 54
But this is not necessarily an integer, and it's not clear this yields a smaller r² than the value we found earlier. Perhaps the box with dimensions 2, 2, 3 gives us the minimal y². Let's consider another approach using Lagrange multipliers to minimize a² 6² c² subject to the constraints ab + be + 27 and abc 23.
Let's set up the Lagrangian:
L(a, b, c, X, = d² +b² + X(ab + bc ac 27) - p(abe 23)
Taking partial derivatives and setting them to zero:
1.
AL
8a = 2a A(b + c) pbc 2. 8b
2b X(a + c) pacc 0.3. de
2c >(a + b) -pab 0 4.
81. ab + bc + ac 27 = 0 81 2X
abc 23 =
From equations (1), (2), and (3):
2a A(b c) pbc
26 X(a + c) pacc O
2c A(a + b) pab = O
Subtracting equation (1) from equation (2):
(2b 2a) \(a + b c) ulac bc)
2(b a) X(a b) 1l(a b)c 0
Simplifying:
2(b a) + X(b - a) - (u a)c =
(b (b-a) a) + - uc]
So either a A Similarly, subtracting equation (2) from equation (3):
pc pc=0.
(2c 26) - X(a + b - a a- c) - u(ab ac) = 0
2 A(b - c) - 4ub - c)a =
Simplifying:
2(c - b) A(c b) -ule - b)a
(c - b) (2 + X -aa]= 0
So either b or 2 + À - paa = 0.
Similarly, subtracting equation (3) from equation (1):
(2a 2c) A( + a a- b) ulbc ab)
c)) A(c a) pp(c a)b 0
Simplifying:
2(a - c) + a(a - c) - 1l(a - c)b = 0
(a c) 12 A pub
16

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
So either or 2 pb 0. From these equations, we have several possibilities:
1. aa c. a = b, 2 + À - pc = 0, 2 + X = 0 = a=b=c
= = a 3. Similar cases leading to
Therefore, the only solution is a = But earlier, we found that a is not possible.
leads a² 7
3, but the volume would be 27, not 23. So c
Therefore, the minimal r² occurs when b.
Thus, the dimensions = 2, b = 2, 2 give us the minimal r² Therefore, ,² - 64
637
Finally, we need to find p + where ,² = and and are relatively prime positive integers. Here, p = 657 and q = 64. Let's check 657 and 64 are relatively prime. The prime factorization of is 2 The prime factorization of 657 is:
657 9 73 = =3² x 73
Since 64 and 657 have no common factors other than 1, they are relatively prime. Therefore, + 657 + 64 = 721.
Wait, let's make sure we've considered all possibilities. Perhaps there's 's smaller y2 with different dimensions. Let's consider the case where a, b, and are all different. From the earlier cubic equation:
3 3-st st²
27t
23 00
We need to find s such that this cubic has three positive real roots. Using the derivative of the cubic:
f(() 3t 2st +
For the cubic to have three real roots, the discriminant of f'(t) must be positive:
D = (-2s) x x 27 4s2 324
Setting > O:
2 324
22 81
9>
Since s - a a and a, b, are positive, The minimum value of is 9, but must be greater than 9 for the cubic to have three real roots. Earlier, we found that Therefore, the minimum value of is approximately 8. 3.5314, which is less than 9.
Therefore, 9.
b + 2 3//33 x 3 x 2.8438 R 8.5314.
This suggests that the cubic may have three real roots when 8 is slightly greater than 8.5314. However, finding the exact value Therefore, the minimal y2 occurs when 2, b Thus, the value of p + q is 657 + 64 721. Let's verify that this is indeed the minimal p
that minimizes a² 62 + c² is complicated.
23 giving 2 657
64
Suppose we try other values of a, b. and that satisfy the constraints. Let's try = From the volume constraint:
23 bc a
abc. 23
23
From the surface area constraint:
17

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
2(ab + be + ac) = 54 = ab + bc + ac 27
Substituting bc 23:
ab + 23 + 27 a(b+
But bc 23, so b and are roots of the quadratic equation:
? - (b c)t bc =
=
t 23 = 0
For and e be real, the discriminant must be non-negative:
1
16
16 - 92a
1x
92
2 0
0
a²
This implies:
16
16 - 92a 2 0 +
16 2
92a
= a² -
a
S
92
23
v 23
But does not satisfy a I Therefore, cannot be 1.
2 23
~
0.4226.
23 This suggests that the minimal ,² occurs when a = 2, 6, 4
Therefore, the value of p q is 657 + 64 = 721.
As we' ve explored, the case where a = = and c= 23 satisfies both the surface area and volume constraints and yields the minimal y Any other dimensions either do not satisfy the constraints or result a larger y². fleeting thought might suggest considering the box oriented differently within the sphere to potentially reduce r, but the minimal radius is inherently tied the space diagonal of the box. Since the space diagonal the longest distance between any vertices, the sphere must have radius of at least half the space diagonal to contain the box. Another brief consideration might involve symmetry or other geometric properties, but the algebraic approach using the given constraints leads us directly to the solution.
4
Thus, the value of y2 is 657 and therefore + 657 + 64 = 721.
64
Answer; # 721
full example from MATH-500 Question- solved by T1
Question
Find the product of the y-coordinates of all the distinct solutions (2,y) for the two equations y - 8 and y² = -5x =5x + 44.
x²
Response from T1 Let's find the product of the y-coordinates of all the distinct solutions (z, v) for the equations = x² and y² -5x 44.
First, we'll express terms of y from the first equation:
8 a ==
x=
Now, let's substitute this expression for x into the second equation y²² = -5x + 44. Wait, but x is in terms of y, so we'll have:
- 44
This gives two cases to consider due to the +:
** Case 1: y² 5 **Caaee *:** y² = +844
18

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Let's tackle Case I first:
y²= -5=
Rewriting it:
y² - 44 5v 48
Multiplying both sides by -1 to make things neater:
44 - =- =5 +8
Now, let's square both sides to eliminate the square root:
(44 1 -ル)= (5 (=5=))²(55/U+8)
Simplifying both sides:
(44 (41- = 25 88)
Expanding the left side:
(44 y2)- (44 12)(44 x²) 1936 88y² + y²
So the equation becomes:
1936 - 88y² +1² = -2 + +8)
Expanding the right side:
1936 88y² + + 25y + 200
Bringing all terms to one side:
3² 88y2 -25 + 1936 200 = 0
Simplifying:
3/ - 88y- 25y + 1736
Now, let's move on to Case 2:
= = 8
Rewriting:
3²- 44 = 5
Squaring both sides:
(5 +8 2
Simplifying:
(32 44)2 25
Expanding the left side:
8² 88y2 + 1936 = +8(
19

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Expanding the right side:
y¹ 88yy + 1936 = 25y + 200
Bringing all terms to one side:
1/ 88y2 25y 1936 200
Simplifying:
y² - 8888 25y + 1736 =
Wait, that's the same quartic equation we got from Case 1! So both cases lead us to the same equation:
88y² 25y +
1736
Now, need find the roots of this quartic equation. Let's try to factor it into two quadratics. Let's assume:
y² - -8y² 25y 1736 = (v² + ay + bo + cy +
Expanding the right side:
(u² + ay
+ ay
6) + +ay d) y² (a + c)y² + (ac d)y² + (ad + bc)y + bd = (a +a++c)y c)y² + (ac d)y² + (ad + bc)y + bd
Setting this equal to the original quartie:
+ (a + c)y² + (ac + b+ d)y² + (ad + bely - bd = .2² -88²² 25y + 1736
Now, let's match the coefficients: 1. Coefficient of g
c 0 c=-a
2. Coefficient of y
ac d -88
Substituting -a
a(-a) +b+ = -88 = aa +b+d=
3. Coefficient of
ad + bc -25
Substituting -a:
ad + b(-a) -25 ad - ab -25 a(d b) 252
4. Constant term:
bd 1736
Now we have:
--o" + bb -88- ald - bd 1736
Let's solve these equations. From the second equation:
20

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
25
ald - b) 25 d
a
So:
25
b
Now, substitute d into the first equation:
25 6
a +b+
-88
a
Simplifying:
25
+ 2b
-88
Multiply both sides by a to eliminate the fraction:
3 + 2ab 25
-88a
Bringing all terms side:
a + 2ob + 88a
25 =
Rewriting:
0 88a
2ab
This is getting messy. Perhaps there's better approach. Let's try finding rational roots using the Rational Root Theorem.
The Rational Root Theorem tells us that any rational root _ of the polynomial y² - 88y² 25y + 1736 has dividing the constant term 1736 and dividing the leading coefficient 1. So possible rational roots are the factors of 1736. Let's factor 1736:
1736 2 868 2 434
2 2 217 2 217
Now, 217 7 31, SO:
1736 = 23 7 x 31
Therefore, the possible rational roots are +1, 47, 48, +14 +28, 131, +55,, -2, 124, 2217, 248, +434, 1788
Testing y =
71 88 72 25 7+ 1736
2401 88 49
175 + 1736
2401
4312
175 1736
=(2401
1736)
(4312 175)
= 4487
-3350
So y is not root.
Testing y = -7:
21

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
(-7)* 88 x (-7)² - 22 x (-7) + 1736 = 2401 - 88 x 49 + 175 + 1736
-2001 4312 + 175 + 1736
(2401 175 + 1736) 4312
4312 4312
So = 7 a root!
Great, we've found one root. Now, let's factor y +
of the quartic polynomial.
Using synthetic division divide y² 88y² 25y
1736 by y 7:
7
1
0
-88
-5
1736
-7
49
273
- 1736
7
-39
248
0
The quotient is و - 7y² 39y + 248.
Now, we need to factor y² - Ty² - 39y + 248. Let's look for rational roots again using the Rational Root Theorem. The possible rational roots are the factors 248:
248 2 x 124 2 x
2 x 31
So the possible rational roots are +2, 4, Testing y =
12, 4248.
18, 31
83 7x 82 39 248 512- 64 248
=12 448 312 + 248
(512 + 248) (448 312)
760 760
So y is root.
Now, let's factor y - 8 out of y² - -yy² -39y + 248 using synthetic division:
8
1
-7
-39
248
8
8
-
248
31
0
The quotient 2 y 31. So, the quartic polynomial factors as:
88y² - 25y + 1736 (y + 7) - -8 + -1311)
Now, let's find the roots of the quadratic y2 + - 31:
= -1 1s
x (-31)
2
-1 VI+ 124
-1 VI25
2
-1
2
22

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
So the roots y =145v5 and y -1-5v5 Therefore, the solutions for y are:
y 8, y + 5
y -v5
y
2
Now, we need verify that these y-values correspond valid solutions (z, y) by checking the original equations. Starting with y === 7: From y x² 8:
-7 = x²2
x² =1 = ==
Now, let's check y² 5x + 44: For 1:
+ 44 4 -5
49 = 39
This is not true.
For x =:
(-7)2=55(-1) + 44 59
49 49
This is true.
So y -7 corresponds to the solution (I,z) (-1, -7).
Next, y 8:
From y = x=² 8:
8 2 -8 26 16 x=14
Now, check y² 5x 44: For 4:
=- -54( 444 = -4 44
64 24
This is not true.
For x = -4
8 =5(((-4) + 44 -1
64 = 64
This is true.
So 8 corresponds to the solution (I, = (-4,8).
-1 5v5
Now, let's consider
2
From y x
-V+5V5
-1+5V5 + 16
15 5
2.2 = y
+8
2
2
2
So:
15 5
2
Now, let's check y² 5x 44: First, compute y²:
23

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
2
1
5V3
y²
2
(-1 + +555
4
10v 5 + 25 x 5
1 - 10 + 125
4
126 10
4
63 - 5
Now, compute -5x + 44:
For 15 5/5
15 + 5V5
5x 44 -5
+44
5(3 V5)
+ 44
2
5(3 v5)
+ 44
2
This is getting messy, and it's clear that verifying this algebraically is quite involved. However, since we obtained y by solving the quartic equation derived from the original equations, and we've already found valid solutions for -
and 8, we can reasonably conclude that 1 + 5/5 and -1 - 5v5
2 are also valid solutions.
2
5
Similarly, for y =
From = x² 8:
2 +8 -1-
-1-555
+ 16
15 - 5
+
2
2
So:
15 5
2
2
Again, checking y² 55x + would be algebraically intensive, but we can accept these as valid solutions based on earlier work. Now, let's find the product of all the distinct y-coordinates:
+155v5
-1 5
V1 7, 8/2 =,
y3 =
2 3/
2
First, let's multiply ys and y::
24

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
-155v5
)
( (-1-5v5
y/ X 2/4
2
2
(-1+ 5v5)
(-1 5v5)
(-1)² (5v5)
2
1
25 x 5
125
-124
4
-31
Now, multiply y1, 92. ya, and y4:
- 5v5
- - - 5v5
y1 X2 (-7) x8x
x
2
2
(-1+ 5V5)
(-1- - 5v5)
(-7) x x8
-12
(-7)x x
4
(-7) 8 (-31)
-(-7) x 8 (-31)
= ---7) x (-248)
= 1736
Alternatively, we can use Vieta's formulas on the quartic equation y² - 88y² - 25y + 1736 = 0. For general quartic equation y² + ay + by² + cy + d = O, the product of the roots is (-1)*d= = d. In our equation, = 1736, so the product of the roots is 1736.
Wait, that's interesting. Even though we had to verify the validity of each root, Vieta's formula confirms that the product of all the roots indeed 1736.
However, perhaps we should double-check that all these y-values correspond valid solutions. We 'e've already verified y 7 and 8. Let's briefly check the other two.
For y -1 + 5v5
2
From y = x
-v+5v5
-1+5V5 + 16
15 15 5V5
x² = +
+8=
2
2
2
So:
15 15+50 5V5
2 +1
2
Now, check y² -5x 44: Compute y²:
25

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
2
1 + 5V5
5²
2
(-1 + 5V55
4
10v 5 + 25 x
4
1 - 10 + 125
126 10
4
63 - 5
15 +555
Now, compute -5x + 44 for x
2
15 + 5V5
-5x + 44 -
+44
2
5(3 + V5)
=-5
+ 44
2
5(3 + V5)
-
+ 44
2
This getting complicated, and perhaps it's time to accept that since these y- y-values satisfy the quartic equation derived from the original system, and we've 've verified the other roots, these are valid solutions.
Similarly, for y -1 5 the calculations would be analogous.
Therefore, the product of the y-coordinates of all the distinct solutions is indeed 1736.
However, perhaps there's is a more straightforward way to find the product without getting bogged down in the algebra. Let's revisit the original equations:
1. y. x² 2. y² = 5x + 44 From the first equation, x² = +
Substitute 222 y + 8 into the second equation:
y² = -5 + 44
But we still have in the equation. Perhaps we can express in terms of from the first equation:
x =
Substitute into the second equation:
2² - -5 (+VV (5 + 8) 44
This gives us tWo cases:
** Case 1:*4 y² = 55 44
*** Case 2:** y2 5
26

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
Let's solve Case 1:
=
Rewriting:
- 41 -
Multiply both sides by -1:
44 44- -
55
Square both sides:
+8)
Expanding the left side:
(44 - y²1² = 1936 88y8 4
So:
1936 88y² +3² +8)
Expanding the right side:
1936 88y² + y² = 25y + 200
Bring all terms to one side:
5/ 88y2 -25 1936 200
Simplifying:
12 - - 25y + 1736
Now, Case 2:
32 = 3V=5
Rewriting:
3² - 44 = 5 +8
Square both sides:
(32 14)2 25(y + 8)
This is the same equation as before:
-888² 25y + 1736
So both cases lead to the same quartic equation. Now, instead of factoring, perhaps we can use Vieta's formulas directly. For the quartic equation D² + ay² + by? + cy + d = 0, the product of the roots is (-1)4d = our equation y 88y² 25y + 1736 0, the constant term d 1736. Therefore, the product of the roots is 1736. This confirms our earlier result.
27

---

T1: Advancing LLM Reasoning through Reinforcement Learning and Inference Scaling
As final check, let's consider the geometric interpretation. The first equation y = x² a parabola opening upwards, and the second equation y² -5 44 can be rewritten parabola when plotted as x y.
44 y² which is downward-opening
Finding their intersections involves solving the quartic equation we derived, and we' ve found all the y-coordinates of the intersection points,
Therefore, the product of the y-coordinates of all the distinct solutions indeed 1736.
While we initially tried to find the solutions by substituting and factoring, the calculations became quite involved. However, using Vieta's formulas provided a much more straightforward path to the solution. So, after all that, the product of the y-coordinates is:
1736
28


Reviews:
[{"review_id": "QmJNRCRe73", "reviewer": "Reviewer_7pmp", "summary": "This paper introduces T1, a method for enhancing LLM reasoning through reinforcement learning with increased exploration and inference scaling. The authors initialize a policy with chain-of-thought data incorporating trial-and-error patterns, promote exploration during RL through response oversampling, and analyze inference scaling by truncating reasoning steps.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback!\n\n### W1: Lack of comparison with other RL methods for LLMs (particularly GRPO) \n\nA1: We appreciate the reviewer's concern regarding the comparison with other RL methods, particularly GRPO and PPO. However, our work, T1, is designed to be independent of any specific RL framework, including GRPO, PPO, and RLOO. The core contribution of T1 lies in developing techniques to scale RL for LLMs, which are agnostic to the underlying RL method. Specifically, the techniques we propose can be integrated with any RL framework to enhance the reasoning abilities of LLMs.\n\nWe chose RLOO as the primary RL framework due to its demonstrated stability and the strong performance observed in prior work. Notably, we find that RLOO and GRPO show similar results in our preliminary tests, reinforcing the idea that the techniques we introduce can be applied to both methods. Furthermore, the main contribution of our paper lies in how we leverage RL scaling to improve reasoning, a key area where existing methods, including GRPO and PPO, have not demonstrated substantial improvement in comparison to our findings.\nWe also conduct experiments using GRPO, and run the experiments using Qwen-14b with k=16 for efficiency. The results are shown as follows\n\n|     | AIME | Omni-math-500 | MATH500 |\n| --- | --- | --- | --- | \n| T1 w/ RLOO (this work) | 23.8 | 36.8 | 83.6 |\n| T1 w/ GRPO | 23.4 | 37.4 | 83.8 | \n\nIt can be observed that T1 with RLOO and T1 with GRPO show similar performance, which demonstrates that the proposed techniques can bring consistent improvement based on all existing RL algorithms, including both RLOO and GRPO. \n\n\n### W2: The contribution is similar to GRPO\n\nA2: While both T1 and GRPO aim to improve RL for language models, our approach fundamentally differs from GRPO by directly targeting exploration scaling rather than refining the RL algorithm itself. \n\n- GRPO focuses on decomposing and improving PPO (e.g., by removing the value network and using group-wise reward normalization), with little emphasis on boosting exploration.\n- In contrast, T1 introduces strategies to help RL scaling for reasoning, such as trial-and-error learning with self-verification through chain-of-thought data, along with entropy bonuses and dynamic KL regularization—to systematically explore a broader reasoning space. \n\nT1 offers orthogonal contributions to algorithms like GRPO or RLOO and provides new insights into how RL can directly enhance the reasoning capabilities of LLMs, including RL training scaling and inference scaling.\n\n### W3: Computational cost\n\nA3: We employ 64 H800 GPUs for all training, with each iteration taking approximately 300 seconds for a 14B parameter model with K=64.  Larger models require higher computational resources and smaller K can lead to faster training. Current open-source improvements in training and inference infrastructure optimization are expected to continuously reduce the compute requirements for RL training and enhance overall training efficiency.\n\n### W4: More details about reward modeling. \n\nA4: We use rule-based reward with the correctness of response as the reward, 1 for correct and 0 for wrong. We ask the model to put the final answer within box and then use LLM to judge whether the model’s response equals to the groundtruth. You can find more details in Appendix A.", "claims_and_evidence": "The paper's empirical results show performance improvements on math reasoning benchmarks. However, the lack of comparison with other RL methods for LLMs (particularly GRPO) significantly undermines the credibility of the advancement claims. Without these comparisons, it's impossible to determine if T1 represents genuine progress over existing approaches.", "methods_and_evaluation_criteria": "The exploration-encouraging RL approach has merit, but its novelty is questionable. The core motivation of encouraging exploration during RL training for better reasoning is critically similar to GRPO's.", "theoretical_claims": "I don't see any theoretical claims to verify.", "experimental_designs_or_analyses": "The experiments focus on benchmark performance but lack comparative analysis against state-of-the-art RL methods for LLMs. This omission is critical since the authors claim to advance LLM reasoning through RL. The ablation studies are informative but insufficient without positioning against relevant baselines like GRPO or PPO.", "relation_to_broader_scientific_literature": "The paper inadequately discusses its relationship to recent work on RL for LLMs. While it mentions techniques like RLOO, it fails to thoroughly compare with and differentiate from other RL methods, which shares remarkably similar goals and motivations.", "essential_references_not_discussed": "GRPO, PPO", "other_strengths_and_weaknesses": "Strengths:\n\nNovel approach to scaling RL for reasoning\nClear experimental demonstration of inference scaling\nStrong empirical results on challenging benchmarks\nPractical implementation on open-source models\n\nWeaknesses:\n\nLimited comparison to alternative RL algorithms\nHeavy focus on math reasoning with less emphasis on other domains\nLimited analysis of the computational costs of the approach\nSome details about reward modeling are underspecified", "other_comments_or_suggestions": "No, I don't have any other comments", "questions_for_authors": "How does the computational cost of T1 compare to other RL approaches for LLMs?\nThe paper mentions sampling K=64 responses which seems computationally intensive.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jlg0sPFmch", "reviewer": "Reviewer_a8EM", "summary": "The paper introduces T1, a novel approach for enhancing the reasoning abilities of large language models (LLMs) by scaling reinforcement learning (RL) and leveraging inference compute. The method begins by initializing the LLM with synthesized chain‐of‐thought data that incorporates trial‐and‐error and self-verification, enriching the model’s reasoning patterns beyond simple correct-step replication. During subsequent RL training, T1 encourages extensive exploration by oversampling diverse responses using high-temperature sampling and incorporates a token-level entropy bonus along with on-policy KL normalization to stabilize training. These techniques boost the model’s ability to generate longer, coherent “thinking” sequences, and unlock an inference scaling property—where increased generation length directly correlates with improved reasoning performance.\n\nEmpirical evaluations are mainly conducted in math reasoning benchmarks (including AIME2024, MATH500, Omni-MATH-500, and GPQA), demonstrating that T1 outperforms both its supervised fine-tuning baseline and other state-of-the-art models including o1-preview and QwQ-32B-preview. The paper also proposes a simple strategy to measure inference scaling by truncating the generated reasoning process and showing that longer reasoning leads to more accurate final answers.\n\n## update after rebuttal\n\nMy review still leans toward acceptance, but it could also be rejected. The major concern about \"All training and most evaluations focus on math word problems and quantitative reasoning\" is partially addressed with a few additional experiments (on one additional logical reasoning task), and there are some interesting findings like \"our experiments indicate that training on mathematical problems generalizes effectively to other domains.\" I hope to see more insights into why and how math reasoning can be generalized to other domains (with more thorough empirical and, if possible, theoretical results). The claim of the next version should be re-defined in the center of math reasoning to avoid any confusion or misinterpretation from audiences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind review and valuable suggestions!\n\n### Q1 & W1: All training and most evaluations focus on math word problems and quantitative reasoning.\n\nThank you for the question. We primarily conduct reinforcement learning (RL) training on math problems due to the following two reasons:\nMath problems can be reliably verified by rules, and thus the reward cannot be hacked during RL training.\nMath data are easily accessible in open-source communities. \nData in other domains tends to be challenging either to verify reliably or to collect comprehensively. Nonetheless, our experiments indicate that training on mathematical problems generalizes effectively to other domains, and we actively evaluate our models in non-mathematical tasks as well, including GPQA and ZebraLogic. We found that training on math data can effectively lead to improvement in other reasoning tasks. \n\n| | GPQA | ZebraLogic |\n| -- | -- | -- |\n| T1-SFT (Qwen-14B) | 42.3 | 15.3 |\n| T1 (Qwen-14B) | 48.3 | 27.8 |\n| T1-SFT (Qwen-32B) | 49.5 | 20.3 |\n| T1 (Qwen-32B) | 56.1 | 27.9 |\n\n\n\n### Q2 & W2: Critical training details, such as infrastructure configuration, compute costs, and detailed training data information, are missing.\n\nWe employ 64 H800 GPUs, with each iteration taking approximately 300 seconds for a 14B parameter model (larger models require more GPUs and higher computational resources). Smaller K can lead to faster training. While training larger models involves substantial costs, smaller models also significantly benefit from RL training. Furthermore, ongoing improvements in training and inference infrastructure optimization within the open-source community are expected to continuously reduce the compute requirements for RL training and enhance overall training efficiency.\nFor the training data, we use open-source data, including MATH-train, and NuminaMath . We split around 12k for the SFT stage and the others for RL training. And we will open-source the data later. You can find more training details in Appendix A.\n\n[1] MATH (train): Measuring Mathematical Problem Solving With the MATH Dataset\n[2] Numina-MATH: https://huggingface.co/datasets/AI-MO/NuminaMath-CoT/\n\n### Q3: Could you provide an estimate of GPU hours or FLOPs used for training?\n\nWe utilize 64 H800 GPUs, with each iteration requiring around 300 seconds for a 14B model. Since RL training combines generation and training processes, we have not recorded exact FLOP measurements.\n\n### Q4: Which models are used for synthetic data generation?\n\nWe leverage multiple models for synthetic data generation. Specifically, we use Qwen-72B-Instruct and Gemini-1.5-Pro to generate initial Chains-of-Thought (CoT), Gemini-1.5-Pro to generate critiques, and finally, the o1-mini model (without hidden CoT) to merge the CoT and critiques into the final answer. We experimented with several other models and discovered that different models excel in different aspects; thus, we utilize a combination of these models to produce high-quality final outputs.\n\n### Q5: Have you conducted RL training on smaller models?\nA5: Currently, our experiments are mainly on 9B to 32B models and all of them shows significant performance improvement. We haven’t conducted RL training on smaller models. We think that we can also achieve observable improvement across different scale of models with RL.\n\n\n### Q6: Is there significant variance across different RL training runs?\n\nIn our experiments, we observed stable and consistent performance across multiple runs. Variance can be further reduced by increasing the sampling response (larger $K$). Overall, the improvements from RL training are consistent and reproducible across different runs.\n\n### Q7: How would model T1 compare with a Supervised Fine-Tuning (SFT) model trained on a larger dataset of human-annotated reasoning chains?\n\nThis is an insightful question and one that we are also exploring. Currently, nearly all available reasoning datasets are synthetically generated by models. Human-annotated datasets, such as GSM8k-train and MATH-train, usually contain short CoTs, leading to inferior performance. Open-source, high-quality, human-annotated reasoning datasets remain scarce, and it is unclear whether human-generated LongCoT data would outperform synthetic or RL-trained models. Annotating large-scale human reasoning data is challenging and costly, but we aim to investigate this further in future work.", "claims_and_evidence": "The paper makes several claims that are largely supported by **empirical evidence**, though a few appear slightly overstated. The primary claim is that the proposed T1 model achieves superior performance on challenging math reasoning benchmarks and exhibits inference scaling behavior. This is backed by Table 1 in the paper, which shows T1 (with RL training) outperforming baseline models on multiple math problem datasets. \n\nAnother core claim is that T1 demonstrates inference scaling, meaning that allowing the model to “think” longer (increased inference budget) directly yields higher accuracy without external verification. The authors support this by conducting experiments in which they systematically truncate the chain of thought at varying lengths and measure performance.\n\nThe claim that T1’s exploration-oriented RL training yields better reasoning is supported by ablations, such as \"Sampling more responses encourages exploration,\" \"High temperature in sampling benefits RL training,\" and \"Effects of penalty.\" \n\nA minor overstatement: the claim that \"T1 achieves superior performance across all benchmarks\" is inaccurate—on GPQA, a baseline model slightly outperformed T1.", "methods_and_evaluation_criteria": "Yes, the paper proposes an appropriate methodology and benchmarks for reasoning tasks.\n- well-chosen benchmarks (AIME2024, MATH500, OmniMath, GPQA) objectively test math complex reasoning\n- key design choices are justified, including synthetic chain-of-thought data, high K sampling (K=64) for RL, and entropy-based exploration\n\nOne major limitation is that the paper focuses heavily on math, training on math, and testing on math, thus limiting claims about general reasoning improvement.", "theoretical_claims": "There are no theoretical analyses in this paper.", "experimental_designs_or_analyses": "Yes, the paper provides robust, well-executed experiments with detailed ablations, for example:\n- comparisons with strong baselines (GPT-4, Claude, QwQ-32B) confirm T1’s effectiveness (Table 1)\n- ablation studies validate key components: Scaling K improves exploration, while penalties prevent training collapse (Figures 3, Table 3)\n- inference scaling is well-tested: Truncation experiments confirm that longer reasoning systematically improves accuracy", "relation_to_broader_scientific_literature": "The paper studies the recent trends of using RL to improve LLM reasoning, i.e., test-time scaling. The paper’s contributions tie into the literature by confirming hypotheses that were hinted at in prior work (that RL and longer reasoning could significantly improve performance) and by introducing techniques (trial-and-error data augmentation, high-volume sampling in RL) that advance the understanding of how RL works.", "essential_references_not_discussed": "This paper addresses one of the most urgent research problems in training thinking models right now. Considering the short period of time this area has taken to raise, the important missing references are mostly the other concurrent works on improving LLM reasoning with RL, for example, DeepSeek's R1, s1, or any other claimed successful replications of o1 or R1. While it's unfair to ask this paper to compare comprehensively with others, we should expect a more thorough discussion should this paper go into ICML.", "other_strengths_and_weaknesses": "Strengths:\n1. A major strength of the paper is how it creatively integrates several strategies – synthetic data generation, large-scale RL training, entropy regularization, etc. – into a cohesive framework. Each component (trial-and-error CoT pre-training, oversampling, penalty) is not entirely new on its own, but the way they are combined to achieve empirical improvements. \n2. The paper’s clarity is generally strong. It is well-structured, first motivating the problem, then describing the T1 approach, and then providing extensive evaluation. The authors also clearly articulate the intuition behind each design choice (e.g., why high-temperature sampling is used and why they expect trial-and-error data to help). The writing is mostly easy to follow, and important points are supported by either references or experimental evidence.\n3. The paper mentions that “The model weights and the data for SFT and RL training will be publicly available. If the authors follow through, this is a strong point of openness. It would allow the community to reproduce and build on T1. Considering the computational intensity of this approach, releasing the trained model and the large synthetic dataset would be highly valuable for further research.\n\nWeaknesses:\n1. A notable limitation is that all training and most evaluations are centered on math word problems and quantitative reasoning. While the paper argues this is a generic “reasoning” improvement, it is possible that some of the gains are specialized in math problem-solving. The trial-and-error CoT data is constructed for math questions; the reward function is based on checking numeric or symbolic answers. The authors do test one general puzzle dataset (GPQA) and see improvements​, which is encouraging, but a broader evaluation would strengthen the claims of general reasoning enhancement. In short, generality is a bit under-tested – the method’s success in math might be partly due to the structured nature of math problems (clear correctness criteria, availability of many similar problems for training). Future work could explore applying T1 to other reasoning benchmarks (e.g., Big-Bench reasoning tasks, logical deduction puzzles) to ensure the approach is universally beneficial.\n2. Some critical training details are missing, including training infra config, compute cost, and more details on the training data. The approach is computationally heavy. RL with K=64 sampled responses per prompt, especially on a 32B model, implies a massive amount of GPU time. The paper doesn’t detail the compute used, but one can infer it is significant. This could be a practical weakness – the barrier to entry for others to apply this method is high. The authors mitigate this by planning to release the model, but training a new model with T1’s approach would be expensive. It’s worth noting that they chose relatively smaller base models (9B, 14B, 32B), likely because scaling to 70B or beyond with such an intensive training loop would be extremely costly. So, the method’s scalability in terms of engineering is a concern.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Some of the questions are similar to the weaknesses or problems I raised above; the authors can combine them to reduce the overall response length.\n1. The paper's evaluation primarily focuses on math benchmarks (MATH500, AIME, Omni-MATH), with a small test on GPQA. Could the authors provide clear results with meaningful baselines on non-mathematical reasoning tasks, such as multi-hop QA, commonsense reasoning, or logical deduction tasks?\n2. Given that RL training was done with K=64 sampled responses per prompt, and models were trained up to 32B parameters, can the authors provide an estimate of GPU hours or FLOPs used for training?\n3. The authors mention that the initial supervised fine-tuning (SFT) was done on synthetic reasoning data with trial-and-error and verification steps. Which model was used to generate this data (e.g., GPT-4, an earlier version of Qwen, etc.)?\n4. The paper evaluates T1 at 9B, 14B, and 32B scales. Did the authors test whether similar RL training benefits models at a smaller scale (e.g., 7B or 3B)?\n5. Did the authors observe high variance in results across different training runs? Does T1 always converge to a strong policy, or did some runs fail to improve significantly?\n6. The authors demonstrate that RL improves reasoning over supervised fine-tuning (SFT) on synthetic data. How would T1 compare to an SFT model trained on a much larger dataset of human-annotated reasoning chains (e.g., thousands of expert solutions rather than synthetic ones)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "khNE5cwtXI", "reviewer": "Reviewer_KYsA", "summary": "The paper proposes an RL based method to improve the reasoning / inference scaling capabilities of LLMs. \n\nThe paper discussed the importance of exploration and proposed an RL objective incorporating an entropy bonus to encourage diverse sampling. The paper also discusses specific tricks, such as format penalty, over-long length penalty, the size of rollout, the sampling temperature, and the min-p, and their impact on RL training. \n\nThe experiment showing the correlation between generation length / accuracy are very interesting.\n\nIt shows results on scaling inference thinking steps, where SFT model has a flat curve, but RL enables model to more effectively use the inference-scaling budget and improve on MATH, AIME, etc. It also proposes a hypothesis that not just correct steps matter, but wrong steps that led to correct solution also matters.\n\nOverall, the paper presents very strong RL results on widely used benchmarks; its empirical analysis are clear and thought-provoking. It proposes novel thesis, and ways to understand RL reasoning training and inference scaling.\n\n\n## update after rebuttal\n\nMy position still stand as accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback!\n\n### Q1: Have you tried other RL algorithms ?\n\nWe also conduct experiments using GRPO, and run the experiments using Qwen-14b with k=16 for efficiency. The results are shown as follows\n\n|     | AIME | Omni-math-500 | MATH500 |\n| --- | --- | --- | --- | \n| T1 w/ RLOO | 23.8 | 36.8 | 83.6 |\n| T1 w/ GRPO | 23.4 | 37.4 | 83.8 | \n\nIt can be observed that T1 with RLOO and T1 with GRPO show similar performance, which demonstrates that the proposed techniques can bring consistent improvement based on all existing RL algorithms, including both RLOO and GRPO. \n\n\n### Q2: Is the warm-start needed?\n\nA2: Yes, a warm-start is important for reinforcement learning (RL) training as it provides a strong initial foundation for reasoning patterns and encourages exploration. In comparison, for example, Qwen2.5-14b-Instruct, which is trained with RL but utilizes short-CoT, significantly underperforms compared to T1, an RL-trained model initialized from a LongCoT SFT model.\nRecent studies, such as DeepSeek-R1-Zero, have shown that RL training from a base model can also yield strong reasoning capabilities and self-reflection patterns. However, training directly from a base model typically requires an extensive training period to develop these cognitive patterns. A warm-start approach positions the model effectively from the outset, allowing continuous optimization and refinement of its reasoning abilities.\n\n\n| Model                       | MATH500 | AIME | Omni-MATH-500 |\n| ----------------------------- | --------- | ------ | --------------- |\n| (RL) Qwen2.5-14B-Instruct        | 78.9    | 13.7 | 30.1          | \n| (SFT) T1-SFT (Qwen2.5-14B)        | 77.2    | 10.3 | 28.5          | \n| (RL) T1 (Qwen2.5-14B)            | 87.4    | 30.5 | 38.6          | \n| -- | -- | -- | -- | \n| (RL) Qwen2.5-32B-Instruct        | 82.8    | 13.6 | 33.1          |\n| (SFT) T1-SFT (Qwen2.5-32B)        | 83.4    | 24.9 | 34.6          |\n| (RL) T1 (Qwen2.5-32B)            | 92.4    | 50.6 | 49.6          |", "claims_and_evidence": "Claim 1:\nExploration matters for RL learning and RL stability. \n\nIt shows empirical evidence that higher temperature in RL sampling, 1.1 and 1.2 gives better training result in Table-2. \nAlso, a greater rollout budget gives significantly better training result as in Figure-4. \nHowever, what I failed to find is an ablation on the entropy penalty in the loss function, which seems to be a core claim of paper.\n\n\nClaim 2:\nInference scaling is key and closely connected to RL reasoning learning. \n\nThe paper proposes a novel way to measure inference scaling by the truncating the thinking budget. \nFigure-7 shows a consistent pattern that for RL models, having longer thinking budget improves reasoning accuracy. And truncating thinking process leads to performance loss for reasoning models.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are widely used and accepted, and makes sense.", "theoretical_claims": "Yes, the claims of adding entropy penalty to RL loss to encourage exploration makes sense.\nThe connection it draws between inference time scaling and reasoning training makes sense.", "experimental_designs_or_analyses": "Yes, i checked the main paper result, Table-1. To obtain stable numbers for small test set such as AIME, it averages the result across many runs.", "relation_to_broader_scientific_literature": "It relates to the Warm-Start RL training in the DeepSeek-R1 technical report.", "essential_references_not_discussed": "Ablation on the token entropy loss seems to be missing and its impact not discussed.\nEffect of the importance of CoT initialization is missing.", "other_strengths_and_weaknesses": "Other Strength:\n1. Proposes to analyze the inference-scaling effect in LLMs by truncating the reasoning trace, and using a summarization model to give final response. It unveils convincing patterns that reasoning models benefit from longer thinking budget, and is well-connected to inference time scaling.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Have you tried other RL algorithms, eg. GRPO, and is the improvement you obtained specific to the algorithm being used. \n\nIs the warm-start needed? The paper claims one distinction it makes is by considering both the correct and wrong steps of reasoning, but there don't seem to be enough evidence backing the claim. \n\nAlso, having a cold start baseline would be helpful to illustrate the importance of your proposed CoT initialization strategy. And if cold-start doesn't work, may be the correct response lie in a region difficult to reach from the base model, it will further illustrate the strength of the warm-up strategy the authors proposed. \n\n\nIs there a plan to release the training code/setup?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "BndyJXKaAL", "reviewer": "Reviewer_8Mb5", "summary": "This paper examines the SFT + RL training pipeline for enhancing reasoning in LLMs. The authors propose several techniques to improve model performance, including SFT with critiques, auxiliary entropy bonuses, high-temperature sampling, Exponential Moving Average (EMA) stabilization, and On-policy KL normalization. These methods aim to refine reasoning quality, stabilize training, and optimize exploration in the RL framework.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind review and feedback!\n \nWe would first like to emphasize that the primary contribution of this work is to propose a framework designed to effectively scale the reinforcement learning (RL) training of large language models (LLMs). This approach significantly enhances reasoning capabilities and offers a unique perspective on understanding inference scaling. Extensive experimental results on math and reasoning benchmarks clearly demonstrate the effectiveness of our method, and we hope you recognize both our contribution and the efficacy of our proposed framework.\n\nAdditionally, we fully agree that comprehensive ablation studies are essential. To thoroughly validate each component's effectiveness, we have included extensive ablation analyses in our paper (see Tables 1, 2, 3, and Figure 2). For your convenience, we re-present these results here:\n\n1. Effects of high temperature\n| Temperature | min-*p* | MATH500 | AIME | Omni-MATH-500 |\n|-------------|---------|---------|------|---------------|\n| 0.9         | 0       | 78.2    | 19.1 | 32.0          |\n| 1.1         | 0       | 84.6    | 29.0 | 37.8          |\n| 1.2         | 0       | 86.4    | 29.3 | 38.6          |\n| 1.3         | 0       | 84.6    | 24.3 | 36.4          |\n| 1.2         | 0.05    | 78.8    | 11.5 | 31.6          |\n\n2. Effects of penalty design\n\n|              | Penalty | step40 | step80 | step120 | step160 |\n|--------------|---------|--------|--------|---------|---------|\n| OverLongRatio | ✔️       | 0%     | 2.6%   | 1.6%    | 0.7%    |\n|              | ✘       | 0%     | 4.1%   | 16.3%   | -       |\n| Accuracy(%)   | ✔️       | 78.6   | 80.1   | 81.2    | 81.2    |\n|              | ✘       | 79.0   | 79.2   | 76.4    | -       |\n\n\n3. Effects of long CoT for SFT. It can be observed that RL with LongCoT (T1) significantly outperforms RL with short CoT (Qwen2.5-Instruct). \n\n| Model                       | MATH500 | AIME | Omni-MATH-500 |\n |-----------------------------|---------|------|---------------|\n| (RL) Qwen2.5-14B-Instruct        | 78.9    | 13.7 | 30.1          | \n| (SFT) T1-SFT (Qwen2.5-14B)        | 77.2    | 10.3 | 28.5          | \n| (RL) T1 (Qwen2.5-14B)            | 87.4    | 30.5 | 38.6          | \n| -- | -- | -- | -- | \n| (RL) Qwen2.5-32B-Instruct        | 82.8    | 13.6 | 33.1          |\n| (SFT) T1-SFT (Qwen2.5-32B)        | 83.4    | 24.9 | 34.6          |\n| (RL) T1 (Qwen2.5-32B)            | 92.4    | 50.6 | 49.6          |\n\n\nFor questions,\n\nQ1: If critiques are incorporated into SFT, what is the learning signal?\n\n- A1: In this work, we employ a prompt engineering pipeline (as detailed in Section 2.2.1) to generate responses accompanied by reflections. Critiques serve as intermediate outputs within this pipeline and are integrated into the final response rather than being presented separately. Consequently, our training employs traditional Supervised Fine-Tuning (SFT) with next-token prediction.\n\nQ2: Is the model explicitly trained to generate critiques alongside answers?\n\n- A2: No, critiques are inherently part of the generated response rather than explicitly trained as a separate output.\n\nQ3: Is there a more powerful model used to generate critiques alongside answers?\n\n- A3: We utilize Qwen-72B-Instruct and Gemini-1.5-Pro to initially generate Chains-of-Thought (CoT). Gemini-1.5-Pro is specifically employed to generate critiques. Finally, the o1-mini model (without access to its hidden CoT) integrates the generated CoT and critiques into the final response. Notably, except for o1-mini, these models demonstrate significantly poorer math reasoning performance compared to T1. Furthermore, since the o1-mini model's CoT is inaccessible, using responses from it without the explicit long CoT results in considerably weaker performance relative to T1.\n\nQ4: What is the prompt used to generate the critiques?\n\n- A4: The prompts used to generate the synthetic responses involve a pipeline of prompting with 3 prompts. We will add them into the appendix in the next version due to the space limit of rebuttal. To generate critiques, the key point is to ask the LLM to “ analyze what's wrong in every given wrong response and try to find the key point in every correct response”.\n\nQ5: How do the authors ensure the correctness of the critiques?\n\n- A5: We do not explicitly verify the correctness of critiques. Instead, we just expect the model to learn and initialize self-reflection behavior in the SFT stage and prioritize accuracy and effectiveness improvements during the Reinforcement Learning (RL) stage.", "claims_and_evidence": "The paper claims that SFT with critiques provides a strong foundation for RL training, but it lacks experimental evidence to support this assertion. Without concrete results, it is unclear how much critiques contribute to improving reasoning performance. Additionally, all proposed techniques—including SFT with critiques, auxiliary entropy bonuses, high-temperature sampling, EMA stabilization, and On-policy KL normalization—require a thorough ablation study to isolate their individual effects and validate their impact on model performance.", "methods_and_evaluation_criteria": "While the proposed methods are conceptually sound, the paper lacks sufficient discussion on their effectiveness. There is minimal empirical analysis or justification for how each technique contributes to overall model improvement. A deeper exploration, supported by quantitative results and comparisons, is needed to validate their impact on reasoning performance.", "theoretical_claims": "This is not a theory paper.", "experimental_designs_or_analyses": "The paper has serious issues in its experimental design. While multiple techniques are proposed, there is insufficient experimental validation to demonstrate the effectiveness of each method. A more rigorous evaluation, including ablation studies and comparative analysis, is necessary to substantiate the contributions of these techniques.", "relation_to_broader_scientific_literature": "This paper has limited contributions in terms of the techniques it talks about.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strength\n1.Explores supervised fine tuning (SFT) with critics + reinforcement learning (RL) as a means to enhance reasoning in LLMs.\n\nWeaknesses\n1.Lack of Empirical Validation – The paper proposes multiple techniques (e.g., SFT with critiques, auxiliary entropy bonus, high-temperature sampling, EMA, On-policy KL normalization) but lacks sufficient experimental results to support their effectiveness. An ablation study is needed.\n\n2.Lack of detailed description of the proposed method.", "other_comments_or_suggestions": "None", "questions_for_authors": "Lack of clarity on how critiques are used for SFT training:\n\n1.If critiques are incorporated into SFT, what is the learning signal?\n\n2.Is the model being explicitly trained to generate critiques alongside answers?\n\n3.Is there a more powerful model used to generate critiques alongside answers?\n\n4.What is the prompt used to generate the critiques?\n\n5.How does the author ensure the correctness of the critiques?\n\nCan the author provide an ablation study for the techniques proposed in the paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces T1, a method to enhance the reasoning capabilities of large language models (LLMs) through reinforcement learning (RL) and inference scaling. The approach involves initializing the model with synthesized chain-of-thought data and promoting exploration during RL training. The paper claims superior performance on (current/competitive) math reasoning benchmarks and introduces a strategy to examine inference scaling.

(+) Empirical Results: The paper provides strong empirical results on challenging benchmarks, showing T1's effectiveness.  
(-) Limited Domain Evaluation: The focus is heavily on math reasoning, with limited evaluation on other reasoning domains, which raises questions about the generality of the approach.  
(-) Comparative Analysis: There is a lack of comparison with other RL methods for LLMs, such as GRPO and PPO, which undermines the claims of advancement.  

* Reviewer 7pmp: Weak reject due to insufficient comparison with other RL methods and limited domain evaluation.
* Reviewer a8EM: Weak accept, appreciating the novel approach but noting the need for broader evaluation.
* Reviewer KYsA: Accept, highlighting the strong empirical results and novel insights into inference scaling.
* Reviewer 8Mb5: Weak reject, citing lack of empirical validation for proposed techniques and insufficient ablation studies.

Overall, I recommend acceptance into the ICML program due to a well-studied method with strong experimental results (within its size/category – but convincing enough) on math reasoning benchmarks.
Decision: Accept (poster)