Demystifying Long Chain-of-Thought Reasoning
Authors: Shiming Yang
Yuxuan Tong
Xinyao Niu
Graham Neubig
Xiang Yue
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Reasoning
Math
Chain-of-Thought
CoT
Supervised Fine-tuning
Reward Design
Primary area: deep_learning->large_language_models

Abstract: Scaling inference compute has become a key driver of advanced reasoning in large language models (LLMs). A proven approach for scaling inference compute is to generate long chains-of-thought (CoTs), enabling models to engage in structured reasoning strategies such as backtracking and error correction. Reinforcement learning (RL) has emerged as a crucial method for developing these capabilities, yet the conditions under which long CoTs emerge remain unclear, and RL training requires careful design choices. In this study, we systematically investigate the underlying *mechanics of long CoT reasoning*—examining the factors that enable models to generate extended reasoning trajectories. Through extensive supervised fine-tuning (SFT) and RL experiments, we identify three key findings: 1) while SFT is not strictly necessary, it significantly simplifies training and improves efficiency; 2) reasoning capabilities tend to emerge with increased training compute but are not guaranteed, making reward shaping essential for stabilizing CoT length growth; and 3) scaling verifiable reward signals is critical for RL, and we find that leveraging noisy, web-extracted solutions with filtering mechanisms shows promising potential, particularly in out-of-distribution (OOD) reasoning tasks such as STEM problem-solving. These insights provide practical guidance for optimizing training strategies to enhance long CoT reasoning in LLMs.

Demystifying Long Chain-of-Thought Reasoning
Shiming Yang 12
Yuxuan Tong"
Xinyao Niu
Graham Neubig 4
Xiang Yue 44
Abstract
1. Introduction
Large language models (LLMs) (Brown et al., 2020; Tou- vron al., 2023; Chowdhery et al., 2023; Anthropic, 2023; OpenAI, 2023) have demonstrated remarkable reasoning in domains like mathematics (Cobbe et al., 2021) and pro- gramming (Chen et al., 2021). A key technique for enabling reasoning in LLMs is thain-of-thougee (CoT) prompting (Wei et al., 2022), which guides models to generate interme- diate reasoning steps before arriving at a final answer.
Scaling inference compute has become a key driver of advanced reasoning in large language models (LLMs). A proven approach for scaling inference compute is generate long chains-of-thought (CoTs), enabling models to engage in structured reasoning strategies such as backtracking and eгror correction. Reinforcement learning (RL) has emerged as a crucial method for developing these capabilities, yet the conditions under which long CoTs emerge remain unclear, and RL training requires careful design choices. In this study, we systematically investigate the underlying mechanics of long the factors that enable models to generate extended reasoning trajectories. Through extensive supervised fine-tuning (SFT) and RL. experiments, we identify three key findings: 1) while SFT is not strictly necessary, it significantly simplifies training and improves efficiency; 2) reasoning capabilities tend to emerge with increased training compute but are not guaranteed, making reward shaping essential for stabilizing CoT length growth; and 3) scaling verifiable reward signals is critical for RL, and we find that leveraging noisy, web-extracted solutions with filtering mechanisms shows promising potential, particularly in out-of-distribution (OOD) reason- ing tasks such as STEM problem-solvine These insights provide practical guidance for optimizing training strategies to enhance long CoT reasoning in LLMs. Our code is available
Despite these advancements, LLMs still struggle with highly complex reasoning tasks, such mathematical competitions (Hendrycks al., 2021), PhD-level scientific QA (Rein al., 2024), and software engineering (Jimenez et al., 2024), even with COT. Recently, OpenAl's ol models (OpenAI, 2024) have demonstrated significant breakthroughs on these tasks. A key distinguishing feature of these models is their ability to scale up inference compute while refining CoT strategies-such as recognizing and correcting mistakes, breaking down difficult steps, and iterating alternative to substantially longer and more struc- tured reasoning processes.
CoT reasoning-examinng
Several efforts have attempted to replicate the perfor- mance of ol models by training LLMs to generate long CoTs (Qwen Team, 2024b; DeepSeek-AI, 2025; Kimi Team, 2025; Pan al., 2025; Zeng et al., 2025). Most of these ap- proaches rely verifiable rewards, such as accuracy based ground-truth answers, to stabilize reinforcement learning (RL) at scale. However, a comprehensive understanding of how models learn and generate long CoT's remains limited. In this work, we systematically investigate the underlying mechanics of long CoT generation. Specifically, we explore:
1) Supervised fine-tuning (SFT) for long CoT's the most direct way to enable long CoT reasoning. We analyze its scaling behavior and impact on RL, finding that long CoT SFT allows models to reach higher performance and also facilitates easier further RL improvements than short CoT.
https::/gitte com/ eddycmu/ demystiEy-long-cee
Project Lead 'IN.AI English Name: Edward Yeo
2) Challenges in RL-driven CoT scaling we observe that RL does not always stably extend CoT length and complex- ity. To address this, we introduce cosine length-scaling re- ward with a repetition penalty, which stabilizes CoT growth while encouraging emergent reasoning behaviors such as branching and backtracking.
"Tsinghua University. Work started when interning CMU. "Cameegi Mellon University. Correspondence to: Xiang Yue <xyue2 @addee.addew.coom andrew.cmu.edu2.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Demystifying Long Mhain-of-Thoughh Reasoning
3) Scaling up verifiable signals for long CoT RL- Verifi- able reward signals are essential for stabilizing long CoT RL. However, scaling up them remains challenging due to the limited availability of high-quality, verifiable data. To address this, we explore the use of data containing noisy, web-extracted solutions (Yue et al., 2024). While these "silver" supervision signals introduce uncertainty, we find that, with appropriate mixture in SFT and filtration in RL, they show promise, especially in iitoo--disrrbbttinn (OOD) reasoning scenarios such as STEM problem solving.
2.3. Reinforcement Learning (RL)
After optional SFT initialization, we frame the generation of long CoT as Markov Decision- Decision-making Process (MDP) and optimize its reward with reinforcement learning.
Reward Function. We define a scalar reward r' designed to encourage correct and verifiable reasoning. We only
consider the outcome-based reward for the final answer produced, and do not consider process-based reward for the intermediate steps. We denote the term (ynswer(y) to capture the correctness of the final solution.
2. Problem Formulation
Policy Update. Policy gradient methods such as Proxi- mal Policy Optimization (PPO) (Schulman et al., 2017) and REINFORCE (Sutton & Barto, 2018) are employed to it- eratively update 0. The resulting updates push the policy to generate tokens that yield higher rewards, potentially favoring longer or more complex reasoning traces.
In this section, we define the notation, followed by an overview of SFT and RL methods for eliciting long CoTs.
Research Aim
Our primary aim is to investigate the mechanics of long chain-of-thought reasoning in LLMs. By systematically analyzing the factors that influence long CoT reasoning, we distill key insights from rigorous ablations and offer practical takeaways for enhancing and stabilizing long CoT performance.
2.4. Training Setup
We adopt Llama-3, 1-8B (Meta, 2024) and Qwen2. -7B-Math (Qwen Team, 2024a) the base models. For both SFT and RL, use the 7,500-sample prompt set of MATH (Hendrycks et al., 2021) training split by default, with which verifiable ground truth answers are provided. For SFT when ground truth answers are available, we synthesize responses by rejection sampling (Zelikman et al., 2022: Dong et al., 2023; Yuan et al., 2023; Gulcehre et al., 2023; Singh al., 2023; Tong et al., 2024). Specifically, we first sample fixed number N of candidate responses per prompt and then filter by only retaining ones with final answers consistent with the corresponding ground truth answers. We also discuss data like WebInstruct (Yue et al., 2024) that is more diverse but without gold supervision signals like ground truth answers in $5. We train the models with the OpenRLHF framework (Hu et al., 2024).
2.1. Notation
Let (x1, ;2;-..,Tz) .n) be a query, and let y (yi, ./.....,........................ ) be the corresponding ouiput sequence. We consider a LLM parameterized by 6, which defines a condi- tional distribution over output tokens: (Ve 1.)
We denote by CyT(y) S y the tokens in the generated output that constitute the chain- ooain-of-thougtt. which often reasoning trace or explanatory sequence. The final "answer" can be separate set of tokens simply the last part of y.
2.5. Evaluation
In this work, we use the term long chain-of- of-huughh (long CoT) to describe an extended sequence of reasoning tokens that not only exhibits a larger-than than-hhhall token length but also demonstrates more sophisticated behaviors such as:
We focus on four representative reasoning benchmarks: MATH-500. AIME 2024, TheoremQA (Chen et al., 2023). and MMLU-Pro-11 1k (Wang et al., 2024a). Given that our training data is primarily in the mathematical domain, these benchmarks provide a comprehensive framework for both in-domain and out-of-domail evaluations. By default, we evaluate the models using a temperature of f = 0.7, a top-p value of 0.95, and a maximum output length of 16,384 to- kens. Please refer to Appendix 1 for further details on the evaluation setup.
1) Branching and Backtracking: The model systematically explores multiple paths (branching) and reverts to earlier points if particular path proves wrong (backtracking).
2) Error Correction:
The model detects inconsistencies or mistakes in its intermediate steps and takes corrective actions to restore coherence and accuracy.
2.2. Supervised Fine-Tuning (SFT)
3. Impact of SFT for long CoT
A common practice is to initialize the policy Ty via SFT (Lamb et al., 2016) on a dataset DSFT = ((1,, M where y; can be normal or long CoT reasoning tokens.
In this section, we compare long CoT and short CoT as different SFT data patterns and different RL initializations.
2

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
CoT Type (Teacher Model)
Training Method
Long CoT (QwQ-32B- Previeew)
Short CoT Sowen225-Math-722-1nisee
& SFT SFF+RL
MATH-500 (Math I.D.)
AIME 2024 (Math O.O.D.)
TheoremOA (STEM)
MMLU-Pro (General)
Accuracy (%)
70
Accuracy (%)
10
28
Accuracy (%)
40
26
60
Accuracy
35
24
55
30
50
45
20
25
0.1
0.3
1
345 10
0.1
0.3
2 345 10
0.1
0.3
1
3 35
10
0.1
0.3
2 345 10
# of SFT Tokens (B)
# of SFT Tokens (B)
# of SFT Tokens (B)
# of SFT Tokens (B)
Figure 1. Scaling curves of cost-training L1 a1amm-3. 3. 1-8B with long CoT and short CoTs. SFT with long CoT's can scale up to a higher upper limit and has more potential to further improve with RI..
3.1. SFT Scaling
3.2. SFT Initialization for RL
To compare long CoT with short CoT, the first step is to equip the model with the corresponding behavior. The most straightforward approach is to fine-tune the base model on CoT data. Since short CoT is common, curating SFT data for it is relatively simple via rejection sampling from isting models. However, how to obtain high-quality long CoT data remains an open question. We begin by distilling from the open-weight TWQ-323-Prres 2024b) because it is cheaper and produce better perfor- mance, which will be further discussed in $3.3.
Since RI. is reported to have a higher upper limit than SFT, we compare long CoT and short CoT as different SFT ini- tialization approaches for RL.
Setup. We initialize RL using SFT checkpoints from $3.1, and train for four epochs, sampling four responses per prompt. Our approach employs PPO (Schulman et al., 2017) with a rule-based verifier from the MATH dataset, using its training split as our RL. prompt set. We adopt our cosine length scaling reward with the repetition penalty, which will be detailed in $4. Please refer to Appendix E.4 for more details the RL setups.
ew (Qwen Team,
Setup. To curate the SFT data, for long CoT, distill from QwW-32B-Previewe for short CoT, we distill from Qwen2. 5 which is a short CoT model at the SOTA level in math rea- soning. Specifically, for long CoT, we perform rejection sampling with N E {32, 64, 128, 192, 256} candidate sam- ples, while for short CoT, we use N E (32, 64, 128, 256). Here, we control SFT datasets with smaller N's subsets of ones with larger N's. In each case, the number of SFT tokens is proportional to N. For fairness, we use the same base model Llama-3. 1 8B (Meta, 2024). Please refer to Appendix E.3 for more details about the SFT setup.
Result. The gap between solid and dashed lines in Figure I shows that models initialized with long CoT SFT can usu- ally be further significantly improved by RL, while models initialized with short CoT SFT see little gains from RL. For example, on MATH-500, RL can improve long CoT SFT models by over 3% absolutely, while short CoT SFT models have almost the same accuracies before and after RL.
ruct (Qwen Team, 2024a),
Takeaway 3.2 for SFT Initialization for RL
SFT with long CoTS makes further RL improvement casier, while short CoTs do not. (Figure I)
Result. Dashed lines in Figure show that as we scale up the SFT tokens, long CoT SFT keeps improving models' lower
accuracies, while short CoT SFT saturates early accuracy level. For example, on MATH-500, long CoT SFT scales up to over 70% accuracy and still has not saturated yet, while short CoT converges under 55% accuracy and increasing SFT tokens from around 0.25B to around 1.5B only improves the accuracy by about 3% absolutely.
3.3. Sources of Long CoT SFT Data
To curate long CoT data, we compare two synthesis ap- proaches: 1) Construct long CoT trajectories by prompting short CoT models to generate primitive actions and sequen- tially join them. 2) Distill long CoT trajectories from ex- isting long CoT models that exhibit emergent long CoT patterns.
Takeaway 3.1 for SFT Scaling Upper Limit
Setup. To construct long CoT trajectories, we developed an Action Prompting framework (Appendix E.8) which defined the following primitive actions: clarify. decompose, solut iution Lon_step, reflection, and answer. We em- ployed multi-step prompting with a short CoT model (e.g..
SFT with long CoT can scale up to a higher perfor- mance upper limit than short COT. (Figure 1)
3

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
a) Math-500 Accuracy
b) Exceed Rate
Terminated CoTs
d) All CoTs
80
0.8
Accuracy
60
Exceed
0.6
M
Length
4k
Length
10k
0.4
2k-
5k
Llama Qwen
20
0.2
0
0.0
Ok-
Ok
0
40
80
120
0
40
80
120
40
80
120
40
80
120
Iterations
Iterations
Iterations
Iterations
Figure 2. Both fine-tuned emergent CoT length scaling past the context window size, resulting in deterioration of Math-500 accuracy. The red points on the charts correspond to the iteration where the accuracy dropped to near zero.
1- 1--8--8BB and Qwen2. 3---tt----- 7B models trained under RL. with the Classic Reward manifested
Qwen2. 1---5---nns-t Inst to sequence these actions, while a stronger model, o1 a1-minm ni ni-0912- generates reflec- tion steps incorporating self-correction. For distilling long the teacher model. In both approaches, we adopt the MATH training set as the prompt set and apply rejection sampling. To ensure fairness, we use the same base model (Llama- 1..ma-3.11 1B) maintain approximately 200k SFT samples, and use the same RL setup as in $3.2.
4. Impact of Reward Design on Long CoT
This section examines reward function design, with a focus on its influence on CoT length and model performance.
CoT trajectories, we use QwQ San0-32-Preveee
4.1. CoT Length Stability
Recent studies on long CoT (DeepSeek-AI, 2025; Kimi Team, 2025; Hou et al., 2025) suggest that models naturally improve in reasoning tasks with increased thinking time. Our experiments confirm that models fine-tuned on long CoT distilled from QwQ-320 1-2--Previee tend to extend CoT length under RL training. albeit sometimes unstably. This instability, also noted by (Kimi Team, 2025; Hou et al., 2025), has been addressed using techniques such as length and repetition penalties to stabilize training.
Result. Table shows that distilled data with an emergent long CoT pattern generalizes better than the constructed pattern, and can be further significantly improved with RL, while the constructed pattern cannot. Models trained with the emergent long Co'i pattern achieve significantly higher accuracies on OOD benchmarks AIME 2024 and MMLU- Pro-1k, improving by 15-50% relatively. Besides, on the OOD benchmark TheoremQA, RL on the long CoT SFT model significantly improve its accuracy by around 20% relatively, while the short CoT model's performance does not change. This is also why conduct most of our exper- iments based on distilled long CoT trajectories.
Setup. We adopt the same RL setup as $3.2. We used a 16K context window size with two different models fine-tuned on long CoT data distilled from QwQ-32B-P Praviee using the MATH train split. The models were Llama3. 1-8B and Qwen2 5 -M Math-78 7B. More details can be found in Appendix E.
Results. We observed that both models increased their CoT length as they scaled, eventually reaching the context win- dow limit. This led to a decline in training accuracy due to CoTs exceeding the allowable window size. Additionally, different base models exhibited distinct scaling behaviors. The weaker L.1 ama -.. 1 1-8B model showed greater fluctu- ations in CoT length compared to Qwen-2 2 S-Matth-- 7B, as illustrated in Figure 2.
Takeaway 3.3 for Long CoT Cold Start
SFT initialization matters: high-quality emergent long CoT pattern leads to significantly better gener- alization and RL gains. (Table 1)
Table 1. Emergent long CoT patterms outperform constructed ones. All the models here are fine-tuned from the base model Llama 1888 with the MATH training prompt set.
We also found that the rate at which CoT's exceeded the context window was correlated with response length (Fig- ure 2). This suggests that feedback from exceeding the limit applied downward pressure on the overall trajectory distribution rather than affecting only a few isolated CoTs. Notably, model might be penalized even without ex- plicit eeceedeexceeenncee penalty due to reward or advantage normalization, both of which are standard in RL frame-
Training
Long CoT
MATH
AIME
Theo.
MMLU
Method
SFT Pattern
500
2024
QA
Pro-Ik
Constructed
48.2
2.9
21.0
18.1
SFT
Emergent
54.1
3.5
21.8
32.0
Constructed
52.4
2.7
21.0
19.2
SFT+RL
Emergent
59.4
4.0
25.2
34.6
works.

---

Demystifying Long Chain-of-Thoune Thought Reasoning
a) Classic Reward
b) Cosine Reward
a) Training Accuracy
b) Response Length
MWyy
0.8
0.6
10k
Reward
0.6
Reward
Accuracy
Length
0.4
0.4
0.2
correct
0.2
Classic Cosine
wrong
0.0
0.0
Ok
8,000
16,000
8,000
16,000
0
40
80
120
40
80
120
Generation Length
Generation Length
Iterations
Iterations
Figure 3. The Classic and Cosine Reward functions. The Cosine Reward varies with generation length.
Figure 4. 1.1 a.lama3. 1-88 trained with length shaping using the Cosine Reward exhibited more stable (a) training accuracy and (b) response length. This stability led to improved performance on downstream tasks (Figure 5). Red points on the charts indicate iterations where training accuracy dropped to near zero.
Takeaway 4.1 for CoT Length Stability
CoT length does not always scale up in stable fashion. (Figure 2)
Result. We found that the Cosine Reward significantly sta- bilized the length scaling behavior of the models under RL, thereby also stabilizing the training accuracy and improving RL efficiency (Figure 4). We also observed improvements in model performance on downstream tasks (Figure 5).
4.2. Active Scaling of CoT Length
We found that reward shaping can be used to stabilize emer- gent length scaling. We designed a reward function to use CoT length as an additional input and to observe a few order- ing constraints. Firstly, correct CoTs receive higher rewards than wrong CoTs. Secondly, shorter correct CoT's receive higher rewards than longer correct CoTs, which incentivizes the model to use inference compute efficiently. Thirdly, shorter wrong CoT's should receive higher penalties than longer wrong CoTss This encourages the model to extend its thinking time it is less likely to get the correct answer.
Takeaway 4.2 for Active Scaling of CoT Length
Reward shaping can be used to stabilize and control CoT length while improving accuracy. (Figure 4, 5)
4.3. Length Scaling at Different Model Sizes
We evaluate the effectiveness of active scaling of CoT length at different model sizes.
We found it convenient use piecewise cosine function, which is easy to tune and smooth. We refer to this reward function as the Cosine Reward, visualized in Figure 3. This is sparse reward, only awarded once at the end of the CoT based on the correctness of the answer. The formula of CosEn can be found in equation in the appendix.
Setup. We ran experiments with the Classic Reward and the Cosine Reward on both Qwen-2. 5-323 and To increase the diversity of scenarios, we used the base variant of the 32B model without any fine-tuning, but used the 1.5B model after fine-tuning on long CoT data distilled from QwQ-32B 3--0-32B-PP- review using the MATH train split. For more details on the training setup, see Appendix E.5.3.
Qwen-2
CosFn Legen Limax rū, r.), CosFn Lyan, Limas, r rt),
ifC 1, if 0, if Lyen Lmax-
R(C, Lggaa= =
Result. We observed that the Cosine Reward stabilized the length scaling under RL and improved the training accuracy (Figure 6). It resulted in better performance on downstream tasks at both model sizes (Table 2 and Figure 7).
Fes
Hyperparameters:
rō/r0" Reward (correct/wrong) for Lgen = 0,
Table 2. Qwen 22 3-3-BB trained under RL directly from the base model without SFT. The three reward types used are the Classic Reward, Cosine Reward, and the Cosine Reward with repetition penalty. The scores are shown for steps 40 and 80.
ri/rE: Reward (correct/wrong) for Lieen Limas,
Te : Exceed length penalty,
Inputs:
C : Correctness (0 or 1),
Classic
Cosine
Cosine + Rep
Lym: Generation length.
Task
40
80
40
80
40
80
AIME 2024
15.0
16.9
18.5
20.4
19.2
20.6
Setup. We ran experiments with the Classic Reward and the Cosine Reward. We used the Llama3. .amaa.1-80 1B fine-tuned on ew using
MATH-500
78.8
79.6
81.1
81.9
80.1
81.8
TheoremQA
35.4
36.6
36.8
38.0
37.7
39.1
long COT data distilled from CWQ-32B-Pres the MATH train split, as our starting point. For more details, see Appendix E.5.2.
MMLU-Pro-Ik
45.6
47.1
48.9
47.1
47.3
45.5
Average
43.7
45.1
46.3
46,9
46.1
46,8
5

---

Demystifying Long Mhain-of-Thougee Reasoning
Reward Type
Classic Cosine Length Scaling
Cosine Length Scaling Repetition Penalty
MATH-500 (Math I.D.)
AIME 2024 (Math O.O.D.)
TheoremQA (STEM)
MMLU-Pro-1k (General)
66
Accuracy (%6
64
Accuracy (%6
Accuracy (96
Accuracy (66
40
62
28
38
60
26
58
4.
24
56
34
22
32
20
20
40
60
80
100
20
60
80
100
20
80
100
20
40
80
100
of Iterations
# of Iterations
# of Iterations
# of Iterations
Figure 5. Performance models trained with different reward functions variety evaluation benchmarks.
Reward Type classic cosine
Training Accuracy
Response Length
Exceed Rate
7,000
Response Length
0.65
Response Length
0.14
6,000
Exceed Rate
0.12
0.60
M M Ma
0.10
MW
5,000
0.08
0.55
0.06
AWA
4,000
0.04
0.50
3,000
0.02
4
40
80
120
40
80
120
40
80
120
step
step
step
Figure 6. Swen-22.. 5-1. 53 trained with length shaping using the Cosine Reward resulted in greater stability in response length and better training accuracy.
Takeaway 4.3 for Length Scaling Different Model Sizes
Takeaway 4.4 for Cosine Reward Hyperparameters
Active length scaling effective different model sizes. (Table 2 and Figure 7)
The Cosine Reward can be tuned incentivize dif- ferent kinds of length scaling behaviors. (Figure 9)
4.4. Cosine Reward Hyperparameters
4.5. Context Window Size
The Cosine Reward hyperparameters can be tuned to shape CoT length in different ways.
We know that longer contexts give a model more room to explore, and with more training samples, the model even- tually learns utilize more of the context window. This raises interesting question necessary to learn to utilize larger context window?
Setup. We set up RL experiments with the same model fine- tuned long CoT distilled from SwQ-32B-Prees but with different hyperparameters for the Cosine Reward function. We tweaked the correct and wrong rewards ro, rf rb rz and observed their impact on the CoT lengths. For more details, see Appendix E.5.4.
ew,
training samples
Setup. We set up 3 experiments using the same start- ing model fine-tuned on long CoT data distilled from OwQ-32B-Previeet with the MATH train split. We also used the latter as our RL prompt set. Each ablation used the Cosine Reward and repetition penalty with different context window size (4K, 8K, and 16K). For more details, see Appendix E.5.5.
Result. We see from Figure the Appendix that if the reward for a correct answer increases with CoT length (rf rf), the CoT length increases explosively. We also that the lower the correct reward relative the wrong reward, the longer the CoT length. We interpret this as a kind of trained risk aversion, where the ratio of the correct and wrong rewards determines how confident the model has to be about an answer for it to derive a positive expected value from terminating its CoT with answer.
Result. We found that the model with a context window size of 8K performed better than the model with 4K, as expected. However, we observed performance was better under 8K than 16K. Note that all three experiments used the
6

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
Reward Type
Classic
Cosine + Rep. Pen.
AIME 2024
MATH-500
TheoremQA
MMLU-Pro-1K
22
Accuracy (%)
3
Accuracy (%)
45
Accuracy (%)
14
12
Accuracy
20
2
40
10
18
35
16
0
0
40
80
120
0
40
80
120
0
40
80
120
40
80
120
step
step
step
step
Figure 7. Qwen-22.. 2.2...1.555 trained with length shaping using the Cosine Reward resulted in better performance on downstream tasks.
same number of training samples (Figure 8). We see this as an indication that models need more training compute to learn to fully utilize longer context window sizes, which is consistent with the findings of (Hou al., 2025).
Reward exerted greater upward pressure on CoT length, leading to increased reward hacking through repetition. This, in turn, required stronger repetition penalty. Future work could further investigate these interactions and explore dy- namic tuning methods for better optimization.
Takeaway 4.5 for Context Window Size
Takeaway 4.6 for Length Reward Hacking
Models might need more training samples learn to utilize larger context window sizes. (Figure 8)
Length rewards will be hacked with enough com- pute (Figure 10), but this can be mitigated using a repetition penalty. (Figure 5)
4.6. Length Reward Hacking
We observed that with enough training compute, the model started to show signs of reward hacking, where it increased the lengths of its CoTs on hard questions using repetition rather than learning to solve them. We also noted a fall in the branching frequency of the model, which we esti- mated by counting the number of times the pivot keyword "alternes errntively,, appeared in the CoT (Figure 10).
4.7. Optimal Discount Factors
We hypothesized that applying the repetition penalty with temporal locality (i.e., a low discount factor) would be most effective, as it provides a stronger learning signal about the specific offending tokens. However, we also observed performance degradation when the discount factor for the correctness (cosine) reward too low.
We mitigated this by implementing simple N-gram repe- tition penalty (Algorithm 1). We observed that the penalty was most effectively applied on repeated tokens, rather than as a sparse reward for the entire trajectory. Similarly, we found that discounting the repetition penalty when calculat- ing the return was effective. Specific feedback about where the repetition occurred presumably made it easier for the model learn not to do it (see more in $4.7).
To optimally tune both reward types, we modified the GAE formula in PPO to accommodate multiple reward types, each with its own discount factor 7: Á, = El WM 7 - For simplicity, we set = 1, which proved effective, though we did not extensively tune this parameter.
Setup. We ran multiple RL experiments with the same 2lama3.1-80 model fine-tuned on Py0-32B-Previee distilled long CoT data. We used the Cosine Reward and repetition penalty but with different combinations of discount factors. For more details, please see Appendix E.5.7.
Setup. We used the L1ama3. . 1-8B model fine-tuned on
long CoT data distilled from QwQ-32B 1000-32B-P-e - We ran two RL training runs, both using the Cosine Reward, but with and without the repetition penalty. For more details, please refer Appendix E.5.6.
The repetition penalty resulted in better downstream task performance and also shorter CoT's meaning there was better utilization of inference compute (Figure 5).
Result.
Result. A lower discount factor is effective for the repetition penalty. A higher discount factor is effective for the correct- ness reward and the exceed length penalty, which allowed the model attempts at finding a solution earlier in the CoT to be adequately rewarded for a correct answer (Figure 6).
Observation. Our experiments revealed a relationship be- tween the repetition penalty, training accuracy, and the Co- sine Reward. When training accuracy was low, the Cosine
We observed a rather interesting phenomenon where de-

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
Context Length (Token)
4096
8192
16384
MATH-500 (Math I.D.)
AIME 2024 (Math O.O.D.)
TheoremQA (STEM)
MMLU-Prr-11 (General)
30
40
Accuracy (96
60
Accuracy (%6
5
28
55
4
Accuracy
26
35-
3
30
80
22
45
25
40
80
120
0
40
80
120
0
40
80
120
0
40
80
120
iterations
# Iterations
of Iterations
of Iterations
Figure Performance of models trained with different context window sizes. All experiments used the same number training samples.
creasing the discount factor ? of the correctness (cosine) reward increased the branching frequency in the model's CoT, making the model quickly give up on approaches that did not seem to lead to correct answer immediately (Fig- ure 11, Extract in Appendix D). We hypothesize that this short-term thinking was due to a relatively small number of tokens preceding the correct answer receiving rewards, which means stepping stones to the right answer are under- valued. Such behavior degraded performance (Figure 6). However, we think this qualitative result might be of poten- tial interest to the research community, due to its similarity to the relationship between behaviors like delayed gratifica- tion and the distribution of rewards given to the biological brain (Gao et al., 2021).
supervision signals (MATH), ground truth answers are used for rejection sampling. In contrast, for data from Webln- struct without fully reliable supervision signals but with much larger scale, we sample one response per prompt from the teacher model without filtration. For RL here, we adopt the same setup as in $3.2, using the MATH training prompt set.
Result. Table 3 shows that incorporating silver-supervised data improves average performance. Adding WebInstruct data to long CoT SFT yields substantial 5-10% absolute accuracy gain on MMLU-Pro- 1k over using MATH alone. Furthermore, mixing MATH and Weblnstruct data achieves the best average accuracy across benchmarks.
Takeaway 4.7 for Optimal Discount Factors
Table 3. Adding data with silver supervision signal is beneficial.
"WebIT" is the abbreviation of WehInstruct.
Different kinds of rewards and penalties have differ- ent optimal discount factors. (Figure 6)
Long Training Method
MATH
AIME
Theo.
MMLU Pro-Ik
AVG
SFT Data
500
2024
QA
SFT
54.1
3.5
21.8
32.0
27.9
100% MATH
SFT + RL
59.4
4.0
25.2
34,6
30.8
5. Scale up Verifiable Reward
100% WebIT
SFT
41.2
0.8
21.9
41.1
26.3
SFT + RL
44.6
1.9
22.5
43.3
28.1
Verifiable reward signals like ones based on ground-truth answers are essential for stabilizing long CoT RL. However, it is difficult scale up such data due to the limited avail- ability of high-quality human-annotated verifiable data. As an attempt to counter this, we explore using other data that is more available despite more noise, like QA pairs extracted from web corpora. Specifically, we experiment with the WebInstruct dataset (Yue et al., 2024). For efficiency, we construct Weblnstr deduplicated subset derived via MinHash (Broder et al., 1998).
50% MATH + 50% WebIT
SFT SET RL
53.6
4.4
23.5
41.7
30.8
57.3
3.8
25.1
42.0
32.1
Takeaway 5.1 for SFT with Noisy Verifiable Data
Adding noisy yet diverse data to SFT facilitates balanced performance across different tasks. (Table 3)
5.2. Scale up RL with Noisy Verifiable Data
We compare two main approaches to obtain rewards from noisy verifiable data: 1) the rule-based verifier that is usually accurate but limited in answer forms; the model-based verifier capable of processing free-form responses but sus- ceptible to hacking.
5.1. SFT with Noisy Verifiable Data
We first explore adding such diverse data to SFT. Intuitively, despite less reliable supervision signals, diverse data might facilitate the model's exploration during RL.
Setup. We experiment with three setups, varying the pro- portion of data without gold supervision signals: 0%, 100%, and approximately 50%. We conduct long CoT SFT by distilling from MWQ-33BBPreve ew. For data with gold
Setup. We implement the model-based verifier by prompting Qwen2. - raw reference solution. To curate SET data, we prompt Llama-3. 1-8B-11 Instruct to extract short-
with the
8

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
form answers and apply rejection sampling with QwQ-32B-Previews Specifically, generate re- sponses per prompt from WebInstruct-462k and discard cases where neither response aligns with the extracted ref- erence answers. This process yields approximately 189k responses across 115k unique prompts. We perform SFT from Llama- 3. 1-8B on the filtered dataset as initializa- tion for RL. We compare the two types of verifiers with two prompt setups: (1) the 462k full set, which contains prompts asking for free-form answers; (2) the 1 115k subset used in SFT, which is filtered for prompts asking for short-form an- swers by the rejection sampling process. Note that we adopt different RL hyperparamters for these two prompt setups. For further details on the model-based verifier, answer ex- traction and RL hyperparameters, please refer Appendix E.6 & E.7 & E.5.8 respectively.
Takeaway 5.2 for RL with Noisy Verifiable Data
To utilize noisy verifiable data in RL, the model- based verifier performs better on unfiltered data, while the rule-based verifier performs better with appropriate filtration for prompts. (Table 4 & 5)
6. Other Factors that Impact Long CoT
6.1. RL Infrastructure for long CoT still in its infancy
We observe that open-source RL. frameworks (e.g. Open- RLHF (Hu al., 2024) tend orchestrate multiple systems optimized for different training and inference workloads, and that this often results in multiple copies of model pa- rameters in memory. Also, algorithms like PPO alternate between these workloads synchronously and sequentially. These factors result in low hardware utilization, which is particularly pronounced in the long CoT scenario due to higher variance in CoT length resulting in stragglers dur- ing inference (Kimi Team, 2025). We also into some difficulties scaling up model size to 32B, and we decided the number of GPUs required was too large to proceed. We eagerly anticipate advances in the MI. and systems area that will undoubtedly accelerate research on long CoTs
Table 4. Performance of RL with different verifiers unfiltered noisy verifiable data.
MATH
AIME
Theo. QA
MMLU Pro- 1k
Verifier Type
500
2024
SFT Initialization
46.6
1.0
23.0
28.3
Rule-Based
45.4
3.3
25.9
35.1
Model-Basee
47.9
3.5
26.2
40.4
6.2. REINFORCE is more tricky to tune than PPO
We also explored REINFORCE++ (Hu, 2025) as a faster alternative to PPO for scaling up data. However, we found to be significantly more unstable than PPO, leading lower training accuracies (Figure 14). As this instability may be due an untuned setup (Appendix E.5.9), we refrain from making general claims about the algorithm. Instead, we present this as an observation that may be useful to the community.
Table 5. Performance of RL with different verifiers on filtered noisy veriable data. The "MATH Baseline" the model trained with SFT and RL on MATH only in Table 3.
MATH
AIME
Theo.
MMLU Pro- 1k
Verifier Type
500
2024
QA
MATH Bascline
59.4
4.0
25.2
34.6
SFT Initialization
46.6
1.0
23.0
28.3
7. Conclusion
Rule-Based
49.3
2.7
27.8
43.5
In this work, we demystify long CoT reasoning in LLMs. We show that long CoT in SFT raises the performance ceil- ing and enhances subsequent RI. improvements. To address instability in CoT length scaling during RL, we introduce a cosine length scaling reward with a repetition penalty. Additionally, we explore the benefits of leveraging noisy web-extracted data, showing that it improves both SFT and RL when used with reference-guided rewards.
Model-Based
47.7
2.3
26.4
42.2
Result. Table 4 & 5 show that, the model-based verifier performs better on the unfiltered WebInstruct-466k prompt set, while the rule-based verifier performs better on the filtered I 15k subset. The disadvantage of rule-based verifier on unfiltered data might be caused by low training accuracy, while the model-based verifier can achieve higher training accuracy, shown in Figure 13. Moreover, compared to the model trained on annottadd verifiable data (MATH), leveraging noisy yet diverse verifiable data with rule-based verifier after filtration significantly boosts performance on OOD benchmarks, with absolute gains of up to 2.6% on TheoremQA and 8.9% on MML.UPrr--kk.
9

---

Demystifying Long Chain-of-Thought Reasoning
Acknowledgement
D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained code, 2021. URL https: /aa.iiivivvv./...//e.aa/aaaaavva /ttp://wwww..../...........oooet wwwg/orgga.a....
The authors would thank Yuanzhi Li for insightful discus- sions on this topic. The authors would also thank the Sim- pleRL team, particularly Weihao Zeng and Junxian He, for sharing their training experiences and experimental obser- vations. Additionally, the authors appreciate Wenhu Chen, Xiaoyi Ren, Chao Li, Ziqiao Ma, Jiayi Pan, Xingyao Wang, and Seungone Kim for their valuable comments and dis- cussions during the early or final stages of the project. Fi- nally, the authors would acknowledge the DeepSeek-RI and Kimik1.55 teams for their technical report releases, which inspired several additional experiment designs of this pa- per. This work was supported in part by Carnegie Bosch Institute Fellowship to Xiang Yue.
Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., and Xia, T. TheoremQA: A theorem-driven question answering dataset. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: wwwpreveeneeeeeeeeeeepeeeeeeeeee net forum? i dd=Wom397PB53
Impact Statement
This paper aims to provide insights into scaling inference compute and training strategies to enable long chain-of- thought reasoning in large language models. The broader impacts of this work primarily relate to the potential for enhanced reasoning and problem-solving capabilities across various domains, where models capable of interpretable and verifiable reasoning could drive innovation and improve decision-makinn makinn. Our findings emphasize the importance of ensuring robust training data preparation, stability, and align- ment with verifiable ground truths. We encourage future research to actively develop safeguards that ensure these ca- pabilities are used responsibly. This includes careful design of reward shaping and training protocols to minimize unin- tended consequences while maximizing societal benefits.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung. H. W., Sutton, C., Gehrmann, S., al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-13,, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14/68, 2021.
Dao, T. FlashAttention-2:: Faster attention with better paral- lelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
DeepSeek-AL Deepseek-rl: Incentivizing reasoning ca-
pability in llms via reinforcement learning, 2025. URL https: http///axivvv.. http:///awwxv....////////ee 22tp:/149........... 12948.
References
Anthropic. Introducing claude, 2023.
URL
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., KaShun, S., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
https: / htt:///wantthht anthropic.................... com/ index/
introducing- claude.
Broder, A. Z., Charikar, M., Frieze, A. M., and Mitzen- macher, M. Min-wise independent permutations. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pp. 327-336, 1998.
Feng, X., Wan, Z., Wen, M., McAleer, S. M., Wen, Y., Zhang, W., and Wang, Alphazero- like ree-seeach can guide large language model decoding and training. 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Gao, Z., Wang, H. Lu, C., Lu, T., Froudist-Waidd S., Chen, M., Wang, X.J.,, Hu, J., and Sun, W. The neural basis of delayed gratification. Science Ad- vances, 7449)cabb6661, 2021, doi: 10.1126/sc.adv. abg6611. URL https: /tt:://www. http://www.ieeenc.e.e.... scwence. org/ 11//110.11220 2eee/////iiivvve. abg6611.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, P., Cummings,
Gulcehre, C., Paine, T. L.,, Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest)
10

---

Demystifying Long Chain-of-Thooe Thought Reasoning
for language modeling. arXiv preprint arXiv:2308.0800 2023.
Pushing frontiers in open language model post-training. 2024.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, Measuring mathematical problem solving with the math dataset. In Vanschoren, and Yeung. S. (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks, volume I, 2021. URLhttps:
Lightman, H.,, Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https: petttp:://ppeeeeeeee /openrevineeeeeeeeeeeeeeeeeeeeee net/ forum? id=v8LOpN6ECi.
/ //datasets- wwwpeeeehwaaaaoceeeeeeeeeeeeeeee
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., and Roberts, be be3333eedda7735222000077882222-3 hoe flan collection: Designing data and methods for pdf.
heurips.cc/pappe
effective instruction tuning. In Krause, A., Brunskill, E., Cho, K., Engelhardd, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 22631-22648 PMLR, 23-29 Jul 2023. wwwphhttss//pppceeeeeeee v202 #202/longpre23a. html.
Hou, Z., Lv, X., Lu, R., Zhang, J., Li, Y., Yao, Z., Li, J., Tang, J., and Dong, Y. Advancing language model rea- soning through reinforcement learning and inference seal- ing. 2025. URL https: //tp///aaavvv////////eeeeeaae.l/ org/ abs ab50/ 11651.
2501.
ml press/
Hu, J. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint Stti::2501.02662 2025.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradi- ent descent with warm restarts, 2017. URL https: //arxiv. //aaxiv.orggaaattaee.e.aaaaaaaae www.ooog////..... 1608.660000 03983.
Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Openrlhf: An masy-to-use, scalable and high-performance rlhf framework, 2024. URL https: / //arxiv. org/ abs 2405. 11143.
Meta. Introducing meta llama 3: The most capable openly available Ilm to date., 2024. URL https: //ai waa.metta....
OpenAL. Gpt-4 technical report. arXiv preprint 0rXiv:2303.08774 2023.
Jimenez, C. E., Yang. J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-bench: Can language models resolve worldworld github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https: h http://pppeeeeeeeeeeeeeeee wwtp:/wwwwee...rreeeeeeeeeoooeee id=VTF 6.
OpenAL. Learning to reason with 1lms, 2024. URL
https::/ com/i.co//iiiiiiii /nnex/ddx////// wewpniiggto-resson--aaa wimsh
Ouyang, L.. Wu, J.. Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022.
Kimi Team. Kimi k1.5: Scaling reinforcement learning with Ilms, 2025. URL httpt:ttt//aiiiii org/ abs/ 2501. 12599.
Lamb, A. M., ALIAS PARTH GOYAL, A. G., Zhang, Y., Zhang, S., Courville, A. C., and Bengio, Professor forcing: A new algorithm for training recurrent networks. In Lee, D., Sugiyama, M.. Luxburg, U., Guyon. I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL. https: http:::://proceeeee ngs. neurri oc cc/ / [papee_flles/ppper/00////ee 1602 562266009955440003555000aa22a222 pdf.
Pan, J., Zhang, J., Wang, X., and Yuan, L. Tinyzero, wtt.ggtwwwb.ooo/aaaaaaaaaaiiiiie 2025-01-24.
2025. Accessed:
Qwen Team. Qwen2.5-math technical report: Toward math- ematical expert model via self improvemnt, 2024a. URL httpss//arssiv.orr//bbas.
12122.
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, 2024b. URL /tttts:://wwn/// http:///wwwmm......bbb.. github. io/blog/qwq 32tp://wwwwwb----- (e22b-preview.
Lambert, N., Morrison, J., Pyatkin, V., Huang. S., Ivison, H., Brahman, F., Miranda, L. Liu, A., Dziri, N., Lyu, S., Gu, Y., Malik, S., Graf, V., Hwang, J. D., Yang, J., Bras, R. L., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N. A., Wang, Y., Dasigi, P., and Hajishirzi, Tulu 3:
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero:
Memory optimizations toward training trillion parameter

---

Demystifying Long Chain-of- Thhoughtt Reasoning
models. In SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Wang, Z., Li, Y., Wu, Y., Luo, L.., Hou, L.,, Yu, H., and
Shang, J. Multi-step problem solving through verifier: An empirical analysis on model-induced process supervi- sion, 2024b.
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deep- speed: System optimizations enable training deep learn- ing models with over 100 billion parameters. In Proceed- ings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E.,, Le, Q. V., and Zhou, D. Chain-of-thouglee prompting clicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022.
Rein, D., Hou, B. L., Stickland, A. C., Petry, J., Pang, R. Y., GPQA: A
Dirani, J., Michael, J., and Bowman, graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024. URL https: //openreview. net/ forum? 1 T= 67584b98.
Yu, L., Jiang, W., Shi, H., YU, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //openreviww ew. net forum?rrummmmm Bar.rrmmPP.-NNNNgNNND.....
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https: fttps://arxiva /ttpxvaxiiv//////////e..///aa. org/ /ab/ 7707.0634)
Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, C. Scaling relationship on learning mathematical rea- soning with large language models. arXiv preprint TrXiv:2308.01822 2023.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P. Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A., et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.065885 2023.
Yue, X., Zheng, T., Zhang, G., and Chen, W. Mam- moth2: Scaling instructions from the web. arXiv preprint arXiv:2405.0335 2024.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling 1lm test-time compute optimally can be more effective than scaling model parameters, 2024.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Boot- strapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476 155888 2022.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Zeng, W., Huang, Y., Liu, W., He, K., Liu, Q., Ma, Z.. and He, J. 7b model and 8k examples: Emerging rea- soning with reinforcement learning is both effective and efficient. https: /h/wsttttnnnnn notion. site/ 2025. Notion Blog.
Tong, Y., Zhang, X., Wang, R., Wu, R., and He, DART- math: Difficulty-aware rejection tuning for mathemat- ical problem-solving. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, wttpeerreeeeweeee/eeee.e net//orum?
s
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. SGLang: Efficient execution of struc- tured language model programs. In The Thirty- Annual Conference on Neural Information Processing Systems, 2024. URL https: / opwnreview.ee.eeeooeoee / forum?id-VqkAne
2024. URL
id=z1021oQjDS
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirt Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a. URL ht tps: //openree ew .et/ forum i d==1 33.
12

---

Demystifying Long Chain-of-Thoune Thought Reasoning
A. Related Work
Complex reasoning and chain of thought prompting. Large Language Models (LLMs) have demonstrated remarkable capabilities in various natural language processing tasks, including complex reasoning. A significant advancement in improving LLM reasoning ability is the implementation of Chain of Thought (CoT) prompting (Wei et al., 2022). This technique involves guiding models to generate intermediate reasoning steps, thereby improving their performance tasks that require logical deduction and multistep problem solving. Initial studies (Lambert et al., 2024; Wei et al., 2022; Longpre et al., 2023; Yu et al., 2024) focused on short CoT, where models produce concise reasoning paths to arrive at solutions. Although effective for straightforward problems, short CoT can be limiting when addressing more intricate tasks that necessitate deeper deliberation. OpenAI's ol (OpenAI, 2024) series models were the first to introduce inference-time scaling by increasing the length of the CoT reasoning process. This approach helps L.MSs tackle complex problems by breaking them into finer steps and reflecting during prblem-solvingg leading more accurate and comprehensive solutions. In this work, explore long CoT by identifying key factors that enable models to exhibit this behavior, encouraging advanced reasoning capabilities.
Reinforcement learning for LLM. Reinforcement Learning (RL) has proven effective in enhancing LLM performance across domains, RL techniques, such as Reinforcement Learning from Human Feedback (RLHF), align model outputs with human preferences, improving coherence (Ouyang et al., 2022). Recent studies (Kimi Team, 2025; DeepSeek-Al, 2025; Lambert et al., 2024) leverage RL to enable LLMs to explore reasoning paths autonomously for complex problems. DeepSeek-RI (DeepSeek-Al, 2025) achieves strong performance in mathematics, coding, and reasoning tasks without relying on a trained reward model (Lightman et al., 2024; Wang al., 2024b) or tree search (Feng al., 2023; Snell et al., 2024). Notably, this capability emerges readability. Similarly, Kimi K1.5 (Kimi Team, 2025) enhances general reasoning with RL, focusing multimodal reasoning and controlling thought process length. These works highlight RL's role in optimizing reasoning when intermediate steps are hard to supervise, and only final outcomes are verifiable. Our research share similar setup but with more detail on disentangling how different model behaviors emerge under varying training conditions and initialization strategies.
in base models without supervised fine-tuning, albeit the cost of output
13

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
B. Figures and Tables
10
Resporee Langth
Rewand Rewand
Rewaard
Reward
Rewarr
8:000
caneci wior
2,000
mmo0g
wrong
-10
-10
8,000
12,000
15,000
Generation Length
Generation Length
Generation Length
(a) Response lengths under dif-
(b) Reward
(c) Reward B
(d) Reward
ferent Cosine Rewards
Figure 9. (a) Tuning the hyperparameters of the Cosine Reward results in different length scaling behavior. Note that Reward A results in some performance degradation on downstream tasks due to the model's reduced ability to stop within the context window. (b) Reward
r8 rf 10, rō rī. 0. (c) Reward B: r rf 5, -10, rī. (d) Reward C: rā 10,rf 9, rō -00,
'alternatively' Keyword Count (MATH)
Cosine Reward
20
Count
15
10
20
40
60
80
100
Figure CoT branching frequency, estimated by the keyword count the pivot word "alternatively.. decreased under the Cosine Reward with more training compute. We attributed this, along with increased repetition, to reward hacking.
Pivot Keyword 'alternatively,' Frequency
Keyword Count
80
60
40
20
0.990
0.992
0.994
0.996
0.998
1.000
Correctness Discount Factor
Figure Branching frequency in CoT at different % values. Lowering the discount factor increased branching frequency, causing the model to abandon rooliig-solving approaches more quickly.
14

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
4,500
Response Length
4,000
3,500
3,000
2,500
Cosine
2,000
Cosine (Rep. Penalty)
1,500
20
40
80
# of Iterations
Figure 12. Training response length of models trained with Cosine Reward with and without repetition penalty, We see that repetition penalty reduced the length.
0.8
Training Accuracy
0.6
0.4
Filtered (LLM) Filtered (Rule)
0.2
Unfiltered (LLM)
0.0
40
80
120
160
200
240
# of Iterations
Figure 13. Training accuracies of models using rule-based and L.MMbasdd verifiers on filtered and unfiltered data.
a) Training Accuracy
b) Response Length
8k
0.6
Accuracy
6k
Length
0.4
4k
0.2
Reinforce
2k
0.0
Ok
0
20
40
60
80
0
20
40
60
80
Iterations
Iterations
Figure 14. Reinforce with classic reward shows signs of training instability.
15

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
Table Performance of model trained with different discount factors for the correctness (cosine) reward and repetition penalty. We that different reward types have different optimal values.
Correctness Discount
Repetition Discountt
MATH
AIME
Theo.
MMLU
-500
2024
QA
-Pro-lk
50.4
3.5
20.6
32.4
1.000
55.7
5.0
25.7
34.5
1.000
0,999
58.0
4.6
26.0
36.5
0.99
57.8
3.8
24.5
33.3
0.999
53.5
2.1
19.5
30.7
0.999
0.99
55.2
1.7
18.5
32.0
0.99
0.99
47.9
0.2
15.6
25.5
16

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
C. Algorithms and Formulas
C.1. Cosine Reward Formula
tπ
CusFn(t, T,, Tmin 7mor) Mmmi + (Amaz M/min (1 + cos
(I)
T
The formula above commonly used as the learning rate schedule during gradient descent optimization. was introduced by (Loshchilov & Hutter, 2017).
C.2. N-gram Repetition Penalty
Algorithm 1 N-gram Repetition Penalty
1: Input:
2: sequence of tokens
3: I: sequence length
4: N n-gram size
5: P: penalty value
6: m maximum sequence length
7: Output:
IRm
8: seq < s(1 1] (Extract subsequence of length I}
9: ngrams +- 0 {Set of observed n-grams)
10: r - 0 E m [Initialize reward vector)
11: for + 1 to seel N + 1 do
12: ng < (seqli), segli + 1!..... seqli + N-
- 1)) (Current n-gram}
13:
ig ngrams then
for j jo to N do
15:
r|1] P (Apply penalty)
16: end for
17: end if
18:
ngrams + ngrams U {ng}
19:
end for
20: Output:
17

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
D. Extracts
Short-Term Thinking (% 0.99)
But this case, we saw that the fixed points don't t satisfy the equation
p(p(x)) = x.
Wait, maybe the solutions to p(p(z)) are not the fixed points of but something else. Alternatively, perhaps I can consider that p(p(=)) = x implies that is a bijection, but since p(x) is quadratic, it's not necessarily invertible over real numbers.
Wait, but the problem says that p(p(x:) x has four real solutions, so perhaps there are four values of x that satisfy this equation.
Alternatively, perhaps need to find the roots of
p((()) 0,
and solve for But that seems complicated. Maybe there's better way.
18

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
E. Experimental Setup
E.1. Evaluation Setup
Benchmarks
Below are details our evaluation benchmarks:
MATH-500 (Hendrycks et al., 2021): an in-domain mathematical reasoning benchmark. MATH consists of 12,500 problems from American high school math competitions. For efficiency, we adopt MATH-500, widely-used i.i.d. subset of its test split.
AIME 2024: an out-of domainn mathematical reasoning benchmark consisting of the 30 problems from American Invitational Mathematics Examination (AIME) 2024.
TheoremQA (Chen al., 2023): out-of-domain STEM reasoning benchmark consisting of 800 samples. It covers 350+ theorems spanning across Math, EE&CS, Physics and Finance.
MMLU-Pro-Ik (Wang al., 2024a): an out-of-domain general reasoning benchmark. MMLU-Pro comprises over 12,000 questions from academic exams and textbooks, spanning 14 diverse domains including Biology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Math, Philosophy, Physics, Psychology, and Others. For efficiency, we adopt an 1,000-sample i.i.d. subset of its test split, called MMLU-Pro- l.. We tried to keep the distribution identical to the original one. Figure 15 shows the distribution before/after the downsampling.
MMLU-Pro Test Distribution Before/After Downsampling
MMLU-Pro Test (Original)
MMLU-Pro-1k (Downsampled)
8-
7-
Percentage (96)
2-
1-
20
40
60
80
Source
Figure 15. MMLU-Pro test distribution before/after downsampling for the MMLU-Pro- 1k subset. The subset is i.i.d. the full set.
Statistical Metrics We calculate the average accuracy with at least 4 random seeds. To tame the variance caused by the small size AIME 2024, we sample 16 responses per prompt.
Implementation We adopt the vL.LM library to accelerate the inference and SymEval!, an elaborate answer grader capable of processing complex mathematical objects like matrices and functions, keeping consistent with the sampling and reward
hhttps://gt thub. com hooo///oogyx33110o (atpgyyxx33//ymmvall 1symeval
19

---

Demystifying Long Chain-of- Thought Reasoning
implementation in RL setup. Note that few RL experiments are carried out with earlier version of the grader, causing nuanced performance differences.
E.2. Details about Distillation
To distill long CoT trajectories from QwQ -----------eeee Previex, we adopt the temperature 1.0, the top-p value of 0.95 and the maximum output length of 8192 tokens. Our preliminary experiments show that 8192 tokens show almost the same accuracy with QwQ-323 - MATH-500 as 16384 tokens, while costing significantly less time.
To distill short CoT trajectories from Owen2. value of 0.95 and the maximum output length of 4096 tokens, since Qwen2. . 5- - -nns-ttn---ntttteeeeeeennn limit 4096 tokens and our preliminary experiments observe non-negligible ratio of nonsense output when using = 1.0.
we adopt the temperature t = 0.7, the top-p has a context
Note the data distilled with SGLang (Zheng et al., 2024) with an early version of our code.
When applying rejection sampling. adopt the SymEval verifier the grader.
E.3. Details abour SFT Setup
We OLE (Hu al., 2024) for our SFT experiments. By default, we adopt the SFT hyperparameters in Table 7.
For efficiency, we utilize Flash Attention 2 (Dao, 2024) and ZeRO (Rajbhandari al., 2020) stage based on the DeepSpeed library (Rasley et al., 2020). We uniformly set the micro batch size as since we don't 't observe acceleration when increasing it.
Table 7. SFT Hyperparameters
Batch Size
Context Length
LR
Epochs
256
128K
5c-6
E.4. Details about RL Setup
We use OpenRLHF (Hu et al., 2024) for our RL experiments. When describing hyperparameters, we adopt the same naming conventions as OpenRLHF.
By default, we adopt the PPO algorithm with our cosine length-scaling reward function based on the ruled
E.5. Experiment Hyperparameters
Note that the BS column below refers to both rollout baattuuubttcci ch.size (the number of prompts used in compling-training iteration) and train_bateh.a ze (the number of samples used in a training update) because we adopt the same number for these hyperparameters in all RL setups. Also, the Samples es column refers to the number of samples per prompt.
1. DETAILS OF SECTION 4.1 (CoT LENGTH STABILITY)
SFT Data: Long CoT data distilled from MiQ--32B-Pree ew with the MATH train split.
Table 8. Hyperparameters
Base Model
Rewards
GAE
Episodes
Samples
BS
Epochs
Context Length
LR
KL
A
Prompt: 2048
Actor: 5c-7
Llama3.1-8D
Correct: +1
512
0.01
7 =
Gen: 14336
Critic: 4.5e-6
A
Prompt: 2048
Actor: 5e-7
Awen2.5-Math-
Correct:
512
0.01
7
Gen: 14336
Critic: 4.5e-6
20

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
E.5.2. DETAILS SECTION 4.2 (ACTIVE SCALING CoT LENGTH)
SFT Data: Long CoT data distilled from QwQ 32B PeBB-review with the MATH train split.
Table 9. Hyperparameters
Base Model
Rewards
GAR
Episodes
Samples
BS
Epochs
Context Length
LR
KI.
À
Prompt: 2048
Actor: 5e-7
Llama3.1 Correct: +1
8
8
512
0.01
7
Gen: 14336
Critic: 4.5e-6
Cosine:
ró +2
r7. +1
Prompt: 2048 Gen: 14336
Actor: Se-7 Critic: 4.5e-6
Llama3.1-8B
8
8
512
1
0.01
T = 10
7=1
rI =
re -10
X ==
Prompt: 2048 Gen: 14336
Actor: Se-7 Critic: 9e-6
Llama3. 1-BB Correct: +1
16
512
2
0.01
7
Cosine: ro +2
rf. +1
x=
Prompt: 2048
Actor: 5e-7 Critic: 9e-6
Llama3.1-8B
8
16
512
2
0.01
ro = -10
7=1
Gen: 14336
rI = -0
Te -10
Cosine:
TO +2
TL +1
TO 10
A=1
Prompt: 2048
Actor: Se-7 Critic: 9e-6
Llama3.1-8B
T =
%e =1
8
16
512
2
Gen: 14336
0.01
10
7/p 0.99
Rep. Penalty:
P 0.05
N 40
21

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
E.5.3. DETAILS SECTION 4.3 (LENGTH SCALING DIFFERENT MODEL SIZES)
SFT Data:
Owen-2. 52B:: The base model was trained under RL without SFT stage.
Qwen-2.5- 5-1. 5B: Long CoT data distilled from BWQ-32B-Freviie with the MATH train split.
Table I0. Hyperparameters
Base Model
Rewards
GAE
Episodes
Samples
BS
Epochs
Context Length
LR
KL
Actor: 5c-7
= 1 7
Prompt: 2048
Critic: 9c-6
Qwen-2.5-32B
Correct:
1
8
512
Gen: 14336
Constant after
0.01
10-step linear warmup
Cosine:
ro +2 ri = +1
Actor: Se-7
À = % =1
Prompt: 2048 Gen: 14336
Critic: 9e-6
Qwen Owen-2.5-32B
8
512
0.01
s = 10
Constant after
7/p 0.99
TI 0
10-step linear warmup
10
Cosine:
ro +2
ri. = +1
Actor: 5c-7
rM -10
Prompt: 2048
Critic: 9e-6
Qwen-2.5-32B
L = 0 100 Rep. Penalty: 0.05 N = 40
=1
8
512
Gen: 14336
Constant after
0.01
1/p 0.99
10-step linear warmup
Actor: Se-7
A 7
Prompt: 2048
Critic: 9e-6
Qwen-2.5-1.5B
Correct:
512
0.01
Gen: 14336
Cosine with 3% step warmup
Cosine: ro +2 rf. = +1
Actor: 5c-7
_ = -10
= 1
Prompt: 2048
Critic: 9c-6 Cosine with 3% step warmup
Qwen-2.5-1.5B
TL 0
=
8
8
512
Gen: 14336
0.01
-10
1/p 0.99
Rep. Penalty:
0.05
N = 10
22

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
B.5.4. DETAILS SECTION 4.4 (COSINE REWARD HYPERPARAMETERS)
SFT Data: Long CoT data distilled from QwQ 32B PeB--review with the MATH train split.
Table 11. Hyperparameters
Base Model
Rewards
GAF
Episndes
Samples
BS
Epochs
Context Length
LR
KI.
Cosine:
r 0
ri = +10
rg =
=
Prompt: 2048
Actor: Se-7
Llama3.1-88
ri =
Tc = 1
4
4
512
Gen: 14336
Critic: 9e-6
0.01
-10
Yp 0.99
Rep. Penalty:
P 0.05
N 10
Cosine:
TO = +6
TL +5
CD 10
==1
Prompt: 2048
Actor: 5c-7
LJama3. 1-88B
TL
Tr =
512
Gen: 14336
Critic: 9c-6
Te = - 10
Tp 0.99
Rep. Penalty:
P -0.05
40
Cosine:
rá +10
rL +9
To = -10
==1
Prompt: 2048
Actor: 5e-7
Llama3.1-8B
JL = 0
Te =1
4
4
512
1
0.01
Gen: 14336
Critic: 9e-6
Te -10
7pp 0.99
Rep. Penalty:
P -0.05
NN 40
23

---

Demystifying Long Chain-of-Thouue Reasoning
B.5.5. DETAILS SECTION 4.5 (CONTEXT WINDOW SIZE)
SFT Data: Long CoT data distilled from QwQ -2B PeB--review with the MATH train split.
Table 12. Hyperparameters
Base Model
Rewards
GAF
Episndes
Samples
BS
Epochs
Context Length
LR
KI.
Cosine:
r +2
FL +1
" = -10
=
Prompt: 2048
Actor: Se-7
Llama3.1-88
1I = -10
Tc = 1
8
8
512
Gen: 2048
Critic: 9e-6
0.01
Yp 0.99
Rep. Penalty:
P 0.05
N 10
Cosine:
TO = +2
TL +1
CD 10
==1
Prompt: 2048
Actor: 5c-7
LJama3. 1-88B
Tí
Yr
512
Gen: 6144
Critic: 9c-6
Te = - 10
Tp 0.99
Rep. Penalty:
P -0.05
40
Cosine:
r +2
rL +1
To = -10
==1
Prompt: 2048
Actor: 5e-7
Llama3.1-8B
JL
Te =1
8
8
512
1
Gen: 14336
Critic: 9e-6
0.01
Te -10
Ppp 0.99
Rep. Penalty:
P -0.05
NN 40
24

---

Demystifying Long Chain-of-Thougee Reasoning
E.5.6. DETAILS SECTION 4.6 (LENGTH REWARD HACKING)
SFT Data: Long CoT data distilled from QwQ 32B- 22B-Previee with the MATH train split.
Table 13. Hyperparameters
Base Model
Rewards
GAE
Episodes
Samples
BS
Epochs
Context Length
LR
KL
Cosine:
тÜ = +2
r +1
==1
Prompt: 2048
Actor: 5e-7
Llama3.1-8B
8
16
512
2
0.01
rD = -10
-=1
Gen: 14336
Critic: 9c-6
1L 0
Te -10
Cosine:
r +2
rí +1
Co = -10
==1
Prompt: 2048
Actor: 5e-7
Llama3.1-8B
"L
Te =
8
16
512
2
Gen: 14336
Critic: 9e-6
0.01
Fe 100
7p 0.99
Rep. Penally:
P 0.05
N 40
E.5.7. DETAILS SECTION 4.7 (OPTIMAL DISCOUNT FACTORS)
SFT Data: Long CoT data distilled from QwQ-32B-Preee 32B-Previeeeeeeeee with the MATH train split.
25

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
Table 14. Hyperparameters
Base Model
Rewards
GAE
Episodes
Samples
BS
Epochs
Context Length
LR
KL
Cosine:
ró = +2
ri +1
To 10
==1
Prompt: 2048
Actor: 5e-7
Llama3.1-8B
T =
Tc =
512
Gen: 14336
Critic: 9c-6
0.01
Te = -10
Rep. Penalty:
P 0.05
Cosine:
ri = +1
TO = 10
x=1
Prompt: 2048
Actor: Se-7 Critic: 9e-6
Llama3.1-8D
FL 0
Y =
512
1
0.01
Gen: 14336
Te -1
17 0.999
Rep. Penalty:
= 0..55
N 10
Cosine: T6 +2
TL +1
rD
A=
Prompt: 2048
Actor: 5c-7
Llama3.1-8B
TL =
Th = 1
512
Gen: 14336
Critic: 9c-6
0.01
Te = -10
1o 0.99
Rep. Penalty:
P 0.00
40
Cosine:
r +2
Ti = +1
TO 10
==1
Prompt: 2048
Actor: 5e-7
Llama3.1-80
TL
%e 0.999
512
Gen: 14336
Critic: 9e-6
0.01
Te
Tp 0.999
Rep. Penalty:
P 0.05
N 40
Cosine: r +2
ri
To 10
À
Prompt: 2048
Actor: Se-7
Llama3.1-8B
1L = 0
= 0.999
4
512
Gen: 14336
Critic: 9e-6
0.01
- 10
Pp 0.99
Rep. Pemalty:
0.05
40
Cosine:
rf +2
ri = +1
To 10
À ==1
Prompt: 2048 Gen: 14336
Actor: 5c-7
L-am
rT. =
Ne 0.99
Critie: 9c-6
Te = -10
Tpp 0.99
Rep. Penalty:
P -0.05
N 40
26

---

Demystifying Long Shain-of-Though Reasoning
E.5.8. DETAILS SECTION 5.2 RL WITH NOISY VERIFIABLE DATA)
SFT Data: 115k instances of filtered long CoT data distilled from QwQ ZwQ Preview with WebInstruct.
Table 15. Hyperparameters
Prompt Verifier
Episodes Instances
Base Model
Rewards
GAE
Samples
BS
Epochs
Context Length
Cosine:
r0 +2
ri +1
Unfiltered
7D -10
A-1
Actor: 5e-7
Prompt: 2048
1lama3.1-88
(30k sampled)
rf =0
Critie: 9e-6
Symeval
re -10
%p 0.99
30k
Gen: 14336
KI: 0.01
Rep. Penalty:
-0.05 N =
Cosine:
+2
Unfiltered
TO -10
A=1
Actor: 5c-7
Prompt: 2048
Llama3.1-85
(30k sampled)
r.
Critic: 9e-6
30k
Gen: 14336
LLM-as-a-judge
Te
7p 0.99
KL: 0.01
Rep. Penalty:
0.05
N 40
Cosine:
ri
10
==1
Actor: 1e-6
Filtered (115k) Symeval
Prompt: 2048
Llama3.1-8B
r7.
%
115k
Gen: 14336
Critie: 4.5e-6
Fe
7p = 0.99
KL:
Rep. Pemalty:
0.05
N 40
Cosine:
ri
Filtered (115k) IJ-M-as-a-judge
To
Prompt: 2048
Actor:
Llama3.1-8B
r =
%
115k
8
512
Gen: 14336
Critie: 4.5c-6
Te =
7p = 0.99
K1.
Rep. Penalty:
0.05
N 40
E.5.9. DETAILS OF SECTION 6.2 (REINFORCE IS MORE TRICKY TO TUNE THAN PPO)
SFT Data: Long CoT data distilled from SWo--32BPPrewee with the MATH train split.
Table 16. Hyperparameters
Base Model
Rewards
Gamma
Episodes
Samples
BS
Epochs
Context Length
LR
KL
Clip
8 (stopped early)
Prompt: 2048
Llama3.1-8 3-BB
Correct:
512
5c-7
0.01
0.1
Gen: 14336
27

---

Demystifying Long Chain-of-Thouue Reasoning
E.6. Implementation of the Model-Based Verifier
We used Qwen2. 5- 7B Incttrtt as our model-based verifier. was provided with both the reference answer and the suffix of the long CoT. We truncated the long CoT to avoid confusing the verifier. We used the following prompt.
Prompt Template for Model-Based Verifier
Given the fol Fownowigg last 20 lines of the LLM response to math question and the reference solution that question, evaluate if the L.M response is correct based only on the LIMs' s final anawer.
LLM response (last 20 lines):
(out)
Reference solutien: (ref)
Explain your thought process step-by-step before responding with Vudgement:
E.7. Implementation of Short-Form Answer Extraction
We use the 1lama-3 1...mmaa--222.. following prompt template:
model to extract short-form answer from QA pairs in WebInstruct, with the
Prompt Template for Short-Form Answer Extraction
Problem:
(Problem)
Solution:
(Solution)
Based on the Problem and the Solution, extract a short final answer that is easy to check.
Provide the short final answer in the format of "The final answer is $s \boxed(...) $$"
If the answer is mathematica object, write it in LaTex, e.g., "The final anawer is $$ (boxe/\\frac(11211
$$"
the answer is boolean, write it as "True" or "False", e.g., "The final answer is $5
\boxed(Truc)
$$"
If the Problem can't be answered in short form, respond with " like "The final answer is $5
(boxed() $$"
For generation parameters, we use temperature / = 0 (greedy decoding) and set the maximum output length as 512 tokens. After generation, we simply extract the short-form answer from within the \boxed(.....
E.8. Action Prompting Framework
We studied the publicly released CoTs of ol-preview and identified that its thoughts could be categorized into a few types actions (listed below). To construct long CoTs, we designed prompts for each of these actions and implemented multi-step prompting framework to sequence them. The framework ceded control flow of the CoT to the LLM,, with
28

---

Demystifying Long Mhain-of-Thoughe Reasoning
the LLM making branching looping decisions while the framework acted more passively as state machine reacting the LLM outputs. The framework took care of the boilerplate around constructing the CoT with append-only log and managed all of the orchestration.
clarify: Making some observations about the problem in order to identify an approach solve
decompose: Breaking the current problem down into smaller and casier sub-problems to solve.
polution_step. Computing single step in the solution. In the context of math, this could be doing some arithmetic or symbolic manipulation.
re seleectoon Evaluating the current approach and partial solution to see if any mistakes were made, any sub-goals were achieved, or if alternative approaches should be considered instead. Note that we used strong teacher model ol-mini for the reflect ion action as that one was a more difficult prompt to respond to correctly as it requires self-correction.
answer:
Responding with a final answer and terminating the CoT.
E. CONTROL FL.OW
Simplified description of the interaction between the framework and LLM.
Algorithm 2 Action Prompting State Machine
1: Input: prompl
2: Output: ofain_of_thouse sequence
of + [prompt]] (Initialize singleton chain of thought sequence from prompt}
4: state "clarify"
5: while True do
6: if state "clarify" then
7: output + cemppttaction.larifyy
8: (state, thought)) +- parse
9: chain_of thouggtt.ppeend((huuy..
else if state "decompose" then output t ppommppaccion.deccoppee (state, thought) +- porse(output)) chain_of Jhuuinn hpphoug....ppend((()..
12:
13:
14:
else if state = "solution step" then
15:
output + coammppacctoonsolutioon.oo
16:
(state, thought) + parse( (output)
17:
chain (hhoughu.append((houggglg (toought)
else if state "reflection" then
19:
output t- prompptaction cetiipatatttooeeefttiooooo/
20:
(state, thought) +- (output) choin of thought..ppeedd( (thogght)
21:
22:
else if state = "answer" then
23:
output + prompt action reflection()
24: (state, thoughtt) +- pouse())lput)
chain.of thought.append (though(th
26:
return chain.of [thought (Terminate after answer action)
27: end if
28:
end while
29

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
E.8.2. ACTION PROMPTING TEMPLATES
Action: Clarify
You are a very talented mathematics professor,
In a few sentences, VERY CONCISELY rephrase the problem to clarify its meaning and explicitly state what needs to be solved. Highlight any assumptions, constraints and potential risinterpretations.
NOT attempt to solve the problem yet you are just clarifying the problem your mind.
<problem> (goal)
</problem>
Answer in the following format:
<clarifications
Problem clarification as instructed above
s/clarification>
<goal>
Summarize the problem into single statement describing the goal, e.g. Find the value of the variable
</goal>
30

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
Action: Decompose
You are a talented mathematics professcr.
You already have partial solution to problem.
In single sentence, propose candidates for the next subgeal as the next step of the partial solution that will help you make progress towards the current goal. Do not repeat any subgoal, we don't want any infinite loops! Do not suggest using a computer or software tools.
<ourrent goal> courrent_goal] </current geal> <parent goal> (parent_goal)
</parent goal>
<partial solution>
(solution)
</partial solution>
Format your answer as follows:
<thinking>
step-by-step thinking of what the next possible subgoal should be, well some other alternatives that might work remember, we want to solve the parent geal WITHOUT repeating the subgoals that are already DONE.
do not suggest verification checking.
pearen_ggoall </thinking>
<sentence>
single sentence deseribing the subgeal
phrase it if you were thinking to yourself and are considering this hypothesis (don't express much certainty)
</sentence>
<sentence>
single sentence describing an *ALTERNATIVE+ subgoal, without repeating previous ones start off with "Alternatively," </sentence>
<sentence>
single sentence describing an *ALTERNATIVE. subgoal, without repeating previous ones start off with "Alternatively," </sentence>
31

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
Action: Solution Step
You are an extremely PEDANTIC mathematics professor who loves to nitpick.
You already have partial solution to problem. Your task is to solve *only+ the current goal.
You should include symbols and numbers in every sentence possible.
<current goal> (eurrent_goal; </current goal>
<partial solution> (solution)
</partial solution>
VERY CONCISE. Include calculations and equations your response if possible, and make sure to solve then instead of just describing them.
DO NOT SOLVE THE WHOLE QUESTION, JUST THE CURRENT GOAL: (current_goal} Do not repeat any calculations that were already in this prior step: (prior_step)
32

---

Demystifying Long Chain-of- Chain-of-Thought Reasoning
Action: Reflection
You are talented mathematics professcr. You already have partial solution math problem. Verify whether the current subgoal has been achieved.
scurrent goal> courren_gooal. </current goal> (parent_goal) <partial solutien> (solution)
</partial solution>
Format your answer as follows:
everification>
Come up with a quick, simple easy calculation to double check that solution is correct.
This calculation should not re-compute the solution in the sare way, as that would defeat the purpose of double-checking.
Use of the following strategies:
An easier, alternative method to arrive at the answer
Substituting specific values into cquations and checking for consistency
Working backwards from the answer to derive the given inputs and then checking for consistency
consise. Do not suggest using computer.
At the end of your verification, restate the answer from the current solution. Do not calculate it if it hasn't been solved. Phrase it if you are reflecting as you solve the problem. c/verification>
ccurrent achievedd
trus or false, depending on whether the sclution is correct and the current goal has been achieved; (current_goal: </ourrentt_ppll_chheeee
"parent__gal__chieeeee
true or false, depending on whether the parent goal has been achieved: [parent_gpal.targer
s/paront_goal_achleveed <new_goal>
If the solution is not correct or the current goal has not been achieved, suggest an alternative ve current goal here a single sentence.
Start off with "Alternatively," Your goal should be sufficiently di fferent from subgoals that have been sol ved that have timed out: [parent goal_tree! </new goal>>
Action: Answer
Extract the final answer, making sure to obey the formatting ructiooss
Solution:
(solution)
Formatting instructions: (format)
33


Reviews:
[{"review_id": "w9OUW7DzUX", "reviewer": "Reviewer_bk9g", "summary": "This paper investigates the mechanics of long chain-of-thought (CoT) reasoning in large language models (LLMs), focusing on how supervised fine-tuning (SFT) and reinforcement learning (RL) can enhance reasoning capabilities. Key findings include: 1) SFT with long CoTs significantly improves model performance and facilitates further RL improvements; 2) RL-driven CoT scaling can be unstable, but reward shaping, such as a cosine length-scaling reward with a repetition penalty, stabilizes and improves reasoning; 3) leveraging noisy, web-extracted data with filtering mechanisms shows promise. The study provides practical insights for optimizing training strategies to enhance long CoT reasoning in LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their positive and thoughtful feedback! Below, we address the key points raised:\n\n> Q1: “Can the proposed reward shaping techniques be adapted for other types of reasoning tasks beyond mathematics and STEM?”\n\nYes, the reward shaping techniques we propose, particularly the cosine length-scaling reward and repetition penalty, are designed to be general-purpose. While our experiments focus on mathematics and STEM due to the availability of verifiable signals (e.g., MATH, AIME), the core principles—encouraging correct long reasoning without over-optimization—are task-agnostic.\n\n> Q2: Limited exploration of scalability beyond 8B parameters\n\nWe fully agree that evaluating across a wider range of model sizes is important. **Experiments with both smaller (1.5B) and larger (32B) models are currently running.** While we couldn’t complete these runs in time for the rebuttal, **we will do our best to report results during the discussion period** or include them in the final revision.\n\n**We kindly note that this is an academic project, and running large-scale models like 32B presents significant infrastructure and compute challenges.** While such experiments may be routine for large industry labs, they require substantial effort in academia. **We deeply appreciate the reviewer’s perspective and are working hard to extend our coverage despite these constraints.**\n\n> Q3: Potential for reward hacking in RL\n\nWe appreciate this concern and directly addressed it in the paper. As noted in Section 4.5, we observed length-based reward hacking (e.g., via repetition) with increased compute. To mitigate this, we introduced an n-gram repetition penalty and tuned reward discounting to improve learning stability.\n**Figure 5 shows that applying this penalty improved downstream accuracy** while reducing degenerate behaviors. We also found that token-level penalties and distinct discount factors per reward signal (Section 4.6) helped control optimization dynamics and reduced unwanted artifacts.", "claims_and_evidence": "The paper claims that supervised fine-tuning (SFT) with long chain-of-thought (CoT) data significantly improves model performance and facilitates further reinforcement learning (RL) enhancements. Evidence includes experiments showing that models fine-tuned with long CoTs achieve higher accuracy and are more amenable to RL improvements compared to short CoTs.", "methods_and_evaluation_criteria": "The study employs SFT and RL to train models on long CoT reasoning tasks, using datasets like MATH and WebInstruct. Evaluation is conducted on benchmarks such as MATH-500, AIME 2024, TheoremQA, and MMLU-Pro-1k.", "theoretical_claims": "No Theoretical Claims.", "experimental_designs_or_analyses": "Experiments involve fine-tuning models on long CoT data distilled from QwQ-32B-Preview and comparing them with short CoT models. The authors test different reward functions, including a cosine length-scaling reward, and evaluate their impact on CoT length and model accuracy.", "relation_to_broader_scientific_literature": "The study builds on previous work in CoT prompting and RL for LLMs, extending these techniques to long CoT reasoning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses include the limited exploration of model scalability beyond 8B parameters and the potential for reward hacking in RL training.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Can the proposed reward shaping techniques be adapted for other types of reasoning tasks beyond mathematics and STEM?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jptooVVJtV", "reviewer": "Reviewer_f2xR", "summary": "This paper conducts comprehensive experiments for long chain of thoughts. They adopt a variety of methods related to long CoTs, including SFT, RL, source of the SFT data, impact of reward design, noisy reward.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the **very positive assessment** and thoughtful comments. We're glad the key takeaways around long CoT data, reward shaping, and SFT+RL initialization were found to be useful and convincing.\n\n> Q1: On Takeaway 5.1 and the impact of WebInstruct (WebIT) data\n\nWe appreciate the reviewer’s point and would like to clarify that we view this as an important and encouraging finding, especially in the context of studying generalizability in reasoning. **While existing reasoning models are often trained exclusively on math datasets, our results show that incorporating more diverse, real-world data such as WebInstruct can significantly enhance out-of-domain general reasoning performance**, notably improving MMLU-Pro-1k by nearly 10 absolute points.\n\nAlthough we observe a slight drop in MATH performance, **the overall trend shows improved average accuracy across tasks**. This suggests that query diversity in training data plays a key role in building more broadly capable reasoning models, rather than weakening them.\n\nWe agree that this topic deserves deeper investigation, and we plan to explore how to balance domain-specific precision with general reasoning strength in future work.\n\n> Q2: On presentation: \n\nWe appreciate the suggestion to move some expected results to the appendix and will revise the paper to better emphasize the most important insights.", "claims_and_evidence": "The paper provides many orthogonal directions for long CoTs. The style is more like a review paper, but the messages are valuable for the field. I find some claims to be particularly useful and convincing:\n1. Takeaway 3.1&3.3: long, high quality CoTs data is much better than short CoTs data.\n2. Takeaway 3.2: SFT initialization is beneficial for RL.\n3. The takeaways 4.1&4.4: Directly using constrained context window size cannot easily stabilize RL training. \n3. Takeaway 4.2&4.5: reward shaping can be used to stabilize and control CoT length. The reward hacking may happen but can be mitigated using a repetition penalty.\n\nHowever, I think certain results seem to be not surprising and can be relegated to appendix to leave room for more discussions on important messages.\n1. The takeaways 5.1 and table 2: I think the inclusion of WebIT only improves the performance on MMLU, which is not surprising because it bears more resemblance to the WebIT dataset. Moreover, including WebIT also decreases the reasoning ability for models. The message seems to be not new given the well-known forgetting problem in finetuning.", "methods_and_evaluation_criteria": "The methods and evaluations mostly make sense.", "theoretical_claims": "NO.", "experimental_designs_or_analyses": "NO.", "relation_to_broader_scientific_literature": "NO.", "essential_references_not_discussed": "NO.", "other_strengths_and_weaknesses": "NO.", "other_comments_or_suggestions": "NO.", "questions_for_authors": "NO.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOZs7tfLd7", "reviewer": "Reviewer_U7rB", "summary": "This paper systematically investigate the underlying mechanics of long cot reasoning. It conducts experiments on qwen-7b and llama3-8b, and evaluates on diverse math benchmarks. Based on the experiment results, the paper draws several conclusions.\n\n\n##update after rebuttal:\nThank you for the authors’ efforts in addressing the scalability concern raised in my review.\n\nDuring the rebuttal phase, the authors mentioned that they were conducting additional experiments, but did not provide concrete results at the time to support whether their findings generalize beyond the 7B/8B model scale. While I understand the limitations of academic projects, I wasn’t fully convinced by the explanation that they couldn’t run smaller models—especially since testing on models like 1.5B or 3B would have been more accessible. I’m glad to see that these experiments have now been/will be included.\n\nI appreciate the authors' efforts and responsiveness. That said, I still feel that many of the findings are not particularly surprising, and some of the experimental analysis could be more thorough. Therefore, I will slightly improve my score to a weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and for recognizing the timeliness and scope of our work.\n\n> Add experiments on different sized LLMs (i.e., 1.5b-32b-70b)\n\nWe fully agree that evaluating across a wider range of model sizes is important. **Experiments with both smaller (1.5B) and larger (32B) models are currently underway.** While we couldn’t complete these runs in time for the rebuttal, **we will do our best to report results during the discussion period** or include them in the final revision.\n\nIn the meantime, we want to emphasize that:\n- **We already include diverse model families**: our experiments use both Qwen and LLaMA, two distinct model families with different pretraining setups. This increases confidence that our findings are not limited to a single architecture.\n- **We cover a wide range of experimental settings**: As the reviewer noted, we explore many axes—including SFT vs. RL, different reward functions, various data sources (MATH, WebInstruct), and multiple evaluation benchmarks. This breadth helps us surface consistent and robust patterns.\n- **Some findings are not scale-dependent**: Several of our observations, such as the emergence of branching, backtracking, and self-validation more from pre-training and are not related to scales.\n\n**We kindly note that this is an academic project, and running large-scale models like 32B presents significant infrastructure and compute challenges.** While such experiments may be routine for large industry labs, they require substantial effort in academia. **We deeply appreciate the reviewer’s perspective and are working hard to extend our coverage despite these constraints.**", "claims_and_evidence": "the claims in this submissions are supported by the experiments.", "methods_and_evaluation_criteria": "This work primarily aims to understand the underlying mechanisms of long-CoT reasoning through various experimental designs. Most of the experiments seem reasonable to me, but the evaluation is currently limited to a math benchmark and two models (7B and 8B). It remains unclear whether the derived findings are generalizable.", "theoretical_claims": "This submission does not include any theoretical claims", "experimental_designs_or_analyses": "I appreciate the timeliness of this paper. However, it does not include experiments on larger (>8B) or smaller (i.e., 3b, 1.5b) LLMs or non-math benchmarks. Therefore, I find the experiments not sufficiently sound.", "relation_to_broader_scientific_literature": "This paper is timely and contributes to enhancing the community's understanding of long-CoT reasoning.", "essential_references_not_discussed": "Most related works have been referenced", "other_strengths_and_weaknesses": "The paper is timely and designs a large number of experiments to explore the impact of diverse factors of long-CoT on reasoning. Many experimental observations could help improve reasoning. However, my main concern is that the paper only conducts experiments on 7B and 8B LLMs on math benchmarks. Therefore, these experimental observations are likely to hold only for the 7B and 8B models.", "other_comments_or_suggestions": "Add experiments on different sized LLMs (i.e., 1.5b-32b-70b), add evaluations on more benchmarks (i.e., code)", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Summary
This paper studies long chain-of-thought (CoT) reasoning in large language models (LLMs), with a focus on understanding how supervised fine-tuning (SFT) and reinforcement learning (RL) influence reasoning length and effectiveness. The authors report three main findings: 1) SFT greatly improves training ease and efficiency, though it is not strictly required; 2) increasing computational resources during RL can encourage better reasoning, but targeted reward design is necessary to maintain stable growth; and 3) incorporating noisy solutions from web sources, with appropriate filtering, effectively extends RL performance, especially for reasoning tasks beyond the training distribution, such as STEM problems. These results offer clear guidelines for improving the training of models capable of extended CoT reasoning.

## Decision
The paper is well-written, and the proposed idea is interesting. It is a timely and important topic. The reviewers have raised several important concerns and questions. I think overall, the authors have done a good job answering and addressing them. As a result of reviewers' feedback, the authors have quite significantly improved the paper and their results. As requested by the reviewers, the authors conducted additional training and evaluations on large (Qwen2.5-32B) and small (Qwen2.5-1.5B) models to validate the scalability of their proposed cosine reward shaping technique. The authors couldn't share their results in their rebuttal but privately shared them with the AC. They show significant improvements with the cosine+representation pen, the reward design they proposed. I recommend that the authors include those results in the final version of the paper, which will make the paper a good contribution to the ICML community. Thus, I would recommend this paper for acceptance.
Decision: Accept (poster)