SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
Authors: Weihao Zeng
Yuzhen Huang
Qian Liu
Wei Liu
Keqing He
Zejun MA
Junxian He
Venue: COLM 2025 (2025)
Keywords: Reasoning
Large Language Model
TL;DR: The paper explores zero training with rule-based rewards for emergent chain-of-thought reasoning in smaller models, producing significant improvements in both reasoning accuracy and CoT length across all settings.

Abstract: DeepSeek-R1 has shown that long chain-of-thought (CoT) reasoning can naturally emerge through a simple reinforcement learning (RL) framework with rule-based rewards, where the training may directly start from the base models—a paradigm referred to as zero RL training. Most recent efforts to reproduce zero RL training have primarily focused on the Qwen2.5 model series, which may not be representative as we find the base models already exhibit strong instruction-following and self-reflection abilities.
In this work, we investigate zero RL training across 10 diverse base models, spanning different families and sizes including LLama3-8B, Mistral-7B/24B, DeepSeek-Math-7B, Qwen2.5-math-7B, and all Qwen2.5 models from 0.5B to 32B. 
Leveraging several key design strategies—such as adjusting format reward and controlling query difficulty—we achieve substantial improvements in both reasoning accuracy and response length across most settings.
However, by carefully monitoring the training dynamics, we observe that different base models exhibit distinct patterns during training. For instance, the increased response length does not always correlate with the emergence of certain cognitive behaviors such as verification (i.e., the "aha moment"). Notably, we observe the ``aha moment'' for the first time in small models not from the Qwen family.
We share the key designs that enable successful zero RL training, along with our findings and practices. 
To facilitate further research, we open-source the  code, models, and analysis tools.

Published as a conference paper at COLM 2025
SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
Weihao Zeng"!
Yuzhen Huang
Qian Liu
Wei Liu
Keqing He
Zejun Ma²
Junxian He
IHKUST TiikTok
Meituun
Abstract
DeepSeek-R1 has shown that long chain-of-thought (CoT) reasoning naturally emerge through simple reinforcement learning (RL) framework with rule-based rewards, where the training may directly start from the base models-a paradigm referred to as zero RL training. Most recent ef- forts reproduce zero RL training have primarily focused the Qwen2.5 model series, which may not be representative as we find the base models already exhibit strong instruction-following and self-reflection abilities. In this work, we investigate zero RL training across 10 diverse base models, spanning different families and sizes including LLama3-8B Mistral-7B/221 /4B, DeepSeek-Math. Qwen2.5-math-77 and all Qwen2.5 models from 0.5B 32B. Leveraging several key design strategies as adjusting format reward and controlling query difficulty-wi achieve substantial improve- ments in both reasoning accuracy and response length across most settings. However, by carefully monitoring the training dynamics, observe that different base models exhibit distinct patterns during training, For instance, the increased response length does not always correlate with the emergence certain cognitive behaviors such as verification (i.e., the "aha moment"). Notably, we observe the "aha moment" for the first time in small models not from the Qwen family. We share the key designs that enable successful zero RL training, along with our findings and practices. To facilitate further research, we open-source the code, models, and analysis tools.
Mistral-7B-vo.1
Llama-3. 3-88
DeepSeek-Math-P
Mistral-Smalli mall--24
30
1.0
43
13
0.8
32-
0.8
Accuracy (%)
0.4
Response Length
16-
Qwen-2.5-0.50
Qwen-2.5-1.55
Qwen-2.5-76
Qwen-2.5-328
15-
68
0.
12-
05
-3.5 30
0.5
50
50
1.00
120
150
Accuracy
Response Length
Figure 1: Accuracy and response length across training iterations for different models, averaged on GSMSK, MATH500, Minerva Math, OlympiadBench, AIME24, and AMC23, Per-benchmark results are in Figure 11 (Appendix D). All training starts from base models.
"Equal Contribution. Correspondence to Weihao Zeng (wzengak@comeec.usthh))) Yuzhen Huang (yhuanghj@cse.ust.hk)) and Junxian He (junxianh@cse.usthh..

---

Published as a conference paper at COLM 2025
1 Introduction
Large reasoning models, including OpenAl-ol (Jaech et al., 2024), DeepSeek-RI (DeepSeek- AI et al., 2025a), and Kimi-k1.5 (Team et al., 2025), demonstrate remarkable abilities. These models excel at generating long Chains-of-Thought (CoT) (Wei et al., 2022) responses when solving complex tasks and exhibit advanced, reflection-like reasoning behaviors. Recently, DeepSeek-R (DeepSeek-Al) et al., 2025a) has revealed that starting from pretrained models (i.e., base models), pure reinforcement learning (RL) with rule-based reward can lead to the spontaneous emergence of long CoT and self-reflection behaviors, called the "aha moment". This RL training paradigm starting from base models is often referred to zero RL training.
While the success of zero RL training was initially demonstrated using DeepSeek- V3 (DeepSeek-AI et al., 2025b), model with 671B parameters, it remained unclear whether such emergent phenomena persist in generally smaller and less capable open base models. Recent open-source efforts exploring zero-training approaches have predominantly centered on the Qwen2.5-series models (Zeng et al., 2025a; Yeo et al., 2025; Xie et al., 2025; Hu al., 2025; Yu et al., 2025), which, even base models, exhibit strong bilities and display notable cognitive behaviors such as backtracking and verification from the beginning,
cnerruction-oolowiiii
capa-
we will detail in $2.4. Moreover, the analyses of model behavior in these studies remain largely superficial, focusing primarily on metrics such as response length and accuracy. These observations neither clearly establish whether the models' reasoning behaviors actually change nor clarify the mechanisms underlying the emergence of effective reasoning, leaving significant gap in understanding.
To provide a more transparent understanding of zero RL training across different base models in the wild, this paper addresses the following key questions: (1) How do reasoning capabilities develop across various models during zero RL training? (2) Does an "aha moment" still occur for base models that initially lack strong mnsruutioo-following and self-verification abilities? (3) What are the critical factors for ensuring successful zero RL training across diverse base models?
To this end, we perform zero RL training across diverse range of model series and sizes, including Mistral-7B (Jiang et al., 2023), Mistral-24B (Mistral AT, 2025), Llama3-8B (Dubey et al., 2024), DeepSeek-Math-78 (Shao et al., 2024), Qwen2.5-0.5B/ 1.5B/7B /B///////// /3BB (Yang et al., 2024a), as well as Qwen2.5-Math-7B (Yang et al., 2024b). To maintain simplicity in the training recipe, our experiments rely exclusively et al., 2021) and MATH (Hendrycks et al., 2021) datasets for rule-based reward modeling. It is worth noting that we adopt the same training hyperparameters to train all the models. Using GRPO (Shao et al., 2024) as the RL algorithm, combined with several critical factors that we identified, we obtain significant improvements in model accuracy across all base models, along with a notable increase in response length for out of the 10 models, with the exception of Qwen2.5-Math-7B. However, through careful monitoring of training dynamics and reasoning behaviors, we find that different base models exhibit distinct patterns during training. Also, certain specific factors require careful attention to ensure successful zero RL training. Below, we summarize our key findings.
the training sets of GSM8K (Cobbe
1. Increased response length does not always correspond to an "aha moment" Interestingly, for most Qwen2.5 models, which form the foundation of most recent open-source efforts, we do not observe a rise in the frequency of certain cognitive behaviors, such as self-reflection, despite the increase response length. ($2.4)
2. For the first time, we observe significant increase in the frequency of specific cognitive reasoning behaviors, such as verification, in small models outside the Qwen family, notably in the Llama3-8B and DeepSeek-Math-78 models. ($2.4)
3. Enforcing rigid format reward (e.g., enclosing answers within boxes) (DeepSeek- Al et al., 2025a) significantly penalizes exploration (Singh et al., 2023; Wang et al., 2024), particularly for base models that initially struggle with instruction following. This restriction lowers their performance ceiling and often induces overthinking behaviors (Chen et al., 2024). ($3.1)
2

---

Published as a conference paper at COLM 2025
4. The difficulty level of the training data must align closely with the base model's intrinsic exploration capabilities, otherwise zero RL will fail. ($3.2)
5. In contrast to the observation in Shao et al. (2024), zero RL training lifts pass@k accuracy by 10-30 absolute points, strong evidence confirming zero RL training is not just reranking responses. ($2.3)
6. We revisit the traditional training pipeline that performs SFT to learn to follow instructions before RL training. Specifically, we use conventional SFT datasets as cold start for RL-a de facto approach prior to the release of DeepSeek-RR While high-quality CoT data (Li et al., 2024) can rapidly enhance a base model's performance through imitation, we find that it significantly limits the model's ability explore freely during RL. This constraint diminishes post-RL performance and suppresses the emergence of ad advanced reasoning capabilities. (64)
2 On Emerging Reasoning in Zero RL Training
Existing research on zero RL training primarily focuses on Qwen2.5-series models, tracking only superficial metrics like accuracy and response length (Zeng et al., 2025a; Hu et al., 2025; Yu et al., 2025). However, Qwen2.5 models, due to their extensive use of synthetic data during pretraining, already exhibit instruction-folloiing abilities and reflective behaviors, which may not represent base models in diverse scenarios. Additionally, an increase in response length does not necessarily indicate the emergence of cognitive behaviors and can sometimes result from meaningless repetition. To address these issues, this section explores zero RL training across various base models of different sizes and families. By monitoring range of metrics beyond accuracy and response length, we aim to provide more comprehensive and transparent understanding of zero RL training for open base models in the wild.
2.1 Experimental Setup
Training Algorithm: In our study, we follow the zero RL training recipe in DeepSeek- AI et al. (2025a) using various open base models, employing the GRPO algorithm (Shao al., 2024). Here, zero RL training refers to RL directly from the base model without any prior supervised fine-tuning (SFT). detailed introduction to GRPO is provided in Appendix A.
Training Dataset: We use the GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) training datasets. In our experiments, we find that data difficulty is critical for successful zero RL ($3.2) and it necessary to use data that aligns with the model's capability. To investigate this phenomenon, we categorize the data into three difficulty levels: Easy (GSM8K and MATH Iv.1), Medium (MATH Iv.1-4), and Hard (MATH Iv.3-5), with each category containing roughly 8,000 problems.
We use a rule-based reward function that assigns +1 for correct answers and 0 for incorrect ones. Unlike prior works (Luo et al., 2025; Chen et al., 2025), we avoid format-based reward, which may hinder exploration, particularly for base models struggling with format adherence, as detailed in $3.1.
Reward:
We conduct zero RL training experiments on Alamm--311-8B DeepSeek-Math-77, Mistral-v0.1-7B, Mistral-Small-24bb -4---22-220,0 and Qwen-2.5 (0.5B, 1.5B, 7B, 14B, 32B). As we perform experiments for variety of models, under extremely simple settings with small, simple datasets and only correctness reward, we refer to our obtained models as SimpleRL-Zoo to represent simple training recipe for zoO of open base models. In our preliminary experiments, we observe that using complex prompts with models that have weak instruction-followwe we apply simpler prompts to some models (Llama-3.1 Mistral-v0.1-77 and Qwen-2.5- 0.5B/1.5B). Examples of these prompts shown in Figure 10 in the Appendix.
Models:
capabilities often results in instability during training. Therefore,
3

---

Published as a conference paper at COLM 2025
MATH
Minerva Math
Olympiad Bench
AIME24 (Pass81)
AIME24 (Avg832)
Model
GSM8K
AMC23
Avg.
500
Llama,
DeepSeck and Mistral Models
Mistral-v0.177B
21.2
4.2
4.0
2.4
0.0
SimpleRI-770
75.0
15.8
6.6
4.1
0.0
18.6
Llama-3,1-88
39.7
13.6
4.8
3.1
0.0
0.2
2.5
10.6
+ SimpleRL-Zoo
79.2
23.0
9.6
5,3
0.0
0.2
15.0
22,0
DeepSeek-Math-77
28.4
19.4
5.5
4.7
0.0
0.0
10.0
11.3
4 SimpleRL-Zoo
78.5
39.6
21.0
12.6
3.3
29.2
Mistral
Mistral-Small-24BB
78.6
43.6
10.7
11.6
3.3
SimpleRL-Zoo
92.0
70.6
36.8
36.6
16.7
13.1
45.0
49.6
Qwen Series Models
Qwen-2.5-0.5B
36.7
15.8
4.8
2.8
0.0
0.3
12.5
12.1
+ SimpleRL-Zoo
49.5
34.4
10.3
8.9
0.0
0.7
22.5
20.9
Qwen-2.5-1.58
55.7
29.6
6.6
6.5
0.0
0.1
12.5
18.5
+ SimpleRL-Zoo
74.4
59.0
20.2
21.0
6.7
35.0
Qwen-2.5-7B
88.2
64.6
25.7
30.1
3.3
30.0
40.3
+ SimpleRL-Zooo
91.7
78.2
38.6
40.4
20.0
15.6
62.5
55.2
Qwen-2.5-Math-7B
65.5
63.6
12.5
25.8
13.3
8.6
42.5
37.2
4 SimpleRI-Zoo
90.2
80.2
37.5
39.0
40.0
24.0
70.0
59.5
Qwen-2.5-14B
91.6
65.4
24.3
33.5
6.7
37.5
43.2
SiopleRI--Zoo
94.4
80.2
40.4
44.9
23.3
14.2
57.6
56.8
Qwen-2.5-328
92.9
68.6
27.9
31.1
10.0
4.5
45.0
45.9
<> SimpleRL-Zox
95.9
82.4
42.6
46.4
36.7
27.2
67.5
61.9
Table 1: Detailed performance of various models across multiple benchmarks. The blue lines represent the models trained with our recipe. AIME is evaluated in two ways: Pass@1 (single run) and Avg@32 (average baselines use greedy decoding, and models with SimpleRL-Zoo use temperature=1.0 and top-p=0.95. For AÍME24 (Avg@32), sample 32 responses per model with the same settings. Average scores are based on AIME (eass@11) and other benchmarks.
from 32 runs). For AIME24 (Pass@1) and other benchmarks,
Benchmark: We evaluate performance standard mathematical reasoning benchmarks, including GSM8K (Cobbe et al., 2021), MATH 500 (Hendrycks et al., 2021), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024), well competition-level benchmarks such as AIME 2024 and AMC 2023.
For more experimental setup details, please refer to Appendix B.
2.2 Evaluation Metrics
During training, we monitor standard metrics such as accuracy and response length across benchmarks. In our preliminary experiment, we observe that response length as metric is quite superficial and cannot accurately reflect changes in the model's reasoning behavior. Therefore, we adopt the following metrics additionally:
To better understand the model's reasoning patterns throughout the training process, we adopt the cognitive behavior framework proposed by Gandhi et al. (2025) and use GPT-4o (Hurst et al., 2024) identify reasoning-relatee related behaviors, including "Backtracking", "Verification", "Subgoal Setting", and "Enumeration". We compare the consistency between GPT-4o and human annotations of deasoning-relatee behaviors in the Appendix E. We report the ratio of responses that contain such cognitive behaviors. While some recent studies suggest tracking reflection behavior using related keywords (Yeo al., 2025; Xie et al., 2025) as monitoring signals, we argue that these keywords only exhibit only weak correlation with high-level reasoning patterns like reflection and verification. As result, they fail to adequately capture the development of these reasoning processes. We place the setting details, comparisons of different tracking methods, and reasoning behavior cases of different models in Appendix I.
Reasoning Behavior Ratio:
In the early stages of training, the base model exhibits weak instruction- following ability and often fails to stop appropriately, resulting in irrelevant excessively long outputs. After training collapses, the model may also generate repetitive or overly extended responses. Since the model has fixed maximum context length, such outputs
Clip Ratio:

---

Published as a conference paper at COLM 2025
Average
Math500
13.4
Accuracy (%6
566
615
61.2
75
807
214
18.
16.0
170
20
46 1.00
66 80
390
28
Passig P355g8
Figure Pass@1 and Pass@8 accuracy over the training iterations of Mistral-Small--2BB The model is trained on the hard data (MATH levels 3-5) described in $2.1. We evaluate its performance on three benchmarks: AIME24, AMC23, and Math500. The reported average score is the mean across these three benchmarks.
may be truncated during both training and evaluation. To monitor this issue, we define the proportion of truncated outputs as the "Clip Ratio".
Generations that are truncated often result from issues such as repetitive patterns or incomplete reasoning, which typically do not contribute to effective trajectories. To account for this factor, we introduce new metric to track the average length of responses that are stopped under normal conditions.
Average Stopped Length:
For more evaluation metrics details, please refer to Appendix C.
2.3 Main Results
Zero RL Training Improves both Accuracy and Response Length Significantly: Figure and Figure 11 in Appendix D illustrate a steady improvement in both response length and average accuracy across various benchmarks. Table provides a detailed breakdown of the results. Despite using only 8K training samples, we observe significant performance gains for all models. The improvements cover competition-level tests like AIME 2024 and AMC 2023 for most cases. This demonstrates the remarkable generalization capabilities of zero RL training, enabling the model to effectively progress from easier to more challenging problems. In addition to the Qwen series models, we also significantly improve both performance and response length for other models that initially starts with low baselines. For instance, after just 80 training iterations, the DeepSeek-Matee -athh----s performance increases more than threefold, while its response length grows from around 300 to over 1200 tokens.
Zero RL Training also Demonstrates Strong Generalization Perforrance. We also evalu- ate the generalization ability of zero RL training using three benchmarks: IFEVAL (Zhou al., 2023), MMLU (Hendrycks et al., 2020), and GPQA-Diamond (Rein al., 2024). IFEVAL capability, MMLU assesses the model's mastery of general
measures instruction-followii knowledge, and GPQA-Diamon is challenging benchmark that tests domain-specific ex- pertise in chemistry, physics, and biology. Table presents the changes in model performance on IFfival, MMLU, and GPQA-Diamonte being conducted on only 8K math caasoning-relatee examples, the model generalizes effec- tively across a range of tasks. Notably, it shows significant gains in instruction-followii and general knowledge on IFEval and MMLU, as well as substantial improvements on the challenging GPOA-Diamond benchmark, which spans chemistry, physics, and biology.
before and after training. Despite zero RI. training
Steady Improvement of Pass@k Accuracy: As shown in Figure 2, Mistral-Small-A exhibits robust growth in pass@8. Furthermore, as training progresses, the model's pass@1 results eventually surpass the initial pass@8 results of the base model. By iteration 100, the two metrics differ by more than 30 absolute points on average. This suggests significant potential for further improvements in RL, our training rolls out 8 responses for each query and pass@8 represents the model's 's ability to explore correct responses. Surprisingly, the gap between pass@1 and pass@8 does not diminish during training; instead, it widens as training progresses. Figure shows that a significant gap in pass@k persists between the base model and the model after RL training, even at higher values of k the gap is from 13
5

---

Published as a conference paper at COLM 2025
IFEVAL strict-prompt
MMLU
Model
MMLU
GPQA
Avg.
Stem
Lama, Seep and Mistral Models
Mistral-v0.1-78
13.5
26.1
28.0
23.2
22.7
4 + SimpleRL-Zoo
34.6
30.3
28.7
Llama-3.1-8)
16.1
27.1
28.7
22.7
23.6
25.1
40.7
44.5
20.2
32.6
DeepSeek-Math-78
11.5
21.6
22.7
19.2
18.7
SimpleRL-Zoo
16.3
47.4
45.5
27.3
34.1
Mistral-Small-24B
17.4
30.9
31.7
20.2
25.0
SimpleRI-Zoo
23.5
73.9
78.8
45.0
55.3
Quen Series Models
@wen-2.5-0.5B
9.6
23.2
24.9
24.8
20.6
+ + SiimlleRL-Zoo
14.4
32.1
34.6
26.3
26.8
Qwen-2.5-1.5B
33.1
35.4
24.8
27.1
+ + 0000
20.3
42.1
45.2
28.8
34.1
Qwen-2.5-7B
21.3
39.8
38.6
23.7
30.8
+ + SimpleRL-Zoo
49.6
47.0
29.8
38.1
Qwen-2.5MMath-7B
40.6
38.0
27.8
30.1
+ + SimpleRL-Zoo
17.0
55.6
56.6
35.4
41.1
wwen22.5-14B
22.9
63.5
24.8
42.7
+ + SimpleRL-Zoo
79.1
50.0
58.7
Qwen-2.5-32B
62.7
38.9
46.7
+ SimpleRL-Zoo
82.5
49.5
60.6
Table 2: Detailed performance of various models across IFEVAL, MMLU and GPQA. The blue lines represent the models trained with our recipe.
to 30 absolute points when we vary k up to 128. This suggests that zero training not just reranking the model's output distribution within the top k candidates at reasonably large range (Shao et al., 2024), instead, enhances the model's fundamental reasoning abilities.
Growth in Response Length May be Un- healthy: Response length does not always re-
flect genuine growth in reasoning. In some cases, unstable training generate excessive repetitive content until they hit the context length limit, artificially inflat- ing response length without improving reason- ing depth. For example, Figure shows that while most models maintain a low clip ratio below 5% of the data when their average stop- ping length steadily increases, Mistral-7B-v0.1 exhibits high clip ratio and significant fluctu- ations in stopping length. Upon closer inspec- tion of its responses, we find that the responses consist of incoherent, mixed miixed-language gibberish, suggesting that its thinking process is not genuinely expanding. We note that such patterns would not be captured by response length as in Figure 1. These findings indicate that most models demonstrate a meaningful increase in response length. This raises an important question: What exactly do models learn their thinking time increases? We answer this question next.
70
cause models to
PassgK Accuracy (56)
50.9
22.3
K
Figure 3: Pass@k of Mistral-24B based on the average results from AIME24 and AMC23.
2.4 The " "Aha Moment" -Quantifying Emergence of Reasoning Behaviors
Figure 5 illustrates the reasoning behavior ratio on OlympiadBench during model training. By comparing Figure 5 with Figure 4, we observe that fluctuations in the reasoning behavior ratio effectively account for variations in the average stopped length. Interestingly, we find that different models exhibit entirely distinct trends in reasoning behavior changes.

---

Published as a conference paper at COLM 2025
Mistral-7B-yy
Llama-3.
DeepSeek-Math-78
Mistral-Small-24B
100
100
100
3.5
6.0
12
445
Average Stopped Length (K)
17.0
0.
10.0
Clip (%6
0.3
Qwen-2.5-0.55
Qwen-2.5-1.58
Qwen-2.5-78
Qwen-2.5-328
100
1.00
300
1.0
0.9
10-
60
62-
60-
0.8
40
40
-0.6
0.5
22
50
75
100
50
100
30
100
33
300
150
Clip Ratio
Aucrage Stopped Length
Figure 4: Average clip ratio and stopped length across training iterations for different models. We assess the models every five steps on variety of math benchmarks, including GSM8K, MATH500, Minerva Math, and OlympiadBench, as well as competition-level benchmarks like AIME24 and AMC23. The red line indicates the clip ratio, while the blue line represents the stopped length. Per-benchmark results are in Figure 12 (Appendix D).
Smaller models, such Qwen-2.5-0.5B and 1.5B, tend to prioritize learning the "Subgoal Setting" behavior, with its proportion increasing by approximately 4-5 times. Additionally, the proportions of "Verification" and "Enumeration" also show noticeable growth. In contrast, for other base models that inherently possess step-by-step reasoning capabilities, adjustments in "Subgoal Setting" during the RL training process are relatively minor.
DeepSeek-Math-7B, Llama-3.1-8B, and Mistral-Small--44B exhibit substantial increases in the proportions of "Enumeration" and "Verification" behaviors, rising from relatively low initial levels by approximately 3-4 times. This growth correlates closely with their changes in average stopped length, suggesting a shift in reasoning patterns over time. For instance, in SmaB--24B, reflection-orientes behaviors such as "Verification" and "Backtracking" increase dramatically from nearly 0% to approximately 50%, indicating the emergence of reflection behavior from scratch. This shift suggests that the model progressively internalizes verification as part of its reasoning process, offering a promising trajectory for enhancement.
In contrast, Qwen-2.5-7B and 32B demonstrate strong reasoning behaviors from the outset, with minimal changes throughout training. This phenomenon aligns with their slow length adjustments (Figure 1) and suggests that Qwen models inherently possess robust reasoning capabilities. Rather than undergoing a structural shift in their reasoning processes, they primarily benefit from small increases in thinking time, which yield significant performance improvements. Finally, we observe that Mistral-7B-v0.1 consistently exhibits low reasoning behaviors with no noticeable growth, further supporting our earlier analysis in $2.3.
To intuitively illustrate the changes in reasoning behavior, we present examples of Mistral 24B's reasoning before and after training Figures 22. Comprehensive case studies in- volving other models are available in Appendix I.3. In Figure 22, we observe that unlike the base model, the zero training model actively attempts verify if its initial solution is valid by substituting it back into the original expression. Upon recognizing that the first solution does not meet the necessary conditions, the model explicitly initiates a backtracking approach, stating "let's try another possibility," eventually arriving at the correct answer.
Key Factors Shaping Zero RL Training
In this section, identify key factors that influence stability and performance during zero RL training, particularly when dealing with early-stage or weaker models. First, we explore how over-reliance format rewards restricts exploration. Next, we analyze how
7

---

Published as a conference paper at COLM 2025
Mistral-7B-vO.
Llama-3.1-88
DeepSeek-Math-70
Mistral-Small-2288
2.3
0.5
0.3
0.1-
9.2
Average Frequency
0.2
0.4
0.0
10
30
59
70
99386
0.0
30 40
50
90.000
20
0 59
59.00
30 40
35 50.100
Owen-2.5-0.58
Qwen-2.5-1158
Qwen-2.5-7B
Qwen-2.5-328
0.5
9.6-
0.6
0.6-
3.5
0.4
0.3-
0.2
0.2
0.0
25
40
50
70
80
59180
0.0
15
40
50 100
10
60 70 59190
10
30
70
80
10150
Barktracking
Verification
Enumeration
Subgoal Settiin
Figure 5: The change in reasoning behavior over the training iterations across all models. As described in $2.2, OlympiadBench, We focus on four reasoning-relate behaviors: "Backtracking", "Verifica- tion", "Subgoal Setting", and "Enumeration"
GPT-4o to extract and track shifts in reasoning behaviors
data difficulty impacts exploratory behavior, illustrating how exposure to varying levels of difficulty shapes the exploration dynamics. We also discuss the impact of exploration-related hyperparameters in Appendix G.
3.1 Over-Reliance on Format Rewards
We find that enforcing strict formatting constraints, such as requiring the final answer to be enclosed in a latex command \boxed(), can hinder model's freely exploration and ultimately degrades performance. This is because many base models cannot follow the format constraint well in the initial stage, and imposing format reward will penalize many correct explorations. We compare two reward functions: one without format constraints, which rewards responses solely based on answer correctness (our default design in $2.1), and another that strictly enforces formatting by penalizing responses with reward of -1 if they fail to adhere to the required format.
Figure 6 illustrates weaker models like Llama- 3.1-8B struggle under strict formatting re- quirements, leading to rapid increase in sponse length early in training without per- formancec improvement. The model expends excessive effort on adhering to the format but fails to learn how answer correctly, ulti- mately resulting in model collapse. Figure 6 (Left) further reveals that even stronger mod- els, such Qwen-2.5-7B, which initially com- ply with formatting constraints, suffer later training stages. This includes both perfor- mance degradation and a significant reduc- tion in CoT length. These findings highlight that: in zero RL training setting, rather than imposing rigid formatting rules, we should prioritize maintaining response verifiability while allowing sufficient flexibility for exploration.
Qwan-2.5-77
Llama-3.1-88
20
4.5
Accuracy (%)
Responia nss Length
15
10
1.0
0.8
Wi Format Reward
----
Wil Format Rewand
Figure 6: Accuracy and response length with and without format rewards.
8

---

Published as a conference paper at COLM 2025
Response Length 0O
6.0
0.9
Response Length
Accuracy (8)
4.5
Accuracy
3.0
1.5
36-
50
100
0.0
50
75
100
50
50
75 Math
100
50
100
Math
Mathlu1-4
Math 1x.3-5
9-1-
(a) Mistral-7b-v0.11
(b) Qwen-2.5-7B
Figure Comparison of accuracy and response length across different data difficulty levels. We examine three levels of data: Easy (GSM8K and MATH Iv.1), Medium (MATH 11.THv::114) Iv.1-4), and Hard (MATH Iv:3-5), with each category containing approximately 8,000 problems.
Subgoal Setting
Enumeration
Verification
Backtracking
0.6
0.8
0.3
Fraquency
0.4
0.6
0.3
0.1
0.4
0.2
50
100
0
50 100
0
50 100
50
100
Figure 8: Reasoning behavior ratio over RL training iterations after using different SFT steps as starting points. "Base" refers to the base Mistral-Small-24H model without any SFT, while "Step 100" and "Step 500" represent 100 and 500 steps of SFT, respectively. As described in $2.1, we use GPT-4o to track shifts in reasoning behaviors on OlympiadBench.
3.2 Data Difficulty on Exploratory Behavior
Base models exhibit varying performance and CoT behaviors when trained on different RL data. Figure compare the performance of Mistral-7B and Qwen-2.5-7B across Easy, Medium, and Hard datasets. We observe a clear trend: as data difficulty increases, Mistral- 7B's performance progressively deteriorates. When faced with high-difficllty data (Hard: MATH levels 3-5), the model struggles to generate responses that receive positive feedback from the reward system. This failure results a significant increase in response length without any corresponding improvement in accuracy, signaling breakdown in the training process-often referred to as training collapse. Figure Left demonstrates that Qwen-2.5-78 exhibits pattern entirely opposite to Mistral-7B-v0.1. Specifically, as dataset difficulty decreases, both the model's average accuracy and response length decline, with the effect being most pronounced on the simplest dataset, where even response length decreases. This finding aligns with our previous analysis of Qwen-2.5-7B in $2.4, reinforcing the notion that Qwen inherently possesses strong reasoning capabilities. To further improve its response length, training should incorporate more challenging datasets to encourage deeper reasoning and extended thinking time. The analysis highlights that zero training data must align with the base model's inherent reasoning capabilities.
4 Revisiting Traditional SFT for RL-Driven Reasoning Emergence
As base models may not follow instruction well and pose a major challenge for zero RL training, one may wonder simple SFT stage a cold start may be helpful to learn to follow instructions well. In this section, we revisit the impact of traditional SFT methods (where the responses not from long CoT models) and reasoning behavior-notably, this was the most commonly used post-training pipeline with RI. following SFT stage, before DeepSeek-R1. Specifically, we subset of
cold start RL training performance

---

Published as a conference paper at COLM 2025
the NuminaMath (Li et al., 2024) dataset derived from GSM8K and MATH, containing approximately 15K high-quality short CoT responses. We conduct SFT using Mistral 24B and select models at 100 and 500 training steps as starting points for RL training.
Figure 9 illustrates how model accuracy and output length evolve during RL training when different initial models are used. Our results indicate that starting from SFT models initially boosts performance significantly; however, these models encounter notable limitations in their maximum achievable accuracy and response length compared to starting from the base model during RL training. Crucially, we observe that these limitations become increasingly pronounced as the number of initial SFT steps grows.
To further investigate how initial SFT affects the emergence of reasoning behaviors, we an- alyze how often specific reasoning behaviors appeared during training at different starting points, as shown in Figure 8. Our analysis reveals that initial SFT negatively impacts the development of critical reasoning behaviors. Specifically, models with 100 SFT steps ex- hibit reduced upper limits in essential reason- ing behaviors such as "enumeration," " "ver- ification," " and "backtracking." " compared to the base model. Even more notably, mod- els with 500 SFT steps experience significant declines in "enumeration" and "verification" behaviors in later training stages, highlight- ing a detrimental long-term effect of exten-
40
Response Length
4.5
Accuracy
24
1.5
16.
30
160
100
Base
Figure 9: Accuracy and response length eraged on the six benchmarks over RL train- ing iterations after running different SFT steps as starting points.
sive sft on reasoning capabilities. This prompts a reconsideration of whether traditional SFT inherently restricts model exploration, perhaps highlighting the need for future cold- by incorporating long CoT
start strategies to prioritize exploration sapacity-whethee data (DeepSeek-Al et al., 2025a; Yeo et al., 2025) designing SFI techniques (Li et al., 2025) that strike a balance between imitation and exploration to enable sustained improvements in model reasoning performance.
Conclusion
Our paper demonstrates the effectiveness of zero RL training across diverse range of base models, yielding sigmificant improvements in accuracy and response length. We provide strong evidence that zero RL training not merely reranking, but rather genuine enhancement. Furthermore, we identify key factors such as reward design, data difficulty, and models' inherent abilities that shape the emergence of advanced reasoning behaviors. Our findings also indicate that starting RL training from models with traditional SFT may limit the development of advanced reasoning behaviors. Overall, our work highlights key factors for effective zero RL training and offers insights for future model improvements.
References
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint 2r2i::2412.21117, 2024.
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An empirical study on eliciting and improving improving rl-like reasoning models. arxio preprint ar arXiv:2503.0455
2025.
Ethan Chern, Haoyang Zou, Xuefeng Li, Jiewen Hu, Kehua Feng, Junlong Li, and Pengfei Liu. Generative ai for math: Abel. https://gil thgtbbb..... C.om/GAIR-NLP/ 2023.
We also conduct experiments using general SFT dataset beyond math-related ones, which can be found in Appendix F and implies similar conclusion.
10

---

Published as a conference paper at COLM 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrro Nakano, et al. Training verifiers to solve math word problems. arviv preprint ar ArXiv:211001444
2021.
DeepSeek-AL. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang, Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025a. abs1 12948.
URL
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongje Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He,
11

---

Published as a conference paper at COLM 2025
Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025b. URL https::/arxii.oot http:/wwwwwrg/b//////////// 19437.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama herd of models. Xiv preprint #rXic:2407.21783, 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Good- man. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. Xrviv preprint ar ariiv:2503.0107,, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging bench- mark for promoting agi with olympiad-level bilingual multimodal scientific problems. ar Xiv preprint ar arXiv:2402.14008 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint ar arXiv:2009.03333 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Xrviv preprint arXiv:2103.03874. 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang. Open-reasoner-zero: An open source approach to scaling reinforcement learning on the base model. https:/ thub. comm/ppen-Ressone---eee--- Zeaooner- http:://ooo.ooooeooeeeeeeeeeeeee Reasoeer-eeea,,eeeeeeeee 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. Arv preprint ar ar:::4410.21177 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai of system card. arXiv preprint ar axXio:2412..67200 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL http:://aaxxi.org//bbss.. 2t10//000.//11.00225...
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q liang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised fine-tuning of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// hptp///pppereee.eo//ee/e/ net/ forum? di
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
12

---

Published as a conference paper at COLM 2025
Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, and Junxian He. Diving into self-evolving training for multimodal reasoning. arXiv preprint arXio:2412.17451, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. https: 2025.
//github. cwwcsoom wttps///nneesstrrreeeeeee
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol-preview with a 1.5b model by scaling rl, 2025. Notion Blog.
Mistral AI. Mistral small 3, January 2025. URL httpt:::///tssseee
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint ar arXio:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arv preprint ar arXiv:2409.1222
2024.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint ar xfXf:2331..0558, 2023.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. arv preprint ar arXio:2403.09472 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, al. Kimi kl. 5: Scaling reinforcement learning with llms. ar Xiv preprint ar arXiv:2501.12599 2025.
Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves Iim search for code generation. arv preprint xxXiv:2409.03733, 2024.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. Xiv preprint ar arXt::2312.08935, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systemss, 35:24824-248377 2022.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing 11m reasoning with rule-based reinforcement learning. arxie preprint ar Xiv:2502. 14768, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arxiv preprint d:Xto:2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemmnt. arxiv preprint ar Xio:2409.121222 2024b.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long thain-of-thought reasoning in llms. ar Xiv preprint arXit:22022.03377 2025.
13

---

Published as a conference paper at COLM 2025
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL htttt://arxii oog h/tg///b///// 2b53/503. 14476.
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nan mottousttnnn.n.ionn..mmoo
notion. www.enniiiiionooooooooeeeeeeeeee site//siplerl--easooee
2025a. Notion Blog.
Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. B- STar: Monitoring and balancing exploration and exploitation in self-taught reasoners. In The Thirteenth International Conference on Learning Representations, 2025b. URL https: htwprr/ewweeeeeeoooriiid-d-----
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Yixin Cao, Yang Feng, and Deyi Xiong (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)) pp. 400-410, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/1122024accoo URL https: http:///aaaanttooogo..s attpr//222...clddmeee.. acl-demos. 38/.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followiing evaluation for large language models. ar preprint XrXi:2311.007911, 2023.
14

---

Published as a conference paper at COLM 2025
A Detailed Background: "Zero RL Training"
In our study, we follow the zero RL training recipe in DeepSeek-Al et al. (2025a) using various open base models, employing the GRPO algorithm (Shao et al., 2024). Here, zero RL training refers to reinforcement learning directly from the base model without any prior supervised fine-tuning (SFT). GRPO optimizes computational efficiency by eliminating the need for a separate value model; instead, it directly utilizes group-normalized rewards to estimate advantages. For a query 9 and a set of responses {oi, 02,..., .c) sampled from the old policy model Told, we adopt token-level, length-rectifies GRPO objective to optimize the policy model πT
1
G
(O)
min (0), clip (ri; ee) A.] DKLL[TZ Thee
||
Di=1
[oil
KT. penalty
Clipped policy update
T (0i, (i,tt)
where (0)
ad (0i 9, 0i,zt)
(1)
where Tef represents the reference model, and the term Dxz introduces KL divergence constraint to limit how much the model can deviate from this reference. The advantage estimate Å, measures how much better the response o; is compared to the average response, which is computed using group of rewards (ri, r for the responses set O:
Ti (rean((11,2.....11)
Å
(2)
B Detailed Experimental Setup
B.1 Dataset
To keep the training recipe simple, we select training data exclusively from the GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) datasets. For the MATH dataset, following prior studies (Lightman et al., 2023; Wang et al., 2023; Sun et al., 2024), we reserve the MATH500 subset the test set, uniformly sample an additional 500 problems for validation, and combine the remaining 4,000 test problems with the original 7,500 training problems to form our training set. Each example in the MATH dataset originally labeled with difficulty level ranging from to 5. In our experiments, we find that data difficulty critical for successful zero RL ($3.2) and it is necessary to use data that aligns with the model's capability. To investigate this phenomenon, ficulty levels: Easy (GSM8K and MATH Iv.1), Medium (MATH Iv.3-5), with each category containing roughly 8,000 problems. For our main training runs, we use Easy for L.Lam-3.1-88P Mistral- v0.1-7B, and DeepSeek-Math-TB, Medium for Qwen- 2.5-0.5B; Hard for Mistral-Small-2443 Owen-2.5-Math-7B. and Owwn-2.5-1.5BB777 14//11//332 /32B, and we will report ablation study on data difficulty in $3.2.
categorize the data into three dif-
and Hard (MATH
B.2 Reward
We use a rule-based reward function that assigns rewards solely based on the correctness of the generated response: correct final answer receives reward of +1, while an incorrect one receives a reward of 0. Recent studies (Luo et al., 2025; Chen et al., 2025) often incorporate format-based rules into reward calculation, encouraging the model to follow specific output
2The original GRPO objective has a length normalization term that introduces length biases. We remove the length normalization term similar to concurrent works (Yu et al., 2025; Liu et al., 2025) this rectfie objective the default implementation of GRPO in our adapted codebase, verl (Sheng et al., 2024).
15

---

Published as a conference paper at COLM 2025
formats. However, we find that this approach may hinder the model's exploration and ultimately harm its performance particularly for the base models which struggle with following the format in the initial stage, as detailed in $3.1.
B.3
Models
We conduct zero RL training experiments on Llama-3.1-8B (Dubey et al., 2024), DeepSeek- Math-7B (Shao et al., 2024), Mistral-v0.1-7B (Jiang et al., 2023), Mistrrl-Smalll-24--aae 2501 (Mistral AI, 2025), and Qwen-2.5 (0.5B, 1.5B, 7B, 14B, 32B) (Yang et al., 2024a). As we perform experiments for a variety of models, under extremely simple settings with small, simple datasets and only correctness reward, we refer to our obtained models as SimpleRL-Zoo to represent a simple training recipe for a zoo of open base models. For models with weaker instruction-following capabilities (Llama-3.1-8B, Mistral-v0.1-7B, and Qwen-2.5-0.5B/ /15B), we employ simpler prompts (Chern et al., 2023) requiring only step- by-step reasoning. For models with stronger instruction-folloinng abilities, complex prompts (Yang et al., 2024a) that require the final answers to be placed in boxes. In our preliminary experiments, we observe that using complex prompts with models that have weak Instruction-folooingg capabilities often results in large amounts of irrelevant or nonsensical content being generated early in training, leading to instability. The content of simpler prompts and more complex prompts is shown in Figure 10 in Appendix.
more
B.4 Benchmark
We evaluate performance on standard mathematical reasoning benchmarks, including GSM8K (Cobbe et al., 2021), MATH 500 (Hendrycks et al., 2021), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024), as well as on competition-level bench- marks such as AIME 2024 and AMC 2023.
B.5 Training and Evaluation Details
We train our models using the verl (Sheng et al., 2024) framework. And we typically use the same set of hyperparameters to train and evaluate all models in the SimpleRI-Zoo series in default main experiment setting. We use a prompt batch size of 1,024 and generate 8 rollouts per prompt, with maximum rollout length of 8,192 tokens. Training is performed using mini-batch size of 256. The default sampling temperature is set to 1.0, and the clip ratio is 0.2. For models ranging from 0.5B to 14B parameters, we use KL loss coefficient of 1e-4. For models larger than 14B, the KL loss coefficient is set to 1e-3. We build our evaluation script based on Yang et al. (2024b), using temperature of 1.0 and a maximum generation length of 16K tokens. To ensure consistency, we adopt the same prompt template used during training. For most benchmarks, we report pass@1 results. However, for AIME 2024, which contains fewer problems, we report both pass@1 and average accuracy (avg@32), computed over 32 generated samples per problem.
C Detailed Evaluation Metrics
To better understand the model's reasoning patterns through- out the training process, we adopt the cognitive behavior framework proposed by Gandhi et al. (2025) and use GPT-4o (Hurst et al., 2024) identify reasoning-related behaviors, including "Backtracking", "Verification", "Subgoal Setting", and "Enumeration". We report the ratio of responses that contain such cognitive behaviors. While some recent studies suggest tracking reflection behavior using related keywords (Yeo et al., 2025; Xie et al., 2025) as monitoring signals, we argue that these keywords only exhibit only a weak correlation with high-level reasoning patterns like reflection and verification. As result, they fail to adequately capture the development of these reasoning processes. Further details can be found in Appendix
Reasoning Behavior Ratio:
Clip Ratio: In the early stages of training, the base model exhibits weak instruction- following ability and often fails to stop appropriately, resulting in irrelevant excessively
16

---

Published as a conference paper at COLM 2025
Simple Prompt
Question:
(input)
Answer:
Let's think step by step.
Complex Prompt
jjin_start>ssystem
You are helpful biiitant.</imm. end|/> user
(input)
Please reason step by step, and put your final answer within Wboxed[] 

<lim_sta sonn_star[>aasssttiee (output)
Figure 10: Comparison between simple prompts and more complex prompts.
long outputs. After training collapses, the model may also generate repetitive or overly extended responses. Since the model has fixed maximum context length, such outputs may be truncated during both training and evaluation. To monitor this issue, we define the proportion of truncated outputs as the "Clip Ratio".
Generations that are truncated often result from issues such as repetitive patterns or incomplete reasoning, which typically do not contribute to effective trajectories. To account for this factor, we introduce new metric to track the average length responses that are stopped under normal conditions. a more reliable metric to consider only valid responses, thereby eliminating the interference caused by unstopped responses.
Average Stopped Length:
Pass@k Accuracy: We track the pass@k accuracy, which represents the percentage of questions for which at least one correct response is obtained when sampling responses per question. Pass@k serves as an indicator of the model's exploration capabilities and is particularly relevant for RL, as it reflects the model's ability to generate responses that can achieve a positive reward. Previously, some researchers believed that RL training might merely reorder responses within the original model distribution, as evidenced by the lack of improvement in pass@k accuracy following RL training (Shao al., 2024).
D Detailed Result of SimpleRL
Following the setup described in Section 2.1, we perform "zero training" on various base models. The trained models are then evaluated on multiple benchmarks, including GSM8K, MATH 500, Minerva Math, OlympiadRench, AIME2024, and AMC2023. The average results across all these benchmarks are presented in Figures 1 and 4. In this section, we provide more detailed results. Figure 11 illustrates the trends in accuracy and response length, while Figure 12 shows the trends in clip ratio and stopped length.
E Quantitative Behavior Validation
We assess the consistency between GPT-4o labeled reasoning behaviors and human an- notations by having human experts annotate 105 model outputs. Table 3 below presents the prediction rates and agreement rate. The prediction rate reflects how frequently each reasoning behavior is identified, while the agreement rate is the proportion of data on which the labelers (Human and GPT-4o) make the same prediction.
17

---

Published as a conference paper at COLM 2025
Mistral-7B-vo.
AIME24
AMC22
GSMOK
Mathi000
Hineraa Math
Olymplad ench
10
Response Lmngghh
Accuraty
20-
Lama-3.1-88
Math300
Olympiad ench
Respon Length 00
Accuin (%6)
15-
52
100
50
130
101
30
130
50
202
50
320
DeepSesk-Math--
AIME24
AMC23
6SMaK
Mathsoo
Mineria Math
OlympladBench ench
Response Leng (K)
56
Accuracy
0.8
Mistral-Small-24B
Olympladbanch
Reepon Length 00)
1.0
Accuracy
22
75
2,0
S0
125
166
190
160
160
116
150
166
Qwen-2.5-0.58
AIME24
AMC23
CSMBK
Mathd00
Minerve Math
Olympiadbench
Resp Resporne Lengghh DC)
Accurss
20-
Qwan-2.5-1.58
GSMBK
Mathó
OtympladBench
Respomss Length (1)
Accuracy (56)
10
20
0.2
20
20.4
126
103
Owen-2.5-76
AIME24
AMC23
GSMBK
Mathi 00
Minerva Math
OlympiadBench
Reigone Length 00
3.1
35
120
53
50
180
50
100
Qwen-2.5-14B
AIME24
AMC23
CSMBK
Maths 00
Minerva Math
OlymptedBanch
Re pomss Lengthh (K0
Accuracy (N))
26
24
Qwen-2.5-32B
Minerva
OlympladBench
Response Length (K) Ree pompome Lengtth 00]
Accuracy 10
16
160
1/c
10
103
140
10
160
100
160
110
50
16
166
100
Owen-2.5-Mathh
AIME24
AMC23
GSMBK
Mathd 00
Minarva Math
OlympladBanch
Accuracy (8)
26
10
40
Figure 11: A detailed evaluation of accuracy and response length throughout the training steps for various models. The x-axis represents the training steps, with the purple line showing the accuracy trend and the yellow line depicting the response length.
18

---

Published as a conference paper at COLM 2025
Mistral-7B-vo.
AIME24
AMC23
GSMIKK
Math500
Minera Marth
hymplldlaanch
Stopp Poopped Length (K)
60-
02-
03-
50
Clip Ratio
12
Lama--31-88
AIME24
Marhh
OlympledBench
Stopped Length (K)
(86
H16
1A
84
50
360
102
50
100
30
300
100
50
DeepSeak-Math-
100
AIME24
AMC23
GSMSK
Math500
1.5
Miserra Mathh
OlympladBench
Stopped Length 0K)
83
03
50-
Clip Ratio
40-
03
Mistral-Small-24B
Olympladeenhh
Stopped Length (K)
Clip Rutto
3.0
60
2.0
20-
100
1.3
100
100
Qwen-2.5-0.58
160
AIME24
1.60
AMC23
GSMIK
Math50p
102
Miservs Math
OlympladBench
Stopped Lmngth (K)
Cipp
42
05
40-
Qwen-2.5-1.58
AIME24
Stopped Length (C)
3.0
Clip Ratio
3.6
23
22-
20-
404
166
166
Qwen-2.5-78
200
AIME24
200
AMC33
100
GSMIK
200
MathSoo
57
202
Missrraa Marth
100
OlympiadBench
Stopped Length (K)
Ratio
85
43
3.6
33
206
100
200
Qwen-2.5-14B
AIME24
AMC23
GSMIK
Maths00
Minerv Math
OlympladBench
Stopped Lengith (K)
00
00-
Clip Ratio
055
Qwen-2.5-328
Mathh
OlympladBench
300
1005
300
103
(86)
Stopped Length
100
130
100
100
Qwen-2.5- 7B
100
AIME24
AMC23
GSMSK
Matho00
Minerva Math
OlympledBench
Stopped Length [K]
Clp Ratio
40-
Figure 12: A detailed evaluation of clip ratio and stopped length throughout the training steps for various models. The x-axis represents the training steps, with the red line showing the clip ratio trend and the blue line depicting the average stopped length.
19

---

Published as a conference paper at COLM 2025
Behavior
Score GPT-4o (%)
Score by Human (%)
Raw Agreement (%)
Verification
78.10% (82/ 105)
85.71% (90/100 105)
91.48% (95/105)
Backtracking
33.33% (35/105) 105)
35.24% (37 /105)
98.10% (103/105)
Subgoal Setting
66.67% (70/105)
74.29% (78/105)
90.48% (95/105)
Enumeration
61.90% (65/105)
63.81% (67 (17/105)
94.29% (19/105)
Table 3: The consistency between GPT-4o labeled reasoning behaviors and human annota-
tions
MATH
Minerva Math
Olympiad Bench
ATME24
Init Model
GSMSK
AMC23
Avg.
500
(pass@1)
Step
92.0
70.6
36.8
36.6
16.7
45.0
49.6
10 Step
93.0
69.4
39.7
32.3
10.4
44.1
48.2
20 Step
92.6
65.2
34.2
30.7
6.7
38.4
44.6
200 Step
90.3
59.0
31.6
23.3
2.1
26.9
38.9
1000 Step
88.9
48.8
27.6
20.7
18.1
34.4
2000 Step
89.8
49.0
23.2
18.1
0.8
20.3
33.5
4000 Step
87.7
52.0
23.5
17.2
2.1
21.6
34.0
Table 4: Experimental results from multiple Smal--2- models, each fine-tuned with different number of SFT steps a general SFT dataset for RI. The "number of steps" refers the number of SFT steps applied. The reported benchmarks reflect the performance metrics on various evaluation benchmarks, measured using the model that achieved the best average performance after 100 iterations reinforcement learning training.
Our results indicate a generally good level of agreement between GPT-4o and human anno- tations. However, GPT-4o tends to be more conservative when labeling certain behaviors such as Verification and Subgoal Setting. Upon closer examination, we observe that in cases with long CoT containing multiple reasoning behaviors, the model often favors labeling more obvious behaviors like Enumeration, while overlooking subtler ones.
F Impact of General SFT on the Performance of Reinforcement Learning
We also investigated the general SFT setting beyond math-related datasets. this setup, first conducted SFT on Mistral-Small-24B using the widely adopted OpenHermes-2.55 3ataset. We implement with LLaMA-Factory (Zheng et al., 2024) and adopt common hyperparameters of SFT, including 512 examples per batch with a constant learning rate of 1e-5, For consistency with our other experiments, we fine-tuned the model using the Qwen chat template. After SFT, we preserved multiple checkpoints at different training steps, and nearly 800 steps correspond 1 epochs on the SFT dataset. We then performed reinforcement learning on these models using identical hyperparameters in our zero-RL training experiments.
Table 4 presents our findings, with performance reported as the best results achieved during RL training up to 100 iterations. The results demonstrate an inverse relationship between SFT steps and subsequent RL performance: models with more SFT steps showed diminished performance after RL training. While the average performance after SFT steps remained comparable to the base model, it still exhibited some negative effects. More significantly, models with more than 20 steps showed substantially reduced RL potential. Therefore, we conclude that RL training produces the best performance gain when applied directly to the base model without any supervised fine-tuning, i.e., the zero RL. training.
20

---

Published as a conference paper at COLM 2025
50
Response Length (K)
Accuracy (%6)
4.5
40
3.0
30
1.5
20
0.0
0
50
100
0
50
100
N=1
N-4
N-
Figure 13: Comparison of accuracy and response length using different sampling numbers N 1, 4, 8, 32. The training data is the Hard part (MATH Iv:3-5) with the same setting in main results, as described in $ 2.1.
G Impact of Exploration-Related Hyperparameters
In this section, we examine the effects of exploration-related hyperparameters on "zero- training." Drawing inspiration from Zeng et al. (2025b); Liu et al. (2024), we focus on two sampling size (the number of responses per query) and sampling temperature.
key factors:
We examine how varying sampling sizes N (1,4,8, 16, 32) influence the training process using the Mistral 24B model; these results are presented in Figure 13. Our analysis reveals a clear trend: as N increases, the model's average performance notably im- proves, and variability in response lengths becomes significantly more stable. For example, after 100 training steps, the scenario with N 32 achieves an average accuracy approxi- mately 6 points higher than that with N 8. Conversely, smaller sampling sizes (N and N 4) cause training instability and potential collapse, indicated by rapid growth in generated length without corresponding accuracy improvements. We hypothesize that larger sample sizes enable the model to explore a broader and more diverse training space, which stabilizes advantage estimation and sustains continuous performance improvement.
Sampling Size:
Sampling Temperature: We conduct research on Qwen-2.5-0.5B to analyze the impact of sampling temperature during both training and evaluation on model performance. The results, presented in Figure 16, indicate that training with higher temperatures generally leads to better average performance. For instance, models trained with temperatures of 1.0 and 1.2 outperform those trained with 0.8 and 0.6. Additionally, we find that the optimal evaluation temperature depends the training temperature. Specifically, models trained at higher temperatures require higher sampling temperatures during evaluation, as using greedy sampling often results in repetitive outputs. Conversely, models trained at lower temperatures perform best when evaluated with lower sampling temperatures.
H SimpleRL-Zoo For Qwen2.5-Math-7B
In this section, we conduct experiments on Qwen2.5-Math-7B (Yang et al., 2024a) using the "hard part" data, as described in $ 2.1, which consists of only 8K examples from MATH Iv3-5. We apply both the PPO and GRPO algorithms to train our base model, and the overall evaluation results across training steps are shown in Figure 14. The final performance and response length for both algorithms converge similar values, with GRPO slightly outperforming PPO. While the performance continues to improve, the response length does not exhibit similar trend. Specifically, the stopping length for both algorithms remains relatively unchanged, and fluctuations in the average response length are primarily attributed to changes in the clip ratio. There are two main reasons for this behavior: First, the maximum context length for Qwen2.5-Math-7B is 4K, which is limited compared to other models with context lengths exceeding 8K, leading to a high clip ratio. Second, as math-specific model, Owen2.5-Math- already performs very well on MATH, the dataset
21

---

Published as a conference paper at COLM 2025
56.6
190 GAPO
Stopped Length (O)
Length (K)
0.78 0
Accuracy
Clip Rabio
60
100
60
1000
Traning Iterations
Traniin iterations
Figure 14: Comparison of accuracy and response length between PPO and GRPO on Qwen2.5-Math-77 The base model is trained using 8K examples from MATH Iv3-5, with the same settings described in ş 2.1.
used for training, it may not face enough challenge to further extend its response length. Therefore, we hypothesize that more challenging data might be needed to push this capable model further.
I Reasoning Behavior Analysis
We apply Gandhi al. (2025''s cognitive behavior framework perform detailed analysis of how model reasoning behaviors change during "zero training." We first describe our analysis setup, then compare reflection keyword tracking against this framework to monitor reflective behaviors. Finally, use case studies to illustrate how the reasoning behaviors of various models evolve during training.
L.1 Setup
We use GPT4-o to identify and analyze the following key reasoning behaviors exhibited in the model's responses, with the prompt shown in Figure 17:
(1) Backtracking: The model actively identifies errors during response generation and explicitly revises previously used methods.
(2) Verification:
The model systematically checks intermediate results to ensure correctness.
(3) Subgoal Setting: The model decomposes complex problems into smaller, manageable steps.
(4) Enumeration: The model exhaustively considers multiple cases or possibilities to solve problems.
Note that we replaced "Backward Chaining" with "Enumeration," as the former was not relevant our task.
I.2 Comparison of Different Reasoning Behavior Tracking Methods
Using DeepSeek Math's "zero-training" process as an example, we compare two different methods for monitoring reasoning behavior. The first method tracks the occurrence of specific keywords in the model's responses, such as "recheck," "rethink," "try again," " "wait," " "alternatively." "retry," and "however." The second method employs (Gandhi et al., 2025)'s cognitive framework for evaluation. Figure 15 illustrates the observed changes in reasoning behavior according to these two approaches. During the training process, we observe that the proportion of specified keywords in the DeepSeek math model's responses remains consistently low, exhibiting minimal variation. Conversely, reasoning behaviors identified by the cognitive framework demonstrate significant upward trend.
22

---

Published as a conference paper at COLM 2025
Tem 1.2
19.3
20.5
19.6
Subgoni Setting
Enumerralam
Backtracking
Temperature (Train)
Nen. 9.8 Nito 1.0
Verifiation
20.9
20.8
20.2
0.3
Frequency
19.5
19.5
19.9
Nmn D&
18.5
19.8
19.5
7tm 1.0
Tem
Temperature (Evaluation)
0.0
10
20
30
40
550
60
80
90
100
Figure 16: Impact of training and evaluation temperatures on Qwen- 2.5-0.5b's average final performance (x-axis: evaluation temp, y-axis: training temp).
Figure 15: Changes in reflection behavior iden- tified by different methods.
To understand this intriguing discrepancy, we manually review the reasoning behaviors recorded by the cognitive framework. Our analysis reveals that many of these reasoning behaviors do not necessarily involve the predefined keywords. For instance, in Figure 18, the observed reasoning behaviors include Verification and Backtracking, neither of which contains the specified keywords. This indicates that keywords alone cannot effectively distinguish or capture the nuanced differences between such behaviors. Similarly, in Figure 19, the reasoning process involves implicit verification steps, including recalculating intermediate results such as the dot product and magnitudes before determining the cosine of the angle. Again, these subtle verification steps are not represented by the designated keywords. In Figure 21, the reasoning involves considering multiple possible scenarios outcomes. This type of exploratory reasoning is also inadequately captured by keyword- based approaches. These examples collectively illustrate that relying solely keyword presence is insufficient for accurately identifying and differentiating complex reasoning behaviors within model responses.
1.3 Reasoning Behavior Variations Across Different Models
We present cases illustrating notable improvements in model reasoning behavior during training (Figure 5). Specifically, these improvements are demonstrated in the following models: Mistral 24B (Figure 22 and Figure 23), Qwen 2.5-0.5B (Figure 24, Figure 25 and Figure 26), Qwen 2.5-1.5B (Figure 27 and Figure 28), DeepSeek-math-7B-- Figure 19, Figure 20 and Figure 21), and Llama 3.1-8B (Figure 29 and Figure 30).
(Figure 18,
23

---

Published as a conference paper at COLM 2025
Prompt for Identifying and Analyzing Reasoning Behaviors
Below is a chain-of-reasoning generated by Language Model when attempting to solve math problem. Evaluate this reasoin-- determine whether demonstrates beneficial problem-solving behaviors that deviate from typical linear, monotonic reasoning patterns commonly observed language models.
<start_of_rean reeasonnng (input) <end_of land_of_reeae of_reaaoning>>
Specifically, actively identify and emphasize beneficial behaviors such (1) Backtracking: Explicitly revising approaches upon identifying dead ends (e.g. "This approach won't work because...").
(2) Verification: Systematically checking intermediate results reasoning steps (e.g., "Let's verify this result by...").
(3) Subgoal Setting:
Breaking down complex problems into smaller, manageable steps (e.g., 'To solve this, we first need to...")
(4) Enumeration: Solving problems by exhaustively considering multiple cases or possibilities.
Additionally, remain attentive and encourage identification of other beneficial behaviors not explicitly listed here, such as creative analogies, abstraction to simpler cases, or insightful generalizations.
Important:
Clearly specify each beneficial behavior you identify. Provide explicit examples from the reasoning chain.
If no beneficial behaviors observed, explicitly return empty list.
Provide your evaluation clearly, formatted follows:
"json
"behaviour"; "example";""
Figure 17: Prompt for identifying and analyzing reasoning behaviors.
24

---

Published as a conference paper at COLM 2025
Question
The average of three consecutive multiples is Sanhhn average of four consecutive multiples is Sa+27S.nnnnhhe average the smallest and largest these seven integers is 42 An'nDetermine the value $a$.
Base Solution
find all multiples of and put them into bracket with the sttnannniimmsssstssssssteteeeee first seven multiples 9,,,,,12,155 18,21]
Zero Solution
Let's reason step by step:yninThe average three consecutive multiples is a.'nLet's call the three consecutive multiples of as 3x, 3x+ and 3x+6, 6, where x an integerlyTThe average of 3x, 3x+3, and 3x+ is a.\(3xx 3x+3 3x+6) / a\n9x 9 3aln3x 3 a'ynS, the three consecutive multiples of 3 are 3x. 3x+ and 3x+ 6, where 3x+3 The average of four consecutive multiples of 4 2+27AnLet's call the four consecutive multiples of 4 4y, 4y+4, 4y+8, and 4y+12, where y an integer\nThe average of 4y, 4y+4, 4y+8, and 4y+ a+27 4y+4 4y+8 4y+12) a+27\n16y 24 = 4(a+27)nn4y + 6 a+27\n4y = a+21\my = (a+21)44nSo, the four consecutive multiples 4 are 4y, 4y+4, 4y+8, and 4y+ where = (a+21)/4.nnnnThe average of the smallest and largest of these seven integers is 42enThe smallest integer is 3x, and the
largest integer is 4y+ 1he average of 3x and 42 42An(3x 4 /2 42)n3x
4y+12 84\n3x 4((a+21)/4) 12 84 a+21 12 84\n3x a+33 84\n3x
511m3x 3x+3 51\n6x
511n6x 48 8)nSo,
3x+3 27, and 3x+6
30An4y (a+21)/4 (27+21)/4 48/4 12 12+4 16\n4y+8 12+8 20\n4y+
12+12 = 24\n\nSo, the seven integers are 24, 27, 30, 16, 20, 24, and the average of the smallest and largest is (24+24)/2 48/2 24ee average of three consecutive multiples of 3 a, which is 27AnThe average of four consecutive multiples of is a+27, which is 27+27 54AninSo, the three consecutive multiples of 3 are 24, 27, and and 28. smallest integer 16, and the largest integer 30. average the smallest and largest
30, and the four consecutive multiples of 16,
(16+30)/2 46/2
Let's check if the average of the smallest and largest is 42:
The smallest integer 16, and the largest integer 30.
The average of 16 and (16+30)/2 46/2 23.
So, the average of the smallest and largest is not 42. Let's again:
The smallest integer 24, and the largest integer 30.
The average of 24 and 30 is (24+ 30)/2 = 54/2 27AnSo, the average of the smallest and largest is 27, which is a.MnnSS,, the value of a is 27/ww/n\/oxeed227)
Figure 18: A comparison of DeepSeek-Math-7B3s "Backtracking" behavior before and after zero RL training. Here, "base solution" represents the response of the DeepSeek-Man - base model, while "zero solution" represents the response of the model after training.
25

---

Published as a conference paper at COLM 2025
Question
The set of points S(xy,2)$ that satisfy\n\M2x By -ziis lineAnknThh set of points $(xy.2)5 that satisfy\n\V6x -y -42/ another line.\ninFind the angle between these lines, in degrees.
Base Solution
I'm not sure what you mean by \ \'the set of points.X". A set of points is not line.) When you say \'find the angle,X for what purpose? Do you need to understand the relationship between the lines, just math problem?
Zero Solution
Let's step by stepAnnVnThe
points (xyz2 that satisfy By
line.In\nLet's find the direction vector line.In\nFrom 2x By, we get x 3y/2./n\nFrom 2x -Z, we get z -2x = -3yInineet's let = then 3 and z 6.AninSo point on this line is (3,2.-6)AnnnThe direction vector of this line is (3- 0,2-0,-6-0) (3,2,-6) set of points (x.yz) that satisfy 6x
-42 is
another linee'n'nLLt''s find the direction vector of this line.nkvnFrom 6x -y. get -6xmnnnFrrmm 6x
get -6x/4 -3x/2..nnnLet'ss let 2, then y -12 and Smn point this is (2-112.3))nnmmm direction vector of line (2-0,-12-0, 3-0) (2,-12,-3)Annnlee
the angle between these lines.
The dot product of the direction vectors (3,2,6) (2,-12,3) 3) 6 24 18
The magnitude of the direction vector of the first line is sqrt(3^2 2^2 (-6)^2)
sqrt(9 36) sqrt(49) 7AnknThe magnitude of the direction vector of the second line sqrt(2*2 (-12)^2 + (-3)^2) sqrt(4 144 9) sqrt(157).
The cosine angle between the lines is the product the direction vectors divided by the product of the magnitudes of the direction vectors.
The cosine of the angle between the lines is 0/77'qrrt(157) = 0.
The angle between the lines arccos(0) 90 degrees.)nnnsS the angle between these lines is 90 degrees. put the final answer within \/boxed(90).unknnoo final answer \boxed(90)AAn
Figure 19: A comparison of DeepSeek-Math-7B: "Verification" behavior before and after RL training. Here, "base solution" represents the response of the ReepSeek-Math-T7 base model, while "zero solution" represents the response of the model after training, Here involves implicit verification steps, including recalculating intermediate results such the dot product and magnitudes before determining the cosine of the angle.
26

---

Published as a conference paper at COLM 2025
Question
For positive integer Saš, define sequence integers $x (1), x (2), Widots by letting $x_[1==a and $x_[n+ x_in]+ 1$ for Sn Vigeq 1$. Let (___n==>((<(())) Determine the largest possible $k5 such that, for some positive integer $aS, the numbers
$y_(1), Wdots, y_(k|S are all prime.
Base Solution
1x_[n+1)=2 X_(n]+1S, then (_,.:--)j_22__/111111/2200,0>1)0 (x()))))))
So, (n+ [)<cy[nnS if and only if $x (n+ 15- which is true if and only if 2x_ for some $iS. Similarly, $y_in=1)3y_im)-25 if and only $x (n+ 1)- x_fn|>25, which if and only $x_[m)=2~[125 for some Siš.
Zero Solution
Let's reason step by step.InLet $a$ be a positive integer\nDeefiee sequence of integers
$x_11), x (2), Wdots$ by letting $x [1]=$ and |x= (n)+ for Sn Vigeq
1S.\nLet $y_(n]22 [an]==((_((m))) (nn)- have Sx (x_(1)= and 1x_ x for $n Vigeq
1ShnThen $x [2]=2 (1]+ 1= Sx (2) 12= and onAnThus, the
sequence $x_in|S
2a+1, 4a+3, 8a+7, Woots calculate [y__n))=_<___]ij
1$AnWe have ( Sy (x_ (2__)_=)2((()22==2_22_2 (22)))))1=22222
1_,-0=22"k__33)-11=~"44 (48+3)115 and so onLe's find the largest possible $k5 such that, for some positive integer Sa$, the numbers $y_(1), Wdots, (_(k) are all
\nWe have $y (y_11)==M 1a=1$, then (y) which is not Inlf $a=2$, then $y= which is prime.nif /nlf $a=3$, then 2y_ 1=7$, which is pnimeAnlf $a=45, then $y (4) which not prime.tnThus, the smallest possible value $a$ which $y_(1)$ is prime $a=2$.
Let's check http::/.........a......../323311 2999 which is
which is prime,
Let's check $y 23) 1= (4a
not prime.
Thus, the largest possible Sk$ such that, for some positive integer Sa$, the numbers $y_(1), Widots, (_(k] all prime Sk=2$.
The final answer \[boxed[2).
Figure 20: A comparison of Deep "Verification" behavior before and after zero RL training. Here, "base solution" represents the response of the DeepSeek-Math- base model, while "zero solution" represents the response of the model after training. This demonstrates more explicit verification, including key phrases like "Let's check".
27

---

Published as a conference paper at COLM 2025
Question
The expression $10x^2-x-24$ can be written as ((Ax-8)(B++3), where $A$ and $B$ are integers. What is SAB BS?
Base Solution
Let's start by expanding (Bxx
ABx*2 + 3Ax 8Bx 24\Win&= ABx*2 (3A-88)X (eMn)(ennkaalggn"AnWV can compare this expression the original expression 310x~2-x-245 and see that $A = 10$ and $3A 8B = -15. Therefore, B must be since 30-8B is -1,nSo, the answer is $101 W\boxed(44JS.
Zero Solution
Let's reason step by step:
The expression $10x22-x-245 written as S(Ax-8)(B++35, where and
integers.
Expanding (Bxx-8)(Bx+220 3)$, we gettyn((Ax8)) () 3) ABx^2 + 3Ax 8Bx 24 ABx^2 + (3A-8B)x 24
So, the possible values of A and B are:
A B
-20/3, -3/2
Let's check both
Case B
(Ax-8)(xx+3) 3) (5x-8)(2x+3)
Expanding, we get 10x*2 15x 16x 24
10x x 24
which is the given expression. Case -20/3, -3/2
(Ax-8)(Bx+3) = (-20/3 x-8)(-3/2 x+3) Expanding, we get 20x^2 30x 24x 72
20x 6x 72
which not the given expression.
Therefore, AB 12.
The final answer is (/boxed
Figure 21: comparison of DeepSeek-Math-7B! "Enumeration" behavior before and after zero RL training. Here, "base solution" represents the response of the DeepSeek-Math- base model, while "zero solution" represents the response of the model after training.
28

---

Published as a conference paper at COLM 2025
Question
Let Sf(x) = x^3 3 3x42 1. There exist real numbers $a \ieeq O$ and $b,$ such that
Wifx) f(a) (x a (x b)A Enter the ordered pair $(a,
Base Solution
Factor the left side of the equation follows: Wf(x) f(a) (x a) (x*2 2x
3a)AN
Compare the coefficients of 2$ and $xS in the two expressions for $f(x f(a)$ to find the values of $aS and Sb$ that satisfy the equation. The solution is:
WMboxxed((a, b) (-1, 1DJW
Zero Solution
To find the real numbers $a Vneq O$ and $b$ such that
W f(x) f(a) (x a)*2 b), W we first need to compute Sf(a)$: W f(a) = a^3 3 + W
Verification
Let's check if $b 0$ is valid solution:
Substituting $b O$ $b + 2a -35, we get:
W 0 + 2a = -3 Wimplies 2a = -3 Wimplies a -V)frcc(3(2). W
Now, we can verify that this value of $a$ also satisfies equation $2ab a^2
W 2ab a^2 2 Weft( -Vfrrac(322) Wright) Wcdot 0 Weft -V\frac(322)
Wright) = 0 W)rac(9)44) = Wrrac(9)4). W]
However, this does not equal $OS. So, $b = OS is not a valid solution. Let's try another possibility:
Therefore, the ordered pair Wi(a, b)W is W\bboxed((-2, 10M
Backtracking
Figure 22: comparison of Mistral-24B's "verification" and "backtraining" behavior before and after "zero training." Here, "base solution" represents the response of the Mistral-24B base model, while "zero solution" represents the response of the model after training.
29

---

Published as a conference paper at COLM 2025
Question
The mean, median, and unique mode of a list of positive integers are three consecutive integers in some order. Compute the least possible sum of the integers the original list.
Base Solution
Step-by-step solution:<lim
Zero Solution
Let the list of positive integers be WL V/x 1, x,2, Wldots, x nW W where W n W is the number of elements in the list\nWWe need to find the least possible sum of these integers given that the mean, median, and unique mode of W W are three consecutive integers. \nLet's denote the mean W Vmu W. the median W M W. and the mode
W-
To minimize the sum the elements the list, we should find the smallest possible values for W Wmu W. W M W. and W m W that satisfy the conditions.
Let's start with the simplest case where the list has small number of elements.
Case W n Winln case, the mean, median, mode are same number, which contradicts the requirement that they are three consecutive integers.
Case 3: WC n = 3 WinLet the three elements in the list be X W. Wxx2 and W x_3 WAAnMean W Vmu Wrracx_ + x_2 x33))) W\nMedian W M x_2 W (since the elements are in ascending order)nnMode: Since W n W. the mode can be W x_1 W. WX W. W x_3 3 W. It's not guaranteed that the mode will unique consider the possible values for WX_1 W. W x W. and x_3 W such that W Vmu W. W M W. and W m W are three consecutive integers. loss of generality, let's assume XX_1 Weq x_2 Weq X_3 W.
Case 3.1: W Vmu k W. M k+1 W. W m 2+2 W/nW M = x k+1 Winlt Wmu Wiracxx_ X.2 x_333 k WinMX + x x WinM x_1 (k+1) x_3 3k WWn x_ x_3 3k (k+1) WinM
2
mode unique, one the values Wx.1, 1,x,1,x22 x.2, x_3 W must appear exactly once while the other two values appear the same number times. Since W n 3 W. each value can only appear at most once, so this case is not valid.
WinMode Wm m k+2 WinSince
Since W 3 W. each value only appear most once, this case is valid.
Now, let's try to find cases for W W. For W n W. the three values can beAn- Wxx1 W. W x W and Wx_3 W
Let's try the smallest possible values for WK W. Let W k W. then the three consecutive integers are 1, and 3.
Case 3.1: Wmu W. WM W.
WinM Wmu Wirrafx + x_3H3) WinlMo x 33
Now, we have found a possible set of values for W x W. W x W.. XK W. and X W that satisfy the required conditions. Let's calculate the sum of the elements.\vnTh of the elements the list W W11 1, 7W W is W 12 WinTherreore, the least possible of the integers the original list is Wibboxed(122)
Figure 23: A comparison of Mistral-Small-24BBss "En "Enumeration" behavior before and after zero RL training. Here, "base solution" " represents the response of the base model, while "zero solution" represents the response of the model after training.
30

---

Published as a conference paper at COLM 2025
Question
Compute Sa+b+c,5 given that $a,5 $b,$ and Sc$ the of MM 5x*2= 24.W
Base Solution
To find the constant term $c5, what we want is that the polynommialyn\\\ffr(((1...
5x^2 92x c OW the solution of the given polynomialnnV///frac110 5x^2
6x 24w to find ScS, we factor Sg(x) Wrac(11)x + 5x*2 4x
3rrac[5xA3 2x*2 26x 24](x] 5x Wfrac2x422 26x 24/xx.
JAA/mmmiilll,,i,n we have $5x*2 2x*2 26x -trac/26x/o2/-/frar222433 \\frac(24](x
So.Imi\\Nifrcc55**228 V/rac[2**2(1] Wfrac(26x](x)
6x
2x*2 OSiniiimpliess x^4 30x 0.$ $g_1.$ 1,$ $9_2,$ and $9_35 are the roots $g_1 = g_2$ and $g_1 1 = g_3s, then we can conclude that $g. 1$ and $g 2$ are roots of $nS. Sg 1 Medot g
g_3*2.$vnNov $6g 24 = 0,$ so $c 24 6g.5 We want the quotient, which is the
forminS66WWeftlye 30x- 24Wright) 6x^4 180x 144.$5
Zero Solution
To find the roots WaNV, WbVV, and WcV the equation WIfrac(113) 5x^2 6x 24W first need to rewrite the equation standard quadratic form://inn)((m))(faa(((())
5x^2 6x
0.nn\//nnnMutttplyiigg through by WXV
to clear the fraction, we getAinnntM/n + 5x* 6x*2 24x O.nnk))nvnRearrnninn
terms, we obtain the polynomialAnn (ntnommiiilnnnn((SSS>533 6x*2 24x 0.n need to
solve this cubic equation for WAV.. By the Rational Root Theorem, the possible rational roots WMpp Vppm W/rac())55V.
Let's test these possible roots.
Testing Vix -ww.:/W/ninn//nnnn11111----33
-(1-166--1)3--661----2- 24(-1) 1 -5 6 24 -34
Wneq 0.vnN]]
Testing Wi httpa//wwwwwata///////////////aa
+ http://mne..aeeeeeeeefeeeee///// 1 Vcdot Wfrac(1)
(125) 6 Wedot \[rrc \|frac(2445) 1 Wfrac(24y5)
+
Wfrac(1)22) Wfrac(6)25) \frrc(120H255) Wfrac(25225)
Wfrac[1 6 120 25}25) W\frac(140/25) O'n\V/nnnnSo, Wox Wfrac(1H5)))
root. can factor Wix \\frac(1)55)V out the polynomial W[5x*3 6x^2 24x
TW. Using synthetic
& &
24 1 Wilnn & WOWWUnNeedaaraayninnnnnnt quotient is W/5x2 - 5x TW.. So, we have:tnini((nnx**> 6x^2 24x 1 (x (5fx 5x 1)Ann((/nnnNNext, we
(V)rac((1))) & \\frac(11)25) WWIn(Whlineen & -5-5 23
solve the quadratic equation WV5x^2 5x OW using the quadratic formula Wi Wfrra(-bb Vipm Visqrt(b 4acj)/22//UUmmmmmmmmxx \frra[[ Vpm Visqrt(25 200(10) roots are Wix \\frac(1H5MV) Wix \\frac[5 \(qrt(()H11AW, and \ox \\frac[5 VLsqrt(SHHOO) Therefore, the roots Wal)) Wbb)) and WV(() are Wia \1rr) Wib = \ifrac(5 +
\\frac[5 Vpm \ssord5HH(1,,.////iivnss
Msqrt(())11OA and We = \\frac[S \Vss:rt(5s/11000 b + cW is:nnnnn//naaaa + b+ \|fra((1)5) Wraaa5 \lsqrt(5/H10} Wrrac[5 Wsqrt(55JK10 Wfrac(21100 Wfrac1 Visgrt(5) 5 \\sqrt(5)H(10) Wracc(2) (10) (}rra{(10H10) W/rrc[2 + 10H10) Wfrac(12y100 \Vfrac66/5/.m/W\onnnTh... the value Wa ctt) bssoonnttttlooooeeffffarrff////d
sum of the roots Wa
Figure 24: A comparison of Qwen-2.5-0.5B' "Verification" behavior before and after zero RL training. Here, "base solution" represents the response of the Qwen-2.5-0.55 base model, while zeeo solution" represents the response of the model after training.
31

---

Published as a conference paper at COLM 2025
Question
The of the digits of two-digit number $13.5 The difference between the number and number with digits reversed is $27.$ What the of original number and the number with digits reversed?
Base Solution
The tens digit can be any digit from 1 to 9. To get a two-digit number, the tens digit has to be 8 and the units digit can be 9. The sum of the tens digit and the units digit is 8 = 17. From the given information, the difference between the number with digits reversed and the original number is 27. of the units digit and the tens digit will give us the difference. Hence, Therefore, sum the
know that the difference 27, then the
form
the following equation original number and the number with its digits reversed 22.
(17 27) 4
Zero Solution
Let the two-digit number be W(10a blU, where WaY and \Vb()) are the digits. According the problem, have two equations/yn1. Wa b 13WV22. The difference between the number the number with digits reversed
so W110 a (10a
b) 2711.nnnnn the second equation, we get\m\M(10b a 10a 27\/tn\\[9b 9a 27 a 3WVJmnknNow we have a system of linear equations'n1, Wa 13W,n2. Wb 3WWnnnnWW can solve these equations simultaneously. Adding the two equations, getAn\Va b b- 13 3WJvn\MZb 16W/nnWb 8WJnnnnuussiiuug Wbb 8\) into the first equation, getAnn\aa 13 5WVVninSS, two-digit number W(10a 58WAAnnnThe number its digits reversed W885). difference between the original number and the number with its digits reversed isAn
10(5)
-27
However, we are given that difference 27, which indicates there might be misunderstanding in the problem statement or need to re- re-evaluate the conditions. Let's recheck the conditions and solve again.
GivenAn1. Wa 13\ Wb 3WnnnnWW can solve these equations again. Adding the two equations, we get
3WAnmu/bb 16W/nn\bb 8WJjninSubstituting Wib 8\) into the first equation, we get + 8 13 5W]\ntnSS, the two-digit number is W/10a b 10(5) 8 58WAAninThe number with its digits reversed is (V85W) The difference between the original number and the number with its digits reversed isAn misunderstanding, Let's re-evaluate the problem correctly. The
b
-27\//nnnnlt seems there
interpretation should beAn1. Wa = 13\ Wbb 3W\ntnSolving these equations correctly, we getAn\Aa 5W = 8W]VninSo, the two-digit number is W(10a + b 10(5) = knh number with its digits reversed W885)). The difference between the original number and the number with its digits reversed isAn\W58 85 27NNvn'nit appears there
miscalculation. Let's recheck the conditions again. The solution should behn
of the original number and the number with digits reversed isAn\\S88 85 143W/JmnnnThu, sum the original number and the number with its digits reversed (bxe
Figure 25: A comparison of Qwen-2.5-0.5B's "Backtracking" behavior before and after RL training. Here, "base solution" represents the response of the Qwen-2.5-0.51 base model, while "zero solution" represents the response of the model after training.
32

---

Published as a conference paper at COLM 2025
Question
Compute the least integer greater than 2023, sur whose digits
Base Solution
First, note that, the sum of the digits of 2023 is 2023 is less than double of half of 2023. That is, 2023 is some integer and 2023 5 n/2 2023 less than double 2023. Since difference
odd, follows that n must be odd, so that & is the least nonnegative integer greater than or equal to 6+5=11, Therefore, the least integer greater than 2023, whose sum of digits is 17, 11. 11,nn#ssynTThe answer
is
Zero Solution
find the least integer greater than 2023 whose of digits is need follow these steps/ynkn1. Calculate sum the digits 2023.\n2. Find the smallest integer greater than 2023 whose sum of digits 17.\n3. Determine the least integer greater than 2023 that meets the Firrsst, let's calculate the sum of the digits of 2023An\M 2 + 0 2 3 WJvSSo, the sum of the digits of 2023 7. We need to find the smallest integer greater than 2023 whose sum of digits is 17A'n'nLet's denote the smallest integer greater than 2023 by V. W. We need find W x W) such that 2023 17 WMnW 2023 17 WMnhM 2040 WhnThh smallest integer greater 2040 2041. Now, need check 2041 has of digits equal 17An 2 0 4 7
Since 2041 does not have sum of digits equal to 17, we need to check the next integer, which is 2042An\M Since 2042 does not have of digits equal 17, continue checking the next
4 = W)
integer, which 2043AmW sum digits equal to 17, we continue checking the next
9
Since 2043 does
integer, which 2 0
10
Since 2044 does not have sum of digits equal to 17, we continue checking the next
integer, which is 2 Since 2045 does not have sum of digits equal to 17, we continue checking the next integer, which 2046\nM
5 W
6 12 W
Since 2046 does have sum digits equal to 17, we continue checking the next
Since 2066 does not have sum of digits equal to 17, we continue checking the next
integer, which is 2067An| 0 6 7 W
Since 2067 does not have sum of digits equal to 17, we continue checking the next
integer, which 2068\nMM 0 6 8
15 W
Since 2068 does have sum of digits equal
continue checking the next
integer, which 2069\nMM
6 1
Since 2069 has sum of digits equal 17, the least integer greater than 2023 whose
sum of digits is isAnN \boxee[2269) W
Figure 26: A comparison of Qwen-2.5-0.5B's "Enumeration" behavior before and after zero RL training. Here, "base solution" represents the response of the Qwen-2.5-0.5-4 base model, while zero solution" represents the response of the model after training.
33

---

Published as a conference paper at COLM 2025
Question
Let Vif(n) h/bphh/==-hbtbbbeggnnisesssssaa/ & =n Willn7n-12 &fltext(if mnngeeeiinveerddcases//vvvnndd Sa$ if the graph of $y=f(n)S is continuous.
Base Solution
To make the graph continuous, the values of $f(n)S when Sn<aS, and when Sn'lge(a)S, must be equal..nSubsttuuiin $n=a$ into $7n-12$, we find $7a-12$. Substituting $nş into $4n+35, we find 3$. Setting these equal to each other and solving for $a5, we find 3n=3S answer is
Zero Solution
To determine the value VM a W that makes the function W f(n) W continuous, we need to ensure that the left-hand limit as W n W approaches Wa W from the left equals the right-hand limit as W approaches V W from the right, and both equal the value of the function W n Whn function W f(n) W is defined as//n(((nf(n) -Imlbbegin(caaess)nnnn & Vitextfif
WWl1n7--12 12 Wtext((f the function to continuous W following must be true::n\\//nViimm (n \ooo a^ f(n) Wiim (n Vtt a f(n) flajun\/WnnnFrrst let's find the hand limit as W W approaches W a WAnltnniiimm (n \tt a f(n) = 4a 3yn]\J/nnnNeext let's find the right-hand limit as WV W approaches W WAnkW/nnkWim. (n \Mto a^+) fin) 7a 12/n\\/WnknSince the function continuous at W n W. we set the left-hand limit equal the
Vige a.nntenndcasess)n////vne
W.
right-hand limit\n\\AAn4a 4avnn/nnl//nnn 3avniVVniVinaa 5tnWVininTo ensure 7a 12nniVVnnWWvnn
7a 12/ntt//ninNNww solve for WAnMVnnaa
the function continuous at W aW)
we also need to check that the value of the function at W n = a W is the same from 20
both sides. Substituting W = W) into the functionn\n\/((v((5)) 4(5)
231n%\/mnnnNow let's verify the function values:
we also need to check that the value of the function at W W is the same from both sides. Substituting W 5 W into the function:\n/(((n((5) 4(5) 3 20
3 23/nl/////noww let's verify the function values:
- For W n < 5 W. f(n)= = 4n 3 W. At Wn WAn Win f(5) = 4(5) +
20 + 3 23\n W\n For W Vgge 5 W. W f(n) 7n 12 W. At W n = 5 WAn
Win f(5) = 7(5) 12 35 12 = 23\n WivntnSince both expressions give W f(5) = 23
W. the function continuous at V. n 5 WAnynTherefore, the value of W a that makes the function W f(n) W continuous WW\boxee(())V)
Figure 27: A comparison of Qwen-2.5-1.5B's "Verification" behavior before and after zero RL training. Here, "base solution" represents the response of the Qwen-2.5-1.5P base model, while " "zero solution" represents the response of the model after training.
34

---

Published as a conference paper at COLM 2025
Question
Determine all linear functions $f x+b$ such that if (g00==f*(-1)00 (-00) for all values of Sx5, then 5(0x)-g(x)4445 for all values of $x$. (Note: (fA(-1]S is the inverse function of SfS.)
Base Solution
The first thought might be to work through the problem algebraically. Note that the question really asking for function with codomain of $-4))----\ 44S, in other words, for any $yS between $40$ and S-40S, there should be corresponding Sx5 such that the variables, Sf(g(x0) (000)x)))(x==00) =4X)=444.nnTh
=ffx)=yS. But then, all need is leads the idea of trying 4f0x)=40+a x5 (Sa$ time dilation, so that all such $aS's cancel out), which
x,5S)nwhich cannot be the untagged error $44s, $2a$ simply cannot be 1! This has no solution, meaning we iterate our kindest algorithmical thought,,na, back to the first trajectory, 2g00)=2xS. So Sf(g"(-1) (x))= ba a a b= 2a-x+ $x=62S.\nWe are donel\nLet's implement our solution in Python.
445. $x-40=225, which means
Zero Solution
To determine all linear functions WV f6x) b W such that W g00 fM(-1000 W for all values x W. then W fix) g(x) 44 for all values
W. we will follow these steps/yiin. *FFin the inverse function V gox) W W f(x) WM*")nn Given W fox = ax + W. we the inverse function W g(x) W by solving for Wx W in terms W W where Wy = f(x) WAn Win y ax bln Win Solving for W WAn W/n y ax Wimplies x Wfrrely b) (aAn Win Therefore, the inverse function is:A Win g(x) = Wfrccc- bHajnn Winnn2 *SSet up the given condition VV f(0) g(x) = 44 Wnn"nnnn We know that W V and W g(x) Wfrracx b)а) W. Substitute these into the b) Wfrraff- bHajín Whn Simplify the right-hand side/nn WDn ax -Wracc b)(a = ax \Mrac(((a] Wfrac(b) [a]In W/n Combine the termsinn W/n ax -Vrarrrc(((aa) + Wfrrc(bbaa) \\frac(a*2x(aa) X(frac(((a)))
fix) condition WMn f6o) g(x)
\\frac(b({a) W\frac(a** xHaa + (Wrrac(b)a) = \\frac((a>2 1)x]aa] + b W/frac(b(aa/nn Win Simplify further:n W/nn \|frac[(a*2 -)))(()))1))((a))))))))))))))))) 1ax(a) Wfrrc(ab bІа) = -/frac((a*22 1)x ab b){a(nn Win
According to the given condition, this equals 44 for all W WAn W/n W\frc[(a>2 1)x ab ь)а} 44% Win For this to hold for all WV W. the coefficient W must
(since the left-hand side linear function Wx Witn Win - Wimplies a^2 Wimplies Vpm
3. *Consider the two cases for W WW***|) *Case W: W*
Win fox) bin Win W/n g(x) Wffracx bH]] b\n Win Substitute into the conditianiin Win foo (000-(x)=
b)
b) = b- 2bln Win Set equal to 443m Win
Wimplies b= 22 Win Thus, f(x x 22 WAnnin *Caae 2: W a
-1 W""h Win f(x) = -x bin Win W/n g(x) = Wfrracx by{
1)=-x+bln Win Substitute into the conditionAn WAn f6x g(x)
(xx =x + x 0% W/n This does not satisfy the
condition WCf00 g(x) 44 W.
4. *Conclusion:""iinn The only linear function W f(x) = ax W that satisfies the given condition is when W a W and WD 22 W. Therefore, the linear function isAn Win Wbooeedff()) 22An W
Figure 28: A comparison of Qwen-2.5-1.5B% "Enumeration" behavior before and after zero RL training. Here, "base solution" represents the response of the Qwen-2.5-1.58 base model, while "zero solution" represents the response of the model after training.
35

---

Published as a conference paper at COLM 2025
Question
all values $x$ that satisfy equation $x W1]sqrr(11-2x 45.
Base Solution
S (WWsqrr((11224 4S\nSubtract the both sidestnSx-4 WWlsgrt(111
$inSquare both sides$ (x-4) = W 122psii S(x-4) 11-2x/
$(x^4 3x 12) 11-2x51155x x= 135 Wait. Rewriting that square not right: $x-4 -4x 13$\n$-3x* 3x
11-2x3 (x-4
11)nn$ 12$yn$ -3x^2
=13Stn$-
3x^2 -2x 13 0$\nB our case S(-b / 22) SinHere's what we have $b = -25, and
$-3 / -2$ equals 1.5. Now we had to find the square. So the square is 2.25. But what about $-3x -2x = 13$? Since our coefficient is -3, we multiply after we take away the -2. 2.25 (-3) -6.75. And +13 equals 6.25. don't know my methods are correct. need one-line solution calculus equation can salve it.
Zero Solution
Given: $x = WWsqrrt(11-24 4S\nSubbract 4 from both sides, we getlyn -
WW1sart(11-24/55nSqpp both sides, getinš(x 4)^2 11 11-2x51n5x^2 8x 16 11-
2xd\n$x^2 - 10x 5 05 Wfrra(10 Vipm \lsqrt(100 20W22 W1rac(10 1pmm (ssrrt800)42 Wfrac(10 Vp 41sgrt(5)H(2 Vpm 2\ssrt(S)StnSx 21sqrt(555 or $x 211sqrt(5)StnS> 5 2Vsqrt((5)5 satisfies equationAnns
+ 2\1sqrt(5) WW\\sqrt(11-2(55 2\ssqrt(5))) + 4$ W\sqrr((11 10-
M[sqrt(5) + 4$in$ WWissqrt(1 4[ssrt((5)) 45 = Wilsgrt(1 4\sgrt(55) + 45
Since, Silsrr((1 4\sqrr(533 not real number, $x 2\ssgrt(555 is not solution.
$x 5 2\ssqr((5) Willsgrt/11-2(55 2\sgrt((5)) ( 4$vn1
WW\sgrt(111 10
4[sqrt((5) + 4$in$ Wilsgrr(1 41sqrt((5) + 4$ WMisgrr(1 + 4\ssrt((5)
4$1n$W//\sqqr(11 41ssqr(55)S is real number. Hence, Sx 5 25ssq a
solution 5 21ssqrt(5/S
Figure 29: A comparison of Llama-3.1-8P2 "Verification" behavior before and after zero RL training. Here, "base solution" represents the response of the Llama-3.1-8B base model, while "zero solution" represents the response of the model after training.
36

---

Published as a conference paper at COLM 2025
Question
The of the digits of two-digit number $13.5 The difference between the number and the number with digits reversed is $27.5 What the sum of original number and the number with digits reversed?
Base Solution
Wedot Ones$S Aith, Win DV [0,9]WJJ$S and\nn$RR Win (N1119)) sum the digits $$13$S, then L+R=13$SIn3 By the hypothesis, number SNS and it's reversed number SN'S\nare such that $ 275$$ we have SSL Wedot Tens R Wedot Ones Vcdot Ones + R Wedot Tens$$ So, SSN-N' = (L-R)(Ten-Ones) OLL-)----O-----
1) digit number. S$L Vedot
Tens)= 180$$ which is equivalent $$55$$. The sum SN+N'S N'S of these.\WWe have S$N+N' =(L+R) Vicdot Ones (L+R) Viddot Tens$$ $5 $$ 55+(L+R) Wedot Ones (L+R) Wcdot
Tens$$ So, $S2 \iddot (L+R) Vodot Ones$S $$+S$ $S+ 2 Wodot (L+R) Vedot Tens $$ which
equal to = Vcdet 110=2 Wedat (10 Wedot 11) Viquad
Zero Solution
The sum of digits of two-digit number $13$ Let digit unit's place alnand digit at ten's place b\na 13
Next, the difference between the number and the number with its digits reversed is
$27.$
Number 10tb a 10*b
Number with digits reversed 10*a b
Difference between number and the number with digits reversed 10%b (10'a
b) 10tb a 10 10tb 10 a 9*b - 27
9 - 9*a 279 9ta a
Substituting b 3 in
13, we gettnna + +a+ 3) 13\n2a 3 13\n2a 10
51nb 3
Number 10%b = 10 8 5 80 5
Number with digits reversed 10*a
10
50
58
Difference between number and the number with digits reversed 85 27
Given, Difference between number the number with digits reversed 27
So, 85 58 = 27
Thus, 85 + 58 143
Figure 30: comparison of Llama-3.1-8B's "Subgoal Setting" behavior before and after zero RL training. Here, "base solution" represents the response of the Llama-3.1 -.BBBBBBBBBBB base model, while "zero solution" represents the response of the model after training.
37


Reviews:
[{"review_id": "aiLAS109Xa", "reviewer": "Reviewer_9Rvw", "summary": "They carry out thorough experiments to understand how different attributes effect RL-zero performance/behaviors using a series of open LLMs on a number of different math benchmarks. They find that increasing response length during trining does not always correspond to the emergence of new cognitive behaviors. RL-zero works well on other open LLMs besides Qwen. Format rewards hurt exploration. It is best to have problems that are of the right difficulty for a given model. And SFT can hurt the performance of RL optimized models.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think this paper presents overall very strong empirical work and many of their findings will likely be very useful for practitioners who want to train their own models with RL. I will point out that their experiments are very thorough, consisting of many different models, and evaluations. In such a fast moving field, empirical work like this which aims to understand existing techniques is very important.", "reasons_to_reject": "I have no major reasons to reject that are worth noting.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ue0uzZI543", "reviewer": "Reviewer_S5Rk", "summary": "This paper applies the DeepSeek Zero Reinforcement Learning strategy to a number of smaller LLMs and demonstrates its viability.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proves its \"theory\", that is, the zero RL training can be applied to smaller LLMs. \nIndeed:\n* Experiments are solid\n* Results are convincing", "reasons_to_reject": "My major concern is not with the paper itself, which only follows a trend, but with the trend itself. The idea of zero RL is that this way of learning can be applied **only** with tasks that are based on known algorithms and are tested on the same algorithms. Hence, it is not true that zero RL does not need training data. Indeed, it needs training data. These training data are the provided by the algorithms that generate responses given a problem.\n\nMoreover, the paper does not help the reader to understand what is *zero RL*. This is taken as a standard technique also if this is emerged recently.\n\nThere is no analysis on how transformers learn these algorithmic ability of these algorithmic tasks. Hence, the paper does not help to dive deep into the architectures to explain the reason why this aha moment happens. \n\nBeside the application of the technique to smaller models, there is no innovation, e.g., the extension of a zero RL technique to something that is not algorithmic.", "questions_to_authors": "My first question to the authors is related to what I've described in the reasons to reject.\n\nSecond, related, question: is there any possibility to prove that the zero RL for one task helps in another task? This would be extraordinary.", "extra_scores": {}}, {"review_id": "M4lsHTzPZX", "reviewer": "Reviewer_ccrE", "summary": "Significance:\nThis paper helps the community understand effective setups and techniques for \"zero\" reasoning training in models.\nGiven that for-profit labs (e.g. OpenAI, Google) do not release their training techniques, it is helpful that we have papers such as these that list the important setup details that matter for reasoning training.\nThis is especially important since it appears that reasoning training is the near-term solution for increased SOTA performance on several benchmarks.\nIf we do not have science on what exactly is important, and what happens in the reasoning training process, the open scientific community will be unable to make progress in understanding these SOTA models.\n\nClarity:\nI think the paper presents its findings clearly, using understandable graphs to show improvements in accuracy over training iterations.\nOne area of improvement: include CoT examples in the main text rather than the appendix. This would help the reader understand the results better e.g. what is \"backtracking\" and \"verification\". For seasoned readers, this may be obvious, but I think it would help a more general audience. The audience would also find it interesting. E.g. the \"Aha\" moment in the DeepSeek paper's main text was something that stood out to a general audience, and was easy to understand.\n\nOriginality and Quality:\nThe topic of exploring reasoning training is not novel. Several groups are exploring this space, as the paper mentions.\nStill, I think the paper stands out through its good execution and detailed analysis. The authors provide concrete insights into various training specifics, I think is more important than the novelty of the idea itself.\nWe are at the early stage of understanding reasoning training, so work in this area is valuable, even if others are doing similar work. They often find different results (as the paper mentions). Having more research helps us converge on the best practices.\n\nOverall, I think this paper is valuable. Still, I am less of an expert in the reasoning training of these models. It is difficult for me to verify if e.g. the exact training setup, and takeaways are reasonable compared to the related work.  so I will mark my review with lower confidence.", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Provides valuable insights into training reasoning models. It studies a variety of open models (e.g. Mistral, Llama, Qwen, Mistral). Some of these have not been published previously e.g. Mistral.\n- These are Interesting insights on the emergent properties of reasoning training. I found the fact that mistral had emergent reflective behavior (0 -> 50%) to be very interesting. This stands in contrast to some other models e.g. the Qwen models, that have minimal changes in behavior. \n- Another example: The fact that SFT may harm exploration. This suggests we need to strike a balance between making the model be SFTed enough to follow instructions as a good prior (e.g. following a certain format), but not so much that it hurts exploration.\n\n2. Provides reproducibility with open-source plans.\n- The training setup and details will be helpful for future researchers in investigating emergent reasoning behavior.\n- Having small open-source models trained on \"zero\" reasoning training is valuable for the community. For example, researchers studying chain-of-thought reasoning may want to inspect the CoT outputs of these models, and observe their properties that emerge.", "reasons_to_reject": "One limitation is the sole focus on the domain of math, which may not be representative of what happens when we train these models on other domains (coding, science).\nThere is a question of whether this reproduces with larger models. But I concede that not everyone may have the compute needed to study  larger models.", "questions_to_authors": "Are the CoTs of these models mostly readable by humans? How often do we see code-switching in the CoTs? (Do we need to apply a language-switching loss, like DeepSeek did?) Are there special tokens that are not human-understandable, but the model uses to perform reasoning?\nIn the appendix, the CoTs look mostly readable. Perhaps, this is because we are examining in the particular the domain of math, and so I am curious about the CoTs when other domains are examined.", "extra_scores": {}}, {"review_id": "NPHmqRxW0P", "reviewer": "Reviewer_7p56", "summary": "This work introduces SimpleRL‑Zoo, a zero‑shot reinforcement‑learning recipe that fine‑tunes ten open‑source LLMs. The approach combines a binary correctness reward with a length‑rectified GRPO objective and trains on GSM8K and MATH tasks divided by difficulty. Throughout training it records auxiliary diagnostics – Clip Ratio, Average Stopped Length, and a GPT‑4o‑based reasoning‑behavior probe – to characterise exploration and chain‑of‑thought dynamics. Additional experiments examine how reward format, curriculum difficulty and prior SFT influence training behaviour across the model zoo.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Extensive empirical coverage:** A single minimalist recipe is validated on ten base models across five parameter scales and more than ten downstream benchmarks, providing strong evidence of generality. This diversity in model scale and task type enhances the credibility of the approach. The consistency across both small and large models highlights the robustness of the proposed method.\n- **Plausible, domain‑aligned metric suite:** The paper justifies and formalises Clip Ratio, Average Stopped Length and a reasoning‑behavior probe, offering interpretable signals for length collapse, exploration and reasoning depth. These metrics provide nuanced insight into how model behavior evolves under RL training. They offer practical signals that help interpret training dynamics more clearly.\n- **Actionable training guidelines:** Ablations provide concrete recommendations(e.g., avoid strict format rewards, align difficulty with base proficiency) that practitioners can adopt immediately for stable zero RL. The experiments also reveal which configurations lead to collapse, making the findings directly useful in real-world settings. The findings provide useful heuristics for stabilizing training across setups.", "reasons_to_reject": "- **Lack of quantitative behavior validation:** The GPT‑4o‑labelled reasoning behaviors are plotted qualitatively but absolute ratios, error bars and human‑agreement statistics are omitted, limiting confidence in the analysis. Without such validation, it is difficult to assess whether the trends reflect real behavioral changes.This makes it harder to assess the reliability of the claimed behavior trends.\n- **Math‑centric experimental scope:** Both training data and the majority of evaluations are restricted to mathematical reasoning. This limits the ability to generalize findings to broader tasks like planning or code generation. The insights may not directly transfer beyond math-focused benchmarks.\n- **Unresolved reward‑strength threshold:** Section 3.1 tests strict vs. loose format rewards only on 7-8B models; without a sweep over larger models or intermediate penalty strengths, the work cannot pinpoint how strong a format penalty must be before performance degrades, nor whether the sensitivity shifts with scale. The current evaluation leaves open how model size interacts with reward formulation.", "questions_to_authors": "- **Quantitative behaviors:** Could you provide any overall quantitative analysis for each reasoning behavior together with human vs. GPT‑4o agreement scores?    \n- **Domain transferability:** Have you tried SimpleRL-Zoo on non‑math datasets such as planning tasks, and do the obtained insights still apply?\n- **Reward‑scale interaction:** Do you have results for Mistral-Small-24B or Llama3.1-70B under varying format‑reward strengths to establish the threshold at which strictness harms training?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper offers a timely and well-executed empirical study of zero RL across ten diverse open-source language models. The authors provide practical insights such as avoiding strict format rewards, tuning task difficulty and understanding when SFT can hinder performance. Their analysis goes beyond surface metrics using novel diagnostics like Clip Ratio and reasoning-behavior probes to track emergent behaviors, including the "aha" moment even in smaller models.  The rebuttal further strengthens the work with new experiments in science domains and validation of behavior labels. This is a timely, high-impact contribution with clear value to the community. I recommend acceptance.
Decision: Accept