Understanding R1-Zero-Like Training: A Critical Perspective
Authors: Zichen Liu
Changyu Chen
Wenjun Li
Penghui Qi
Tianyu Pang
Chao Du
Wee Sun Lee
Min Lin
Venue: COLM 2025 (2025)
Keywords: R1-Zero
Reinforcement Learning
Post-Training
TL;DR: We fix the optimization bias of GRPO, and propose a new algorithm to achieve state-of-the-art results using pure reinforcement learning.

Abstract: DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, to understand how pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a new state-of-the-art.

Published as a conference paper at COLM 2025
Understanding R1-Zero-Like Training: A Critical Perspective
Zichen Liu +12,, Changyu Chen 1, Wenjun Li Penghui Qi"! Tianyu Pang', Chao Du', Wee Sun Lee?, Min Lin
Sea Al
National University of Singapore SSingapore Management University
Abstract
DeepSeek-R1-Zon has shown that reinforcement learning (RL) at scale directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we critically examine R1-Zero-like training by an- alyzing its two core components: base models and RL. We investigate a wide range of base models, including DeepSeek-V3-Base, pretraining characteristics influence RL performance. Our analysis reveals that DeepSeek-V3s SesSeek----Base already exhibit " Aha moment", while Qwen2.5 base models demonstrate strong reasoning capabilities even without prompt templates, suggesting potential pretraining biases. Additionally, we identify an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incor- rect outputs) during training. To address this, we introduce Dr. GRPO, an unbiased optimization method that improves token efficiency while maintaining reasoning performance. Leveraging these insights, we present a minimalist R1-Zero recipe that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing new state-of-the-art.
understand how
o https://githut: hww.:tthub..oo.eeeeeeeeeeeeeteet wwt.:/...cclllooll.co.......l..a http://llsggguueesssaddd-------o
GRPO
Token Efficiency
24 loy1
1 metoulq.o.c) Age clip Taw (0:19.9 w..)
Re. 1. Ro
0.6
0.5
where du R(q.o.)
meani ( (Riq. b........ R(g.oc)1)
RL training progress
std (R(g.:)))...........)))". R(g.0o)1)
Reward
0.4
0.3
Dr. GRPO GRPO Done Right (without bias)
0.2
Iia Xal m,e dup ne,, o,ee)
xalouug er) lo oaa
41 d.d
0.1
GRPO
where d.g R(q.o:) meani (R(g. M(q,o1))).
400
600
800
1000
Output length
Figure 1: Left: Dr. GRPO introduces simple yet significant modifications to address the biases GRPO (Shao al., 2024), by removing the length and std normalization terms. Right: Our unbiased optimizer effectively prevents the model from generating progressively longer incorrect responses, thereby enhancing token efficiency.
1 Introduction
DeepSeek-R1-Zero (Guo et al., 2025) revolutionizes the pipeline of large language model (LLM) post-training by introducing the R1-Zero-like training paradigm: directly applying RL to base LLMs without relying supervised fine-tuning (SFT) as preliminary step.
This new paradigm is appealing due to its simplicity and the demonstrated RL scaling phenomenon: the model reasoning capabilities improve along with continual increase in
*Core Contributors.
*Project Lead.
'Deellpeed with the L.M RI. framework Oat: https://githubs http:////gttuu..ccosal/lsessseee wttp:/cccmmaalll//aeiie

---

Published as a conference paper at COLM 2025
Qwen225-Math77
13.8
80
SimpleRL
28.00
70
Prime-Zero-78
0pen
Accuracy (%)
Oat-Zero-78
50-
40
43.3
36.5
40.3 16.0
41.0
382
30
21-7
20
13.3
52
OlympiadBerm
MATH500
Avgerage
Figure 2: Model performance comparison. Oat-Zero-7B is RL-tuned with minimalist recipe described in Sec. 1 (third paragraph). Please see App. for more results.
model's response length. This phenomenon also accompanied by the "Aha moment", at which the model learns emergent skills such self-reflections.
In this paper, we aim to understand R1-Zero-like training by studying two essential com- ponents: base models and RL. In the first part, we investigate various attributes of base models, with the focus on the Qwen2.5 model family (Yang et al., 2024a;b), which has been used recent attempts
reproduce R1-Zero (Pan al., 2025; Zeng et al., 2025; Liu al., V3-Base (Liu et al., 2024), from which the real
2025b; Hu et al., 2025), well as R1-Zero model was RL-tuned. In the second part, we identify the bias in optimization of GRPO (Shao et al., 2024), which may lead to progressively longer incorrect responses. To this end, we propose simple modification to eliminate the bias, i.e., to get GRPO Done Right (Dr. GRPO), which leads to better token efficiency (highlighted in Fig. 1).
Our analysis on base models and RL suggests a minimalist recipe for R1-Zero-like training: we RL-tune Qwen2.5-Math
using the (unbiased) Dr. GRPO algorithm MATH (Hendrycks et al., 2021) level 3-5 questions with the Qwen-Math template, and achieve state-of-the-art performance (Fig. 2) with only 27 hours compute 8x A100 GPUs. We hope our findings presented in this paper, models released, and the codebase open- sourced could benefit future research in the field. As overview, summarize the takeaways of this paper below:
Overview of takeaways
(Sec. 2.1) Template is crucial to make base models answer questions instead of completing sentences, In addition, all base models already possess math-solving capability prior to RL.
(Sec. 2.2) Intriguingly, Qwen-2.5 base models get an immediate ~ 60% improve- ment by not using template, making us hypothesize that they may pretrain on concatenated question-answer texts when cooking the models.
(Sec. 2.3) Nearly base models already exhibit the " "Aha moment", including DeepSeek-V3-BB
(Sec. 3.2) Dr. GRPO effectively fixes GRPO's bias in optimization, achieving better token efficiency.
(Sec. 3.3) Model-template mismatch can destroy reasoning capabilities before RL reconstructs it.
(Sec. 3.4) Math pretraining on Llama-3.2-3B improves its RL. ceiling.
2 Analysis on Base Models
In this section, we scrutinize a wide range of base models, including the Qwen-2.5 fam- ily (Yang et al., 2024a;b), Llama-3.1 (Grattafiori et al., 2024) and DeepSeek series (Liu et al., 2024; Shao et al., 2024; Guo et al., 2025), asking them 500 questions sampled from the MATH (Hendrycks et al., 2021) training set and analyzing their responses.
2

---

Published as a conference paper at COLM 2025
Question-Answerrig Ability
Exploration Ability
Self- Reff-Reflectin
1.0
Count of Questions
50
Answering Rate
0.8
0.8
40
P
0.4
20
0.2
10
Owen2 w.t.tt-----.s.
Owen2 5-Math-78
0.0
Qwen2 5-88
Llama-33 3.1--3
DeepSeck-Mathh22
DeapSeek-V3 Base 685B
0.0
0.10.0..0..40.5000
Temperature
0.5.0.6... 0.80.91.0
0.0
Temperature
Qwen2 5 Math-1158
Liamaa-311 3.1-88
template w/ template
QwenZ.5-Maaie M-tth-5
MatpSeee--Math-8
Owen2.5-78
DeepSeek-V3-Boss 6858
Figure 3: Model attributes across three aspects. Question-Answering Ability: the extent to which pretrained language model provides direct answer to question rather than continuing or expanding upon it; Exploration Ability: pass@8 measures how well base models explore; Self-Reflection: counts obtained through cross-validation between keyword-based detection and LLM-based detection, as detailed in Appendix D.
2.1 R1-Zero Trainability: Templates Construct Exploratory Base Policies
Since training from base model is a fundamental setting of the R1-Zero-like paradigm, first investigate whether widely used open-source base models, which are typically trained for sentence completion (i.e., Po (x)),, tively elicited through appropriate templates, thereby functioning as question-answering base policy Tt(-lq). In addition to the R1 template (Template 1) in Guo et al. (2025), we consider the Qwen-Math template (Template 2) used by Zeng et al. (2025), as well as No template (Template 3):
have their quuestio--nnseringg capabilities effec-
Template 1 (R1 template). A conversation between User and Assistant. The User asks a question, and the Assistant solves it. The Assistant first thinks about the reasoning process in the mind and then provides the User with the answer. The reasoning process is enclosed within <think> </think> and answer is enclosed within <answer>> < /aaswer>> tags, respectively, i.e., <think> reasoning process here </think> <answer>> answer here </answer>, An (question) InAsssstat:: <think>
Template 2 (Qwen-Math template). < it_start] >system (PPlease reason step by step, and put your final answer within \\ (bxxd lim_end >>n<< start >usser' n question] <|im-end >>
assisstant
Template 3 (No template). {question)
Experimental settings. We include Qwwn2.5-Matth-1555 Qwen2.5-Math-7B. Qwen2.5-7B, Llama-3.1-8B, DeepSeek-Math-7B and DeepSeek-V3-Basee
for experiments. For each model, we first apply No template to get the model responses, then let GPT-4o-mini to judge whether the model responses a sentence-completion pattern. We record the percentage of responses that tend to answer the question the metric. We then apply both R1 template and Qwen-Math template to obtain model responses, and determine the most suitable template for each model based on the metric. Finally, we evaluate the pass@8 accuracy of each model with the corresponding template to assess whether the base policies can explore rewarding trajectories for RL improvement.
an answering format (regardless of quality) or in
Results. The left plot of Fig. 3 shows how well base models (with or without templates) answer the provided questions. We observe that Llama and DeepSeek models all improve the answering ability by employing the proper template (R1 template). However, Qwen2.5 models work best (with 100% answering rate) when no template is used. This intriguing property motivates further investigation which will be discussed in Sec. 2.2. Meanwhile, the lowest answering rate with no template suggests that DeepSeek-V3- V3-Base nearly pure base model. This observation motivates to explore whether a pure base model
3

---

Published as conference paper at COLM 2025
Base model + Template
AIME24
AMC
MATH500
Minerva
OlympiadBench
Avg.
Qwen2.5-Math-11.
(4-shot prompting)
0.0
20.0
50.4
12.1
15.9
19.7
Rl template
0.0
9.6
21.2
6.6
2.2
7.9
Qwen template
20.0
33.0
12.5
22.8
24.2
No template
16.7
43.4
61.8
15.1
28.4
33.1
Dw
(4-shot prompting)
3.3
22.5
61.6
10.7
20.9
23.8
R1 template
0.0
0.0
0.0
0.0
0.1
0.0
Qwen template
16.7
38.6
50.6
9.9
16.6
26.5
No template
0.2
45.8
69.0
21.3
34.7
38.2
Table 1: Qwen2.5-Math models might be pretrained on concatenated question-answer text, resulting in peak performance when no template is applied.
like DeepSeek-Va2 -atteeee----BBeee demonstrates the Aha moment (Sec. 2.3). The middle plot of Fig. shows the pass@8 accuracy of different base models (with template) at different sampling temperatures. This metric can serve as an indicator of base policy's exploration ability. For example, if base policy cannot even sample single trajectory that leads to the correct final answer, it is impossible for RL to improve the policy because there no reward signal. Our results demonstrate that all tested models are exploratory (thus ready for RL), with Qwen2.5 models performing the best (even surpassing DeekSeek-V3-Basse partially explain that most Ri-Zero projects (Zeng et al., 2025; Hu et al., 2025) are based Qwen2.5 models.
This might
2.2 Qwen-2.5 Models Unlock the Best Performance When Discarding Template
We next dig into the intriguing observation (c.f. Fig. 3(Left)) that all Qwen2.5 base models readily serve as chat models even without any template. We take a step further to eval- uate the reasoning ability of Qwen2.5-Math models on five standard benchmarks: AIME 2024 (Li et al., 2024a), AMC (Li et al., 2024a), MATH500 (Hendrycks et al., 2021), Minerva Math (Lewkowycz al., 2022), and OlympiadBench (He et al., 2024). Following common practice, we use greedy decoding and limit the sampling budget to 3000 tokens.
As shown in Table 1, not using any template resulting in an improvement of about 60% compared to the traditional 4-shot prompting. Since Owen2.5-Math (Yang et al., 2024b) uses chat model's data (question-answv) during the pretraining stage, we hypothesize that they might pretrain on the concatenated text maximize log Pa(q:o directly. If our hypothesis turns out true, we shall be more careful about using Qwen2.5 models to reproduce DeepSeck-Rl-Zero, since the base models are already SFT-like without templates.
drastically boost the average performance,
pairs)
2.3
Aha Moment Already Appears in Base Models Including DeepSeek- PeapSeek----Baann V3-Base
One of the most inspiring results of DeepSeek-RL. BateeeekkRRZZerrr is the emergence of self-reflection behaviors, a.k.a., Aha moment, through pure RL training. A few prior studies (Liu et al., 2025b; Yeo et al., 2025) have suggested that there may not be Aha moment in open-source R1 replications because the base models they use already exhibit self-reflection keywords. However, they have not tested DeepSeek-V3- V3--ase, on which the real R1-Zero model was RL-tuned. We complete this missing piece by hosting DeepSeek-Ve -3- ourselves and investigating its responses to the 500 MATH questions with the R1 template. From the right plot of Fig. 3, we can observe that DeepSeek-V3- -3-Baeee- also generates decent amount of self-reflections, further validating the claims of Liu et al. (2025b). We also show examples in App. E (Fig. 13) where DeepSeek- V3-Base generates keywords such as "Aha" and "wait"
An additional important question is whether self-reflection behaviors are associated with improved model performance after RL training. To investigate this, we host DeepSeek-RI- Zero and analyze its responses to the same questions from the MATH dataset. Although self-reflection behaviors occur more frequently in R1-Zero, we observe that these behaviors not positively correlated with higher accuracy. Detailed analysis can be found in App. F.
4

---

Published as a conference paper at COLM 2025
3 Analysis on Reinforcement Learning
Language model generation can be formulated as token-level Markov Decision Process (MDP) M pr)) At each generation step I, the state si E S is the concate- nation of the input question and the output response generated so far: 9; = 9: 9M. 01,... 01-1] The policy (gy()(st) will select the next token 0, from the vocabulary A, resulting in deterministic transition to the next state $1+1 = štj [o,]. The generation
process starts from sampling an initial state $1 q~ pΩ from set of questions, and stops when the autoregressive policy generates the [eos] token exhausts the budget.
Typically, we maximize the
objective (Schulman et al., 2017a):
[(Ta)
E q-Pe
E Lo~
- (Taaa
(1)
where R(q.o)= = E r(s;, 0+) is the return (Sutton & Barto, 2018) of the trajectory q;o, and zet is reference policy. The KL regularization term is usually adopted (B > 0) for reinforcement learning from human feedback (Christiano et al., 2017), where r a reward model learned from data collected by Te- In this case, regularization helps prevent Ti from deviating too far from the distribution where the reward model is accurate (Jaques et al., 2019; Stiennon et al., 2020). However, RL-tuning reasoning models typically employs rule- based verifiers as (Lambert et al., 2024), eliminating the concerns of distributional shift. This allows us to remove the KL term, which not only saves the memory and computation required by Tee during training, but also potentially leads to better performance for R1- Zero-like training (Hu et al., 2025). We will assume 6 throughout this paper.
Policy optimization algorithms. To optimize Ty with the above objective (Eq. (1) with / 0), Proximal Policy Optimization (PPO) (Schulman al., 2017b) maximizes the following surrogate objective:
Fxp) Eappo.o (144
la
Te(0)19,0<t) Te (0r 0zt))
A,, clip( ne(0ilg.0ct)
e, +,1+
(2)
min
where ag the policy before the update, is the clipping hyperparameter, and Å, is an estimator of the advantage function of the 1-th token. A standard way to estimate Å is to compute the Generalized Advantage Estimation (GAF) (Schulman et al., 2015) with learned value model Vi.. However, in the context of LLM RL-tuning, learning the value model is computationally expensive, so methods that estimate Å, without Vp are practically preferred. For example, Shao et al. (2024) proposed GRPO, which first samples group of responses (oI....,0c per question and computes their returns = (Ri...,, RG), then sets the advantage of all tokens from o; A
R, 30-mean(RR RR)
std R)
3.1 GRPO Leads to Biased Optimization
In Prepseek-R1ZZer (Guo et al., 2025), notable trend the consistent increase in response length throughout the training process. This is frequently interpreted as an indication of the development of advanced reasoning abilities such as self-reflection. Recent studies (Pan et al., 2025; Zeng et al., 2025; Hu et al., 2025) have replicated this phenomenon using various algorithms and implementations. However, we argue that the observed increase in response length may also be attributed to bias inherent in the GRPO (Shao et al., 2024) objective function:
(CR F
1
G
no To(0:1 9, 0,,ct) pep (oiz q, cc,
To(o,,19 0,,0)) 1 e, e,1+e) Art
(3)
min
Ag,, clip
C
No.. (0in |q, of
where
Ag R(q, 10l-mmean(aa (R o, (..,
std((R(q,o)),... (.) oc))
5

---

Published as a conference paper at COLM 2025
Correct
1/std(R)
Incorrect
Ai,//oo [o.]
03
03
04
04
GRPO's
Ana
advantage:
std(R)
[0,1
Figure 4: Illustration of the biases in GRPO. Note that the effective advantage of GRPO di is equivalent to reweighted version of the unbiased advantage AL, R(q.0;) mean (R). The terms (R) and joj could bias the optimization by assigning different weights to different questions and responses, denoted by the sizes of the blue circles and the lengths of the orange arrows. Upward arrows indicate positive advantages, and vice versa.
with the return (q, o;) typically only including the outcome verifiable reward in LLM reason- ing (the analysis also applies to process reward cases).
Compared to the objective function in Eq. (2), GRPO introduces two biases (see also Fig. 4):
Response-level length bias: This arises from dividing by |o For positive advantages (Ai, > O, indicating correct response), this bias results in greater gradient updates for shorter responses, leading the policy to favor brevity in correct answers. Conversely, for negative advantages (A,, < O, indicating an incorrect response), longer responses are penalized less due to their larger lo;1, causing the policy to prefer lengthier responses among incorrect ones.
Question-level difficulty bias: This is caused by dividing the centered outcome reward by std (R (1).))1).... 01)...,, ((q. oc))). Questions with lower standard deviations (e-g,, those that are too easy or too hard, with the outcome rewards being almost all 1 or 0) are given higher weights during policy updates. While advantage normalization is a common trick in RL (Andrychowic) et al., 2021), it is typically computed across an entire batch. In contrast, question-level normalization results in varying weights in the objective for different questions, leading to a difficulty bias in optimization.
Length Bias Also Exists Open-Source PPO Implementations. We also examined several popular open-source implementations of vanilla PPO algorithms for LLM post-training. To our surprise, all these implementations normalize the loss by response length (see listing and Table 2), which misaligns with the PPO objective as defined in Eq. (2). This formulation- implementation misalignment was present even before the publication of GRPO. We specu- late that the misalignment might originate from the pretraining stage (Shoeybi et al., 2019), where all tokens are packed into a fixed-length context and normalizing the loss by the context length (i.e., computing loss mean(-1)) improves the numerical stability. However, in the RL-tuning stage, typical implementations (von Werra et al., 2020) normalize the loss by the response length, which not a constant, introducing an unintended length bias.
Comparison between a typical open-source PPO loss implementation that is biased (red) and our implementation (green). MAX_TOKENS is a global constant during the entire training (unless budget curriculum is enabled), which specifies the maximum number of generation tokens. Other constants also work with differences in gradient norm.
Listing 1:
def masked_ mean (tensor, mask, dim):
return (tensor mask) sum (axis=dim) mask. sum ( (axis=dim) return (tensor mask). sum (axis=-1) TOKENS
ppo_loss =
# compute per-token ppo loss
response_mast
# per-token response mask per per-oeesonnee length normalization (e.g., OpenRLHF)
8 boss_variant1 masked_mean _epoo_pppnpppssss.. response_mask, dim=-1). mean ()
OR batch length normalization (e.g., trl, verl)
loss_ masked_mean loso response_mask, dim=None). mean
3.2 Dr. GRPO: Group Relative Policy Optimization Done Right
To avoid the aforementioned optimization bias in GRPO, we propose to simply remove the o and std({ R(q ) normalization terms. Meanwhile, to faithfully

---

Published as a conference paper at COLM 2025
Repository
Code Link
Unbiased?
(von Werra et al., 2020)
PPO Loss
X
OpenRLHF (Hu al., 2024)
PPO Loss
X
verl (Sheng al., 2024)
PPO Loss
X
SimpleRL-Zero (Zeng et al., 2025)
PPO Loss
X
Opee (Hu et al., 2025)
PPO Loss
X
Table 2: Many open-sourced PPO implementations contain length bias.
Output Length
Training dynamics
0.6
0.4
800
Dr. GRPO (Ours)
GRPO (Shao et al., 2024)
400
3. Output Length (Correct)
4. Output Length (Incorrect)
Average Benchmark Score (%)
Evaluation results
400
1.8k
20
300
1.4k-
200
1.0k
50
100
150
50
100
150
50
100
150
Policy iteration step
Policy iteration step
Policy iteration
Figure 5: Comparison of Dr. GRPO and GRPO in terms of training dynamics (Top) and evaluation results (Bottom).
implement the unbiased optimization objective, we could replace the mask. sum sum(axis=dim) with constant value (e.g., generation budget) in the masked mean function in listing 1, as highlighted by the line in green. Notably, these simple modifications recover the PPO objective in Eq. (2), with the advantage estimated by Monte Carlo return with unbiased baseline (Sutton & Barto, 2018). We give detailed derivations in App. A. We refer to our new optimization algorithm as Dr. GRPO. We next experimentally validate its effectiveness.
Experimental settings. We implement our algorithm using Oat (Liu et al., 2025a), modular, mnsearch-friendl] and efficient LLM RL framework. We adopt the Qwen2.5-1.5B base model and the R1 template (Template 1) for online RL-tuning. We implement the verification-based reward function using Math-Verify?, with the following minimalistic rule:
[1
if o contains the correct final answer to q otherwise
R(q, o)
We run RL on questions sampled from the MATH (Hendrycks et al., 2021) training dataset, and compare the vanilla GRPO with the proposed Dr. GRPO. We evaluate the online model on five benchmarks: AIME2024, AMC, MATH500, Minerva Math and OlynpiadBench. More experimental details including hyperparameters can be found in App.
Results. We report various metrics in Fig. 5 to demonstrate that Dr. GRPO can effectively mitigate the optimization bias and lead to better token efficiency. In particular, we first note that both GRPO and Dr. GRPO exhibit similar trend DeepSeek-RI-Zero (Guo al., 2025), namely their response length increases along with training reward (Plots & 2). However, we observe that GRPO tends to continually generate longer responses even when the reward improvement slows down (Plot 2). Although such phenomenon is often referred to as the "emergence" long-CoT through RL (Zeng et al., 2025; Hu et al., 2025), we argue that
potttss:/gittie htwpthtww..ccoooggggggaaa/aaaaaa

---

Published as a conference paper at COLM 2025
R1 Template
Qwen-Math Template
Template
Accuracy (%)
30
20-
ORZ-57K
ORZ-57K
ORZ-5 7K
MATH-12K
MATH-12K
MATH-12K
GSM-8K
6SM-82
C.SM8KK
ASDix22K
ASDM-2K
ASDiv2KK
0
100
200
300
100
200
300
160
200
300
Policy iteration step
Policy iteration step
Policy iteration step
Figure 6: The average benchmark accuracy of different (template, question set} combina- tions during RL training.
it is also confounded by the response-level length bias (Sec. 3.1) during optimization". In contrast, by computing the unbiased policy gradients, Dr. GRPO prevents the response length from growing wildly during training (Plot 2). Moreover, on evaluation benchmarks, the length of incorrect responses is substantially reduced by Dr. GRPO compared the baseline (Plot 4), suggesting that an unbiased optimizer also mitigates overthinking (Chen et al., 2024).
3.3 A Duet of Template and Question Set Coverage in RL dynamics
Recall that the Qwen2.5-Math base models can readily answer questions with high accuracy without any prompt template (Sec. 2.2). Based on this intriguing observation we are interested in how different templates affect the RL training. Furthermore, given the general belief that larger question set coverage leads to better performance (Luo et al., 2025; Hu et al., 2025), we also study the interaction between different templates and different levels of question coverage.
Experimental settings. Starting from the Qwen2.5-Math-1555 base model, we apply R1 template, Qwen-Math template and No template respectively to run RL using Dr. GRPO. All experiments are repeated for different question sets that are detailed in Table 3.
Question set
#
Description
ORZ
57k
Combining AIME, Numina-Math, Tulu3 MATH; diverse and large amount High-school math competition questions Simpler grade-school math questions
MATH
12k
GSM
8k
ASDiv
2k
Basic algebra (1 x questions
Table 3: Different question sets that have different levels of difficulty and coverage.
Results. Fig. 6 shows the RL curves of different runs, from which can make several interesting observations: 1) Templates determine the performance of the initial policies, but RL can improve all policies to comparable performance of ~ 40% (given proper question set); 2) When using the R1 template, question sets have significant impact on the dynamics of RL, with too narrow coverage leading lower plateau performance. However, when using the Qwen-Math template, the best final performance is attained by RL on GSM-8K, demonstrating that training on much simpler (and o.o.d.) questions can largely improve (nearly double) the test accuracy on harder questions. From these observations, we draw the following insights:
The Swwe2.5--ath-11.. base model already possesses strong math-solving capabilities (see the starting point the right plot of Fig. 6). Applying templates in fact destroys the capability before RL reconstructs it. This implies that we should be more conservative in claiming the huge gains brought about by pure RL.
We note that both Zeng et al. (2025) and Hu et al. (2025) employ PPO, which is unbiased by formulation. However, their loss implementations still introduce the length bias (see listing 1).
8

---

Published as a conference paper at COLM 2025
Base Models
Algorithms
Output length (dashed)
18
1.2k
Accuracy (%)
12-
1.0k
0.8k
0.6k
100
200
200
Policy iteration step
Policy iteration step
Uaam332-33 NuminaQA
Dr. GRPO GRPO
Uama3.2-38 FineMath
3lama3.2-3B
Figure Left: The average benchmark performance curves of different base models. Right: The comparison between Dr. GRPO and GRPO with respect to reasoning accuracy (solid lines) and model response length (dashed lines).
When there is a large mismatch between base models and templates (e.g., R1 template mismatches (wen2.5-Math-1.5B) the policy improvement mainly comes from RL- tuning, thus requiring question set to have good coverage (left plot of Fig. 6). Otherwise, even small and completely o.o.d. question set could induce the reasoning ability equally well, by reinforcing useful reasoning behaviors instead of infusing new knowledge.
3.4 Domain-Specific Pretraining Improves RL Ceiling
Recent successful R1-Zero-like replications of math reasoners mostly employ Qwen2.5 base models as the initial policies (Zeng et al., 2025; Cui et al., 2025; Hu et al., 2025), which are already strong math solvers and exhibit self-reflection patterns (Sec. 2.2 and 2.3). In this section we hope to explore the other side: can R1-Zero-like training succeed on originally weak (in terms of math reasoning) base models? We answer this question affirmatively, with the observation that math pretraining would improve the ceiling of RL.
Experimental settings. We adopt the Llama-3.2-3B base model as our starting point, and use the unbiased Dr. GRPO algorithm for RL-tuning with the RI template. We hypothesize that domain-specific pretraining would help RL, hence adopt the which is continual pretrained on the FineMath dataset (Allal et al., 2025). Moreover, as hypothesize that Qwen2.5 models likely to be pretrained concatenated question- response texts (Sec. 2.2), we similarly prepare concatenated dataset from NuminaMath- 1.5 (Li et al., 2024b), and continual pretrain learning rate 1e-5. We refer to the concatanated continual pretrained model as Llama-3.2-3B- NAminaQA.
for 2 epochs with
Results. We present the RL curves of different base models in the left plot of Fig. 7. We observe that RL can even improve the vanilla Llama base model, but the gain is minimal. After continual pretraining (and concatenated continual pretraining) to embed math domain knowledge, Llama models can show much stronger RL performance, validating our hypoth- esis. We also revisit the GRPO's optimization bias with the Llama base model. The right plot of Fig. 7 compares the model performance and response length trained with GRPO and phenomenon,
Dr. GRPO. We can clearly see that GRPO can produce the "'oubbleincrease" potentially leading to misperception that long-CoT can also emerge on Llama models after math pretraining. Unfortunately, the increase of length might be due to the optimization bias (Sec. 3.1), which can be effectively mitigated by the proposed Dr. GRPO (Sec. 3.2 & right plot of Fig. 7).
Closing Remarks
We have taken critical perspective to examine base models used for R1-Zero-like training, as well as algorithms used for RL. Through the analysis, we demystified how pretraining
hnttps://huggiggfaat httpeeeee//ooggggggeeeeeeeeeeeee Finneeeatt-eeee-aaaaaaaaaaaa -ate//ineMatt--Laa----- Llama-

---

Published as a conference paper at COLM 2025
biases influence RL outcomes and how optimization choices, like GRPO, can unintentionally shape model behavior. With the proposed Dr. GRPO, we offer a simple fix that improves token efficiency while preserving reasoning performance. Our results show that scaling RL can be both effective and siticient-sometingg less really is more.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint ar arXi::2402.14740, 2024.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydliček, Agustín Piqueres Lajarin, Vaibhav Srivastav, et al. Smollm2: When smol goes big- dat-eeta-centric training of a small language model. arXiv preprint ar 2rii:2500.027377 2025.
Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk,, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, al. What matters for on-policy deep actor-critic methods? a large-scale study. In International conference on learning representations, 2021.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint urXio:2412.21187 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arviv preprint ar XrXi:25502.01456, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 0tti::20002217883, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. ar preprint ar Xix:::2501 11.5001..1944... 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging bench- mark for promoting agi with olympiad-leve bilingual multimodal scientific problems. ar Xiv preprint ar Xrxv:2240..1400 14008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ar xiv preprint ar arXio:2103.08874, 2021.
Jian Hu, Xibin WU, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Open- rlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint ar arXiv:2405.11143, 2024.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung- Shum Xiangyu Zhang. Open-reasoner--zero An open source approach to scaling reinforcement learning the base model. https:////t wtt.ettt.....opeeeeeeeeeee com/ Sewpu//coooopee-easseeeeee---- Zeere//perr-//-eeeeee 2e 2025.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXit:1907.00456 2019.
10

---

Published as a conference paper at COLM 2025
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free!, 2019.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu Pushing frontiers in open language model post-training. ar preprint ar drv::24411.1111 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843 3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024a.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath, 2024b.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, al. Deepseek-v3 technical report. arXiv preprint ar Xit::2412.1937,, 2024.
Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Oat: A research-frieedly framework for ilm online alignment. https://gi thub. com/ www.cccomsaoooeeeecooeoo sait-sg/at, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment in rl-zero-like training a pilot study. https:://at.lnn http:///aaammmmooooooooo oat-zero, 2025b. Notion Blog.
notion.toon...mmm site/
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with 1.5b model by scaling rl. https://gi com/ agenticc-ppojeec//eeppssaer 2025.
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. 2025. Accessed: 2025-01-24.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint ar xrXiv:1506.02438, 2015.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. Xiv preprint wxXi::1704.0444/ 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ar Xiv preprint ar Xkix:1707.0634/ 2017b.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprini ar arXio:2402.033m 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. ar Xrv preprint ar XrX:::2409.199 19256, 2024.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi- billion parameter language models using model parallelism. ar Xio preprint ar *=Xi=:199.0080:: 2019.
11

---

Published as a conference paper at COLM 2025
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Adoances in neural information processing systems, 33:3008-3021, 2020.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning, https://gl
wcwm/hhgiingaaceeeeee 2020
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.151155 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvemmen. arviv preprint arXiv:2409.122 121222, 2024b.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in lims. arv preprint ar Xi::250022.0373. 2025.
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https: https://hhuttnnip.nniionn wttpa/iiwii..eeeieeeeeeeeeeeeeee
2025. Notion Blog.
12

---

Published as a conference paper at COLM 2025
A Policy Gradient Derivations
In the context of RL for LLM post-training, we typically maximize the value of
I (
E qr La~Too-- [[qq.))]
(4)
where R(q.o) = - r(q. 0zt) is the return (Sutton & Barto, 2018) of the trajectory q, o, and ogr) represents the token-level reward for f-th token in response o. The Monte Carlo policy gradient (Sutton & Barto, 2018) of Eq. (4)
V VaJ = E q~
E Low
Ve log Top ((qg)))(()))))
(119)
E
E
log Ta 0 R
9~>P
O (q)
(5)
o
E
E
log no (0 0.0
Σ r(q
(~
Σ r(q -(q )
E P~PP
E 0
0 log (0
where B(q, 0ct) is variance reduction term, which is invariant with respect to that 21.000-19900211
[ log 9,00 ((:))(4)0)))
Σ 2
)
B(q 0..)
v
9, 0
O
V 8 Σ πD [
[ 0z<r) =0
- r(q,0r)) which is the expected cumu- lative reward in the future (also known as the value of the current state), and denote A(o: q, 0() - rgr)) B(q, as the advantage. In the case of outcome reward, Σ r(q ogr) = - r(9.05)) = R(q.o).
Typically, we set B(q, 0xi)
E
By setting oct = mean (R((q, Rcq,) the policy gradient of Eq. (5) be- comes
G
VOJ(
E 10 m q)) q~Pe
E
V log (000))
(6)
where
A R(q mean R
(R(
We adopt the PPO (Schulman al., 2017b) objective compute Eq. (6):
J(Ta) Eiqq
Po
(o:)
Te... (119)
To(o q, o,,)) Tor., (0 0,
ne(oi, q, 0i
e, +e A. }.}.
A, clip
G
Term (0: (0:11q.00.c)) o,et)"
from which we conclude that both std and |o| should not appear in the RL objective.
13

---

Published as a conference paper at COLM 2025
Unbiasedness of Å. Wenote that A computed above is equivalent to that of REINFORCE Leave-One-Out (RLOO) (Ahmadian al., 2024; Kool et al., 2019) up to a scaling factor, which can be subsumed into the learning rate without affecting the RL dynamics. Specifically,
G An =
G
G
G G R(q,o;)
L ((q
G R(q,o;)
Σ
R(q
G R(q 0;)
G
1
ARLOO 1
B Detailed Benchmark Results
We show the detailed benchmark results for three scales (1.5B, 3B and 7B) in Table We also include the instruct models at the same scale and R1-Distill models for comparison. Note that since we employ the Qwen2.5-Math base models, which have a context length of 4k, we thus limit the generation budget at 3k for all baselines compared. For models that are trained for longer context Reapeneesnner--ee -ero end R1-Distill-Qu DistilllQwenn) we also report their performance at 8k generation budget.
Base model + Method
AIME24
AMC
MATH500 Minerva OlympiadBench Avg.
20.0
32.5
33.0
12.5
22.8
24.2
Qwen2.5-Math-1.58
16.7
43.4
61.8
15.1
28.4
33.1
Oat-Zero-1.5B
20.0
53.0
74.2
25.7
37.6
42.1
RI-Distill-QWwen-1.5B @ 3k
2.5
21.7
52.2
16.3
17.3
22.0
E1-Distill-Qwenn-l @ 8k
20.0
49.4
77.4
25.0
35.8
41.5
Qwen255-Math1155--nnsrruuu
10.0
48.2
74.2
26.5
40.2
39.8
Llama-3.2-3B
0.0
2.4
6.4
6.3
1.3
3.3
RL w. Dr. GRPO
3.3
7.2
10.0
11.0
2.2
6.8
Jlama-3223D-FineMMal
0.0
3.6
18.4
5.9
2.2
6.0
+ RL w. Dr. GRPO
3.3
10.8
38.0
12.9
9.0
14.8
Jlama-3.2-B-Nummnnd
0.0
0.0
0.6
0.0
0.1
0.14
RL w. Dr. GRPO (Oat-Zero-3B)
6.7
18.1
50.0
14.3
14.7
20.7
Llama-3.2-3B-Insttrus
6.7
15.7
38.8
11.8
12.6
17.1
Qwen2.5-Math
16.7
38.6
50.6
9.9
16.6
26.5
-
0.2
45.8
69.0
21.3
34.7
38.2
SimpleRI-Zero-7B
26.7
60.2
78.2
27.6
40.3
46.6
PRIME-Zero-BB
83.8
36.0
40.9
48.0
OpenReasooner-eeno-78 @ 3k
13.3
47.0
79.2
31.6
44.0
43.0
OpenReasonerrZeen--78 @ 8k
13.3
54.2
82.4
31.6
47.9
45,9
Oat-Zero-7B
43.3
62.7
80.0
30.1
41.0
51.4
H1-Distill-Qwen-- @ 3k
10.0
26.2
60.1
23.0
23.1
28.5
R1-Distill-Qwent @ 8k
33.3
68.4
88.1
35.9
47.7
54.7
Qwenn.55Mahh-7B-nsttuuc
16.7
53.0
83.6
29.8
42.7
45.1
Table 4: A comparison on benchmark scores. Ours models are RL-tuned by our minimalist recipe (Sec. 1). means we employ the best template (no template) generate answers, such that the test scores are highest and can faithfully reflect the capabilities of the base models.
C Extended Empirical Results
In this section we present two extended empirical results for (1) the ablation of different bias terms in GRPO and (2) statistical significance of Dr. GRPO's results. We RL-tune the
14

---

Published as a conference paper at COLM 2025
Qwen2.5-1.5B base model on mixture of 3K diverse math questions drawn from ASDiv, MATH, and AIME (pre-2023).
Training reward
Training length
Average benchmark score
Response length
0.6K
Accuracy (%)
Reward
0.2
0.4K
10
50
100
150
50
100
150
50
100
150
Policy iteration step
Policy iteration step
Pollcy iteration step
Dr. GRPO
GRPO w/o Len Norm
GRPO w/o Std Norm
GRPO
Figure 8: Ablation results on the two bias terms in GRPO.
Fig. 8 shows the training and evaluation curves for the following variants: Dr. GRPO, GRPO w/o length normalization, GRPO w/o standard deviation (std) normalization and Vanilla GRPO. From the middle subplot, observe that both Dr. GRPO and the variant without length normalization generate shorter responses compared to the other two. This confirms that the length bias term has more significant influence response length-consistent with our expectations.
In terms of performance, Dr. GRPO and the other ablated variants consistently outperform vanilla GRPO in both training rewards and evaluation accuracy. This indicates that removing bias terms (either length std) improves policy learning, validating our motivation for Dr. GRPO.
Average benchmark score
Average output length
25
1200
Accuracy 1%6)
20
Output length
1100
1000
800
5
700
0
100
200
0
100
200
Policy iteration step
Policy Iteration step
GRPO
GRPO
Figure 9: Evaluation results of 3 independent RL runs. The mean curves are drawn in solid lines and the standard deviation plotted in the shaded areas.
Fig. 9 compares GRPO and Dr. GRPO across three independent runs. We observe that Dr. GRPO consistently demonstrates statistically significant improvements-both in token efficiency and final accuracy-across different random seeds.
D Keyword-based Detection and LLM-Based Identification of Self-Reflection Behaviors
We construct a pool of carefully selected keywords and phrases that signal self-reflection behaviors in the LLM's responses. However, LLM-generated responses often contain hallucinations and off-topic content, leading to the presence of simple, ambiguous keywords that do not necessarily indicate genuine self-reflection. For instance, terms like "wait"
15

---

Published as a conference paper at COLM 2025
and "try again" frequently result in false positive detections. To reduce false positives, we maintain a small, highly selective keyword pool consisting of terms that are strongly indicative of self-reflection. In our experiment, the keyword pool is limited to: recheck, rethink, reassess, reevaluate, re-evaluate, reevaluation, re-examine, reexamine, reconsider, reanalyze, double-check, check again, think again, verify again, and go over the steps.
103
Count self-refflectin keywords
102
10
10
check again double-check go over the steps
re- evallaate
re-examine
reanalyze
reassess
recheck
reconsider
reevaluate
reevaluation
reexamine
rethink
think again
verify again
Keywords
Mwenn.5-MMtth1..8 Qwen2.5-Math-7B
Qwen2.5-7B
DeepSeek-Math-78
Llama-3.1-8B
DeepSeek-V3-Base--G5
Figure 10:
Count of keyword occurrences out of 40,000 responses (500 questions X responses per question X 10 temperatures), y is in log scale.
We present the occurrences of various keywords in the responses generated by different models in Figure 10. Interestingly, different model families emphasize different keywords. For instance, phrases such as "check again", "double-check" "re-evaluate", "re-examine", "recheck", "reconsider", and "verify again" appear most frequently in the Qwen2.5 family. In contrast, "re-evaluate", "re-examine" and "verify again" do not appear in the responses of the DeepSeek family, while Llama models frequently the phrase "think again." " We hypothesize that this phenomenon results from differences in the pretraining data, particularly in relation to reasoning and mathematics.
Although we meticulously select the keyword pool, it may still be insufficient to identify some implicit behaviors of self-reflection that do not contain a specific keyword. Addi- tionally, it can lead to false positives, as illustrated in Case (a) of Figure 11. To address these limitations and more accurately assess the self-reflection capability of base models, we leverage stronger LLMs (GPT-40-min) in our experiments) to analyze the responses and determine whether they exhibit explicit self-reflection (e.g., keywords like "recheck" and "reevaluate")) or implicit self-reflection (e.g., more sophisticated patterns that cannot be easily captured through keyword matching). This approach helps distinguish true self-reflection behaviors from superficial or incidental use of related terms.
While LLM-based detection effectively filters out false positives from keyword-based de- tection and identifies implicit self-reflection behaviors, it can still misclassify responses, particularly when they are lengthy and complex. For instance, Case (b) in Figure 11 shows false positive in LLM-based detection, where the response is categorized by the LLM but does not actually exhibit self-reflection. This type of error can filtered out by keyword- -ased detection. To enhance robustness, integrate keyword-
self-reflection
16

---

Published as a conference paper at COLM 2025
Case(a): What the smallest positive multiple of 23 that more than a multiple of 89?
Case(b); What is the greatest multiple of that is less than 500 and is also a multiple of 4?
Response
Response
To find the smallest positive multiple of 23 that, ...
To find the greatest multiple 3 that is less than 500 and is also multiple of 4, I need to first consider the greatest multiple of 4 less than 500, which is 496. However, 496 is not multiple of 3. I then need consider the next multiple of 4, which 492. 492 a multiple 164 with no remainder. remainder.</think>
Start with the smallest possible value of k by checking if they satisfy the equation. not, increase and check again.
since 492 /
Now, trying k = 21: 23 21 483
<answer> 492 <answer>
Figure 11: Case (a): false positive in keyword- detection. Case (b): false positive in LLM-based detection.
and LLM-based detection through cross-validation. The combined detection results, along with the individual results from keyword-based and LLM-based methods, are presented in Figure 12.
Keyword-base baseed Detection
LLM-based Detection
Cross Detection
80
Count of Questions
Count of Questions
Count Questions
50
40
30
40
20
10
0.5.0.6.. Temperature
0.10.20.30.. 4. 0.5.0.60.70.80.31
0,1
0.8
Temperature
Temperature
0wen2.55Mtth-1.58
Qwen2.5-78
DeepSeek-Math-78 DeepSenk-Va-ase--e
Owen2255Math-78
Llama-3.1-88
Figure 12:
Comparison of keyword-based detection, LLM-based detection, and cross detec- tion. Self-reflections are counted the question level across 500 questions, where a question is marked as having self-reflection if at least one of its eight responses exhibits self-reflection.
Examples of Aha Moment in DeepSeek-Vä5 V3-Base
Fig. 13 shows two examples to demonstrate that the DeepSeek-Ve V3-Base model already exhibits the so-called "aha moment" even before the RL-tuning.
Comparison Between DeepSeek-V3-Base and DeepSeek-R1-Zero
We analyze DeepSeek-V3-Bass and DeepSeek-R1-Zero understand changes in model behavior during R1-Zero training. In Fig. 14, we present the breakdown of response categories across difficulty levels for 500 MATH questions evaluated on both models. The results indicate that most incorrect responses substantial performance gains from R1-Zero training. Meanwhile, we find an increase in unformatted responses, which aligns with the observation in Liu et al. (2025b).
corrected after RL training, demonstrating
In Table 5, we report the average response lengths across categories. Note that truncated responses would fall into any of the other three categories if larger context size were used; thus, we exclude them from the table. The results show substantial increase in response lengths across all categories, including correct responses, consistent with the results in the
17

---

Published as a conference paper at COLM 2025
An element randomly cho- sen from among the first rows
Question(a):
Question(b): Find the least positive inte- ger иl such that 1/sin 45" sin 46 +...
Response
Response
In Pascal's Triangle, every row starts and ends with 1,...
First, I notice that the left-hand side of the equation has the form...
Simplifying, I get
This can be calculated as: "awkward silence" Wait, I'm overthinking. Let's try again. The number of elements in the first rows of Pascal's Triangle (n + 1)1/ [21 (n 1)!]
Aha! I can use this to get sin + +b) sin + b) + b) = sin b) This looks messy, but notice that it has
the form sin cos + cos a sin again.
This gives me an idea. Maybe can
Figure 13: Cases showing that DeepSeek-V3- V3-Base already exhibits " "Aha moment" even before RL tunning.
DeepSeck.V3-Base
DeepSeek-RI -RepSeek-R1-Zero
Number of Responsas
800
Category
Base
R1-Zero
400
Unformatted
880.7
7870.3
Correct 621.3
4965.4
200
Incorrect
1038.9
8206.1
4
5
Level
Level
Table 5: Average response string lengths across categories for DeepSeek- PeepSeek-V--Bsse V3-Base (Base) and DeepSeek-R1-Zeno (R1-Zero).
correct
Incorrect
unformatted
truncated
Figure 14: Breakdown of response categories across difficulty levels in the MATH dataset for DeepSeek- V3-Base and DeepSeek-R1-Zen -R-zero.
Fig. of Guo al. (2025). However, the average length of incorrect responses notably longer than that of correct responses. We hypothesize this is because more challenging questions generally require longer responses due increased reasoning complexity, and incorrect responses more likely to originate from harder questions, resulting in longer average length.
Self-reflection does not necessarily imply higher ac- curacy. To investigate whether self-reflection behav- iors associated with model performance during the inference (acknowledging that self-reflection may improve exploration during training potential pos- itive effect outside this section's scope), we analyze questions that elicit at least one response with self- reflection from DeepSeek-R1-Zero across eight trials. For each question, we sample 100 responses and divide
Accuracy Difference
0.0
-0.1
-0.2
Mean
them into two groups: those with self-reflection and those without. We then compute the accuracy differ- ence between these two groups for each question. As shown Fig. 15, the results indicate that nearly half responses with self-reflection do not achieve higher accuracy than those without self-reflection, suggesting that self-reflection does not necessarily imply higher inference-stage accuracy for DeepSeek-R1- -1--eero.
Median
-0.3
Figure 15: Accuracy difference be- tween responses with and without self-reflection for each question (re- sponses sampled from DeepSeek- R1-Zero).
18

---

Published as a conference paper at COLM 2025
G Detailed Experimental Settings
All our experiments are performed on 8 x A100 GPUs and finished in about one day. We enable the actor-learner collocation supported by Oat (Liu et al., 2025a) optimize the training efficiency. We show the experimental configurations in Table 6.
Parameter
Value
ACTOR
Maximum response length
3000 tokens
Sampling temperature
1.0
(top P, top k) Number of responses per question
(1.0,-1)
8
LEARNER
Optimizer
AdamW
Adam parameters (B1, P2)
(0.9, 0.95)
Weight decay
0.0
Gradient norm clipping
1.0
Learning rate scheduler
Constant
Learning rate
10-6
Inner proximal update epoch
1
KL loss coefficient
0.0
KL penalty coefficient
0.0
Policy clipping parameter
0.2
Table 6: Hyperparameter configurations used in all experiments.
H Prompts Used for GPT-As-A-Judge
Prompt for checking the model's question-answering ability.
Prompt for Checking Question-Answering Ability
I will send you a question and long response generated by an LLM. Your task to determine whether the output attempts to answer the question not. The output may sometimes include irrelevant content, hallucinations, or random, off-topic responses.
Please classify the output into one of the following categories:
Output Format:
Your response must start with single integer (0 or 1), followed by a brief explana- tion.
Return 0: - The output is not trying to answer the question (e.g., irrelevant content, random talking, hallucinations). Example output: '0: The response is off-topic and does not address the question.'
Return 1: - The output attempts answer the question, regardless of how complete or accurate the answer is. Example output: '1: The response engages with the question, even if the answer incomplete or incorrect.'
Question: (question) Response:
(response)
19

---

Published as a conference paper at COLM 2025
Prompt for LLM-based detection to determine whether a response contains self-reflection behaviors.
LLM-based Detection for Self-Reflection
I will send you mathematical question along with a detailed response. Your task is to determine whether the response attempting to answer the question. If the response is off-topic, hallucinated, random talk, or otherwise irrelevant, mark it as 0. Otherwise, assess whether the response exhibits self-reflection. Categorization Rules:
1. Category O: The response is off-topic, nonsensical, incoherent, overly repetitive, or lacks logical reasoning.
Example cases:
The response does not relate to the question.
contains meaningless or hallucinated content. consists of excessive repetition without coherence.
2. Category 1: The response attempts to answer the question but does not exhibit self-reflection.
Example cases:
The response directly solves the problem without revisiting steps. No attempt is made to verify the correctness the answer or explore alternative solutions.
3. Category 2: The response demonstrates self-reflection at any level.
This may include:
Explicit self-reflection keywords, such as: *recheck, rethink, re- assess, reevaluate, re-evaluate, reevaluation, re-examine, reexamine, reconsider, reanalyze, double-check, check again, think again, verify again, go over the steps", etc.
Implicit self-reflection behaviors, such as revisiting the solution, questioning assumptions, or considering alternative approaches without explicit keywords.
If any form of self-reflection is present, always categorize it as 2, regard- less of correctness answer quality.
4. Category 3: The response consists solely of Python code for calculations without exhibiting self-reflection.
Example cases:
The response only provides a Python script to compute the solution without any verification, re-evaluation, or alternative considera- tions.
Output Format:
Your response should first provide very brief explanation of your analysis, fol- lowed by single category number (0, 1, or 3) at the end. You must include the category number at the end of your response.
Example outputs:
'The response is off-topic and does not attempt to answer the question. 0.'
'The response provides a direct solution without self-reflection. 1.'
The response demonstrates self-reflection. 2.'
'The response consists solely of Python code without any self-reflection. 3.'
Question: (question)
Response: [response]
20


Reviews:
[{"review_id": "lSD95DzQBK", "reviewer": "Reviewer_FjK5", "summary": "This work investigates the recent phenomenon of R1-style training to deconstruct it in order to improve it. First, they investigate base models and find that Qwen base already includes instruction-following abilities and Deepseek V3-base already includes self-reflection abilities. Next they argue the GRPO objective incorrectly averages by the response length and divides the advantage by the std of a group of responses. The experiments section train models, mainly Qwen, on math tasks. Based on experiments, it is argued that templates may be unnecessary for models with some amount of instruction-tuning and they may even be harmful to performance. Finally, they improve llama 3.2 3B with math pretraining to show that domain specific question-answer pretraining is key to leveraging RL.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a series of results that help illuminate modern r1-style training. This is especially useful as technical reports from companies often omit smaller details and open-source reproductions. Dr. GRPO is a good introspection of the GRPO method and has some potentially interesting theoretical insights. There are definitely interesting insights about R1 here, although they are quite scattered without a clear story.", "reasons_to_reject": "The work is interesting but disorganized. It presents lots of separate results without a particular story. Still, I believe it is worth accepting if the method part of the work Dr. GRPO can be made a little stronger. As it stands, I believe the work has theoretical issues and it is not strongly \n\n**response length bias may be wrong**\n\nI believe the point on Response Length Bias is wrong but I am open to changing my mind\n- GRPO (and PPO) are extensions of regular policy gradient\n- according to the policy gradient theorem, the gradient $\\nabla J(\\theta)$ is not equal but *proportional* to $ \\propto \\sum_s \\mu(s) \\sum_a q_\\pi (s,a) \\grad \\pi(a|s,\\theta) $\n- this proportionality is the average length of the episode (equation 13.5, Sutton and Barto, 2020)\n- in the case of the MDP of token generation, we know the exact length of the episode, which is the number of tokens generated\n- I believe this is the reason for averaging by the number of generated tokens\n\nWe don't see the corrected behaviour in Dr. GRPO\n- removing averaging should increase the response length of correct answers \n- but Figure 5 (bottom left) shows that the original GRPO has the longer correct response lengths\n- the main practical reason for the averaging is that, without it, a model is incentivized to generate more tokens for a correct answer\n\n**better justifying Dr. GRPO empirically**\n\nOverall, given that there are two innovations here, it would be reasonable to do an ablation and show the effects of each one. A 1.5B run would be sufficient with just removing response length average and just removing dividing by std deviation.\n\nSecond, it is unclear whether Dr. GRPO actually improves results empirically or just reduces output length. The only claim is that it improves \"token efficiency\", which is to say it reduces the length of generations? What is the specific speedup for your experiments and how robust are they? i.e. do you have a mean + std over seeds?", "questions_to_authors": "wouldn't Deepseek base also have to have seen something like instruction-following data in order to follow a template such as User: ... Assistant:.... Can you give an idea how you think Qwen different? Just more explicitly structured data?\n\n\nin Section 3.2, you claim\n> even a small and completely o.o.d. question set could induce the reasoning ability equally well, by reinforcing useful reasoning behaviors instead of infusing new knowledge.\n\ndo you have proof or a citation for that? it seems unrelated to your experiments on templates\n\n\n\nWhat is the takeaway from the experiments with Llama 3.2? Is it that NuminaMath is the right choice for pretraining because of the question-answer format and the initial bad performance could be due to a bad template?", "extra_scores": {}}, {"review_id": "qq7c0IGX1S", "reviewer": "Reviewer_pF1u", "summary": "This paper highlights the two fundamental components of training R1-Zero-like models: the base model and the reinforcement learning technique. It provides extensive insights, including how templates affect model performance, the importance of the base model's self-reflection ability, and the potential bias introduced by GRPO. The authors also propose a simple fix to GRPO, which produces unbiased responses while maintaining comparable performance.", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- In my opinion, this paper provides valuable insights into training R1-Zero-like models. All the findings are supported by rigorous experimental results with clear explanations. I enjoyed reading the paper.\n- The finding that “Aha Moment Already Appears in Base Models Including DeepSeek-V3-Base” is particularly inspiring and insightful. It identifies the cause of the \"aha moment\" and suggests a potential direction for improving the model’s reasoning ability in the future: focusing on enhancing the base model rather than relying on additional reinforcement learning techniques.\n- The finding of bias in GRPO is reasonable. It is somewhat surprising that simply removing certain terms does not sacrifice performance and can even reduce overthinking.", "reasons_to_reject": "I do not see any significant weaknesses in this paper. One minor issue is that Figure 5 overlaps with the caption of Table 2.", "questions_to_authors": "Given the argument regarding the length bias of GRPO, is it possible to design a normalization term that depends not only on the length but also on the reward value?", "extra_scores": {}}, {"review_id": "XIc17rrNat", "reviewer": "Reviewer_dTZp", "summary": "This paper takes apart the R1-Zero recipe and studies the influence of base model and RL algorithm onto the final reasoning capabilities. It finds that base models already have reasoning capabilities and that the GRPO algorithm contains length bias that encouraged longer outputs and thereby more reasoning. The paper implements a correction and finds that it effectively improves RL optimization. Finally, the authors combine their insights about base model capabilities and algorithmic bias to establish a minimal R1-Zero recipe.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is fun to read because it is inquisitive and informative. There are lots of bits to learn about existing models, the way they are compared and tuned, and about RL algorithms. \n- The writing is clear and concise, and findings well visualized.\n- The experiments clearly and thoroughly isolate the questions under study.\n- It shows that small oversights here and there can have a butterfly effect leading to conclusions of new capabilities. It is worth reflecting over how such oversight can happen, and how it can be prevented in future studies. It invites a more critical perspective and demands more thorough (slower?) research, which imho the field is in dire need of.", "reasons_to_reject": "-", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies the RLVR (reinforcement learning with verifiable rewards) setting, exploring the different base models and the details of the RL algorithm. The paper makes several observations:
- They study the effect of prompt template. For Qwen, no template performs best. For other models, appropriate template is needed.
- For all the base models they consider, they find "Aha moment" behaviors, i.e. self-correction and backtracking, in the samples from the base model, without RL.
- They show that there is a strong interaction between the observed improvement from RL, and the prompt template being used.
Overall, these results suggest that some of the observations made about RL may be misleading, and may be caused by base models which are already distilled / pretrained on reasoning-formatted data. 

Finally, the authors observe issues with the standard GRPO method, which they argue could cause bias towards longer answers. First, they remove the normalization of advantages by the group reward std. Second, they remove the normalization by the sequence length, which is normally applied per-sequence. 

This second modification has also been proposed in a concurrent work [1], and [2] also got rid of normalization by reward std (although they use a different way of estimating advantages). Both of these works are concurrent, so should not be viewed as reducing the novelty of this paper.

The authors show that the modified GRPO (dr. GRPO) achieves the same reward with less tokens per problem.

Overall, this paper makes several interesting observations and a reasonable methodological contribution. The main weakness in my opinion is the quality of presentation: the paper is fairly informal, and the structure is not very clear. Often, it is hard to know what exactly is meant e.g. by "Aha moment".

In the discussion period, the authors also did not respond very clearly to the question of reviewer FjK5 about the justification for per-token normalization. Overall, I believe the proposed modification is reasonably justified, and the experiments support the claims made by the authors.

Requested changes: Please include ablation of dr. GRPO pieces and comment on the statistical significance of the results (how many seeds use, how results are aggregated). Please respond to the theoretical question of reviewer FjK5.

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale
[2] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
Decision: Accept