Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
Authors: Jingcheng Hu
Yinmin Zhang
Qi Han
Daxin Jiang
Xiangyu Zhang
Heung-Yeung Shum
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Reasoning Model
Primary area: applications
TL;DR: Open-Reasoner-Zero, The first open source implementation of large-scale reasoning-oriented RL training focusing on scalability, simplicity and accessibility

Abstract: We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training on the base model focusing on scalability, simplicity and accessibility.
Through extensive experiments, we demonstrate that a minimalist approach, vanilla PPO with GAE ($\lambda=1$, $\gamma=1$) and straightforward rule-based rewards, without any KL regularization, is sufficient to scale up both benchmark performance and response length, replicating the scaling phenomenon observed in DeepSeek-R1-Zero.
Using the same base model as DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance across AIME2024, MATH500, and GPQA Diamond, while demonstrating remarkable efficiency—requiring only 1/10 of the training steps compared to the DeepSeek-R1-Zero pipeline. 
We validate that this recipe generalizes well across diverse training domains and different model families without algorithmic modifications. 
Moreover, our analysis not only covers training dynamics and ablation for critical design choices, but also quantitatively show how the learned critic in Reasoner-Zero training effectively identifies and devalues repetitive response patterns, yielding more robust advantage estimations and enhancing training stability. 
Embracing the principles of open-source, we release our source code, parameter settings, training data, and model weights across various sizes, fostering reproducibility and encouraging further exploration of the properties of related models.

Open- Open- Rpen-Reesoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
Jingcheng Hu 12+ Yinmin Zhang? Qi Han
Daxin Jiang"
Xiangyu Zhang
Heung-Yeung Shum?
TTinngua University SStepFun
thugheng zhangyinmin, hanger)
GitHub: petts::/githieg www.
Reasoner- Zero /Oen Reasoner
HuggingFace: https:// http://ugggaaaae...oeeeeeeeeeeee
Reasoner Reasoen Zero
Abstract
We introduce Open-Reasoner-Zert the first open source implementation of large- scale massoning-oriented RL training on the base model focusing scalability, simplicity and accessibility. Through extensive experiments, we demonstrate that a minimalist approach, vanilla PPO with GAE (A = 1, 1) and straightforward rule-based rewards, without any KI. regularization, is sufficient to scale up both benchmark performance and response length, replicating the scaling phenomenon observed in DeepSeek-RI-Zert ---ero.. Using the same base model, Qwen2.5-32B base, D - our implementation achieves superior perfor- mance across AIME2024, MATH500, and GPQA Diamond, while demonstrating remarkable
only 1/10 of the training steps compared to the DeepSeck-RI-Zunn pipeline. We validate that this recipe generalizes well across diverse training domains and different model families without algorithmic modi- fications. Moreover, our analysis not only covers training dynamics and ablation for critical design choices, but also quantitatively shows how the learned critic in Reasoner-Zero training effectively identifies and devalues repetitive response patterns, yielding more robust advantage estimations and enhancing training stabil- ity. Embracing the principles of open-source, we release our source code, training data, and various model weights, fostering reproducibility and encouraging further exploration of the properties of related models.
Introduction
Large-scale scale reinforcement learning (RL) training of language models on reasoning tasks has emerged as a promising paradigm for mastering complex problem-solving skills. Recent breakthroughs, particularly OpenAl's ol [1] and DeepSeek's 's 11-Zero [2]. have demonstrated remarkable training time scaling: as the training computation scales up, both the model's benchmark performance and response length consistently and steadily increase without any sign of saturation. Inspired by these advancements, we aim to explore this new scaling phenomenon by conducting large-scale RL training, even applying it directly to base models, approach we refer to Reasoner-Zero training.
In this work, we introduce Open-Reasoner Zero (ORZ), the first open-source implementation of aurge-scale reasoning oriented reasoning-oriented RL training on large language models (LLMs) with our empirical best practices, designed be robust, scalable and Limple-tofollood Under Reasoner- Zero paradigm,
The work was done during Jingcheng intersship at StepFun. This research supported in part by the National Key Research and Development Program of China (Grant No. 20237D0121100), and National Natural Science Foundation of China (Grant No. 62495092)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ALME2024
AIME2025
35
Accuracc (%%
Accuracy (%)
Aw
18
Open Reee--annne--eee-33
10
Training Steps
Training Steps
MATH500
GPQA Diamond
95
55-
Accurscy (%)
Accuracy %%
40-
75
55
Apen-Rasnnne--emm
50
Upes-Beastoe--em--
BrepSeek-R1- 2arppert----22mm-e---11
Training Steps
Training Steps
Figure 1: Evaluation performance of Open- Dppen-Raaooner-Zero-1777 (7B, 32B] J on benchmarks (averaged on 16 responses) during training. Using the same base model, Qwen2.5-32B base, as DeepSeek-RR I-Zero- achieves superior performance on AIME2024, MATH500, and Qwen-32B, benchmarks-rrquiriigg only tenth of the training steps.
GPQA Diamond
LLMs are trained to master diverse reasoning skills under verifiable rewards, spanning arithmetic, logic. coding and common-senss reasoning (e.g., scientific problems, numerical reasoning, natural language understanding and even creative writing), While DeepSeek's ' RI-Zero outlined their training pipeline briefly, we provide comprehensive study of our training strategy, with in-depth insights into overcoming training instability from value estimation perspectives in RL. Our goal is to democratize advanced RL. training techniques accessible to the broader research community.
Our proposed Open-Reasoner-Zeen built on the same Qwen2.5-32B base model as DeepSeek-R1- Zero-Qwen-32B,, achieves superior performance on challenging benchmarks including AIME24, MATH500, and GPQA Diamond, while requiring only 1/10 of the training steps. Through extensive ablation studies, we summarize some key findings. Specifically, we find that vanilla PPO using GAE (A = 1 and = 1) and without any KL-related regularization, combined with straightforward rule-based reward, is sufficient to achieve steady scalability in both benchmark performances and response length across varying model sizes, different model families, and diverse task distributions, when trained on large-scale carefully curated datasets. Furthermore, we investigate several critical aspects of Reasoner-Zero training: (1) training dynamics, including how performance and response length evolve throughout training: (2) value and advantage estimation effectiveness, illustrating how PPO's learned critic model leads to robust advantage estimates; and (3) comprehensive ablation studies on key design choices. These investigations provide valuable insights into the mechanisms behind successful large-scale reasoning-orienter RI. training.
Our primary contributions are as follows:
1. We provide fully open-source implementation of large-scale RL training directly on base LLM, a strategy we refer to as Open-Reasoned
2. We present novel insights crucial for achieving stable and scalable Reasoner-Zero train- ing, encompassing key findings regarding effective design choices, alongside a thorough investigation for advantage estimation.
3. We validate the generalizability of our recipe across different model sizes, various model families and diverse task domains.
2

---

102
ORZ--128
0.3
yrr propiitel 1
Response Length
42-
ORZ-I.5B
Train Reward
0.6
ORZ--058
/r ///0 fM1e
my
42
6.2
ORZ-7B
ORZ-1.58
0.0
01:
450
Training Steps
Training Steps
Figure 2: Train-time Scale up on Train Reward and Response Length of Open- Zerene (ORZ)- {0.5B, 1.5B, 7B, 32B). Train Reward and Response Length increase steadily, demonstrating consistent scalability across model sizes. Interestingly. the ORZ-32B Response Length exhibits fluctuations without negatively impacting training stability, highlighting the robustness of our minimalist recipe.
4. We release comprehensive resources including code, data, and model to the community.
2 Scale-up Reinforcement Learning from Base Model
In this section, we describe the strategy and critical components for scale-up reasoning-oriented RL directly from base model. Concretely, we begin by reviewing essential background on Gen- eralized Advantage Estimation (GAE) 3 and Proximal Policy Optimization (PPO) 4 algorithms. Subsequently, we discuss key insights derived from our comprehensive ablation experiments that enable successful scale-up RL training. Finally, we detail the fundamental yet critical implementation settings for our approach, covering data curation, prompt design, and reward function specification.
2.1 RL Algorithm
We adopt the PPO [4] as the core RI. algorithm, diverging from the GRPO used in DeepSeek-R1 R1 Zero 2] Specifically, for each input question q (i.e., prompt), the policy model generates group op.,,,,,. ), where n represents the number of sampled responses (i.e., rollout size per prompt). Each response o, constitutes trajectory тÜ = (so, ao .... ST,-1, oT;-1), where s; the state (prompt + previously generated tokens) and a, is the token generated at step (i.e., token f). Using our rule-based reward function, each trajectory receives single terminal reward R₃ (0, 1}. assigned at the end of the sequence (re = O for t < T. 1, Pr-- R2).
of responses {01, 02,-..,
We utilize GAE
to estimate the advantage A, for each token. The general GAE formula is:
T-t-1
AGAE(AA)_
Σ (TA)*5E4R. K=
(1)
where 61+1 Ti+k (+ (o(+++)) is the Temporal Difference (TD) error, Va is the value function parameterized by Φ. 7 is the discount factor, and À controls the bias-variance trade-off.
The general PPO objective updates the policy parameters @ to maximize a clipped surrogate objective function, and the value parameters @ minimize the error between the value estimate (s()) and target value / target typically the discounted return. The standard objectives are:
T--1 Σ
(p) = E Er
min (li())))) (1
(2)
Jvalue (Ø)
Ey
Ey
Σ (Vo(s.) V. v.tugeet
where (0) =s(o](ss) is the probability ratio, and the clipping parameter e is set to 0.2 in our Th (ar1))
cases. Commonly, V.T target is the estimated discounted return G, = A GAE(TTA) + V)(s).

---

Table 1: Comparison of Open-Reasoner-Zero-- with DeepSeek-RI-Ze Reeeck--R--eeo-QQeee -ee.-owwe--322 DAPO-
Qwen-32B on reasoning-related benchmarks. DeepSeck-R1 Ri-ee-- -waeee-----o------22222 results are from 12. DAPO-Qwen-32H1 results were obtained using our evaluation metric on the released checkpoint.
Model
AIME 2024
AIME 2025
MATH500
GPQA Dia.
DeepSeek- RI-Zero Qwen-32B
47.0
91.6
55.0
DAPO- HAPO--Qen-3225 [S]
50.0
DAPO-Qwen-32B*
48.3
37.9
71.8
16.0
Open- ppen-Reasonnrr-Zero-222
48.1
36.0
92.2
55.5
2.2 Key Design Principles
In this study, we explore best practices for caasoning-orientee RL training, emphasizing stability and scalability, Our key findings are summarized as follows:
Choosing PPO over GRPO We select PPO over GRPO due to its superior value estimation en- abled by learned critic. This critic facilitates accurate token-level value estimation, effectively identifying and devaluing detrimental patterns such as repetitive behaviors, named credit assignment. Consequently, PPO achieves notably more robust advantage estimation compared to GRPO. Lacking a dedicated value network, GRPO struggles to distinguish genuinely correct responses from those occurring within negative patterns (e.g., repetitive loops). This deficiency can misdirect reinforce- ment, leading to training instability and eventual collapse, an observation supported by community discussions Detailed analysis is provided Section 3.3
Algorithm Implementations. Our empirical studies suggests that vanilla PPO already provides highly stable and robust training across different model scales and training durations, without the need for additional algorithmic modifications. Nonetheless, appropriate implementations matter. Through extensive experiments, we found that the choice of GAE parameters substantially impacts performance in dassoinng-orienter tasks. Specifically, the discount factor 7 controls the effective sequence length considered during training: a lower assigns exponentially decreasing weights to future rewards, inducing the model prematurely terminate generation in order more immediately obtain rewards. On the other hand, the GAE parameter À balances bias and variance in advantage estimation. Crucially, large-scale training scenarios, the substantial data volume naturally mitigates variance concerns, encouraging us to adopt a bias-free configuration. Consequently, by setting y = 1 and À 1, we fully capture the long-term dependencies critical for reasoning tasks and achieve stable training. Fortuitously, this also leads to significant simplification of the GAE advantage computation our case:
A AGAE(T 1,A=1) R R-Vo(s))..
(4)
T-1 Σ 1=
Ivalue (0)
- 1a0
-v) -
(5)
where R is the single terminal reward. Detailed derivation and pseudocode can be seen in appendix.
Removing KL regularization. We achieve stable training without relying on any KL-based regularization techniques (e.g., KL shaped rewards and loss), different from the de facto RLHF community [6] and Reasoner model 7 2]. Intuitively, KL regularization constrains the policy model remain close to the original base model distribution, potentially limiting exploration during policy optimization. By omitting KL regularization, our approach offers several practical advantages: (1) it obviates the need to navigate the large and challenging-te tootgg--tttutee-- design space inherent to KL regularization, greatly simplifying the training procedure; and (2) it lowers computational overhead and memory usage, eliminating the need to load the weight of separate reference model and calculate log probabilities using it. Together, these benefits facilitate efficient and scalable large-scale RI. training.
OppeRRI: discussion about vanilla GRPO reproduction link

---

1.2k
1.2k
ORZ 57k
Mw
2.4k
MATHI Train 7.5k
0.9k
0.99k
Length
1.8k
0.6k
0.68
WHO, KL
1.2k
0.3k
0.3k
KL. Loss
Reward Shaping
0.0k
0.8
2-1
ORZ 57k.
- 0.95
0.6
0.5
0.8
Reward
0.4
0.7
0.2
0.2
KL. Lows
Kl. Reward Shaping
0.6
0
50 100
1.5
200
50
100
150
200
0
150
300
450
600
Training Steps
Training Steps
Training Steps
Figure 3: Ablation studies for key design choices in Open-Reasoning-Zerro (ORZ). We reward on training set or MATH500 as model performance metrics. Left. Comparison of different GAE values. Mid. Comparisons of KL-related regularizations. Right. Data scale ablation study. These findings collectively inform our minimalist yet effective ORZ training recipe.
Minimal Reward Function Design. In contrast to approaches such as DeepSeek R1, which utilize dedicated format reward to enforce structured reasoning (e.g., enclosing thought processes within <thinks...</hhink>>)) we demonstrate that the simplest, rule-based reward function is not only sufficient but also optimal, as minimal design leaves no room for potential reward hacking. Notably, even unaligned base models quickly adpot to desired format, suggesting this a straightforward task without requiring complex reward engineering.
Scale up Training Data. We identify that scaling up data quantity and diversity is pivotal for Reasoner-Zero training. While training on limited academic datasets like MATH train set leads to quick performance plateaus, our curated large-scale diverse dataset demonstrates impressive potential for continuous improvement without signs of saturation on both training and test sets.
2.3 Detailed Settings
We instantiate our ORZ approach by utilizing the Qwen2.5-(7B, 32B) base models as our main foundation. This methodology involves directly launching large-scale RL from these base models, bypassing any preliminary fine-tuning stages such as supervised fine-tuning (SFT) or distillation, a strategy also explored in recent works [ 9). Inspired by DeepSeek-RR -1-zero (2], we design our prompt template elicit the model utilize inference computation, gradually mastering the reasoning ability, as shown in the Appendix. We detail our implementation below, focusing on the key components that enable effective and robust reasoning-orienten RL. at scale.
Data Curation. With careful consideration of scalability and robustness, our training data comprises tens of thousands of carefully curated question and answer pairs consisting of math and general reasoning tasks. Specifically, we curate our dataset through comprehensive collection and cleaning process. First, we collect public data from various sources, including AIME (up to 2023), MATH (10), Numina-Math collection (II). Tulu3 MATH [12). OpenR 1 I-Math-220k [13] and AoPS forum. We also synthesize general reasoning tasks using programmatic approaches as additional enrichment. These include logical puzzles, multi-step steti-step reasoning problems, and counterfactual scenarios that require the model to apply structured thinking across diverse domains. Considering the RL training paradigm's 's reliance on accurate reward signals, we exclude problems that are challenging to evaluate with our rule-based reward function, such as proof-oriented problems. This careful filtering ensures accurate and consistent reward computation during training, essential for stable policy optimization. We also employ ILM--based filtering to evaluate problem difficulty, removing samples with extreme pass rates to maintain balanced datasct.
5

---

Table Generalization performance Open- Reasoner-Zero on MMLU and MMLU_ bench-
marks. ORZ achieves superior performance on both benchmarks through RL training on reasoning tasks alone, surpassing Innttructt without additional instruction tuning.
Model
MMLU MMLU PRO
Bwen2.5-32B-Baas
83.3
55.1
Bren2.5-32B-Instiee
83.2
69.2
DAPO-Qwen-3215 Open-Reaasmerr7eeo--222
79.7
64.5
84.9
74.4
Reward Function. Unlike DeepSeek-R1-Zeee (2] our scale-up RL training employs minimalist rule-based reward function that solely checks answer correctness, without any additional format rewards. Specifically, this reward function is designed to extract the content between '<answer>' and "</answer>' tags during training and compare with the reference answer. To maintain clarity and simplicity in scale-up RL, implement binary reward scheme awarding reward of for exact matches with the reference answer, and 0 for all other cases. Surprisingly, we found that under our designed prompt unaligned base model can yield well-formattdd responses in high probability. Moreover, the base model can quickly learn the correct format and reinforce for reasoning and answering incentivized by our simple rule-based reward function alone, shown in Figure 4
Experiments
In this section, we present comprehensive experimental results and analysis our Open- Open-Reasoner- Zero models. We begin with an in-depth analysis of training results and ablation studies. We then investigate the correctness and effectiveness of the value function. Finally, we discuss the evaluation results and in-depth analyze the training process. Hyperparameters provided in the Appendix.
3.1 Training Results
We highlight key findings from our training experiments, examining performance through training reward, response lengths, and generation quality to provide a concise view of learning dynamics.
Figure shows the training reward and average response length curves of experiments for ORZ- (32, 7, I.5, JB where we observe consistent improvements in both metrics during training. This indicates that the models are effectively learning the desired reasoning behaviors.
To further understand the characteristics of the generated responses, Figure I (Right) average length of all responses compared to the average length of responses that correct and incorporate reflection steps. We identify five representative reflection patterns (""wait,", "* ""recheck" ""retry", "alternatively."" and "however."" "!) and use this to determine whether response is reflective, following a methodology similar to [4]. Notably, the average length of correct responses that utilize reflection is consistently greater than the overall average response length across all training steps. Furthermore, both of these length metrics exhibit a clear upward trend as training progresses.
(Right)) illustrates
3.2 Ablation Study
GAE Analysis. We investigated the impact of different GAE À values and found A=1.0 to yield superior training stability and final performance. As shown Figure (Leff), training with GAE 1=1.0 resulted in reward that rapidly increases and then steadily grows, consistently outperforming A=0.95, which exhibited much slower reward progression. In the Response Length, the GAE 1=1.0 curve maintains a reasonable increasing spead during the training process; while the GAE X=0995 leading to collapsed length dynamics. These findings indicate that GAE X=1.0 can better balance the training stability and generation quality.
KL Regularization Analysis. We assessed the impact of KL. Loss and Penalty on the performance clearly shows that omitting both KI. Loss and
and training dynamics of ORZ-7B. Penalty (W/O. KL) achieve optimal training stability, performance, and response length scaling. Both

---

1.0
Correct Reflect. Len.
6k
Avg Len.
Correct Fmit Ratio
0.8
Longth
4k
0.6
2k-
ORZ-7B
ORZ-32B
25
50
75
100
200 400
600
Training Steps
Training Steps
Figure 4: Left. Correct Format Ratio. Results demonstrate rapid adoption structured reasoning patterns by the base model trained simple outcome reward function, suggesting complex reward functions are unnecessary for Reasoner-Zero. Right. Reflection patterns in generation. Average Correct Reflection Length consistently exceeds Average Response Length during training.
KL Loss and KL Penalty mechanism not only slow down the training process but also consume addi- tional computational resources. Furthermore, eliminating these components reduces hyperparameter tuning burden and implementation complexity, which is crucial for scaling up RL training effectively.
Data Scale. We compared training with our ORZ 57k dataset against classic academic dataset, MATH train 7.5k. As depicted Figure (Right), leveraging the larger ORZ 57k dataset leads to sustained improvements in both training reward and response length. In contrast, training with the smaller MATH train 7.5k dataset results in performance- both reward and length-plateauing early, These results underscore the pivotal role of data scale in enhancing training performance and affirm that increasing training data quantity can effectively improve the model's reasoning capabilities.
Reasoner-Zero Training on Smaller Models. To demonstrate the robustness and versatility of our ORZ training methodology, we extend the same training pipeline to smaller-scale models, specifically Qwen2.5-(0.5,..55)B The evaluation results clearly indicate that our minimalist RL approach consistently improves reasoning capabilities even at substantially smaller model sizes. Remarkably, meaningful performance gains are observable even at the scale as small as 0.5B parameters. Detailed training curves for these smaller models are provided in the Appendix.
Reasoner-Zero Training Across Diverse Domains. To test whether the ORZ recipe transfers Cwweoooeeeeee-eeessstteeeeeeeeee Following at 40%:40%:: 15::00%::::::::: 15%:5%) while keeping all algorithmic settings unchanged. As summarized in Table 3 the ORZ-7B Be---BBMixee-- Mixed-Domain model delivers large gains non-math evaluations well, without sacrificing mathematical competence. These results confirm that our ORZ recipe generalizes seamlessly across heterogeneous task distributions without requiring algorithmic modifications.
beyond mathematics, trained ORZ-7B in mixed-domaan setting
Reasoner-Zero Training Across Different Model Families. We validate portability by applying ORZ recipe to Llama-3. 1 -8B Base in the same mixed-domain setting, adjusting only the learning rate (from le-6 to 5e-7). Training remained stable with steady reward and response-length growth (see Figure 6). As detailed in Table the resulting ORZ-Llama-3.1 1-8 model demonstrates substantial improvements over the base model. These results confirm that the ORZ methodology is architecture- agnostic and generalizes effectively beyond the Qwen model family.
Training Distillation Models. We further investigate preliminary results on applying the ORZ training pipeline to distilled models to enhance their reasoning capabilities. This two-stage approach follows DeepSeck-RI [2]. Table 5 shows that our model yields essential further gains, with ORZ-RI- Distill-Qwen-14 14B outperforming larger distilled models like mhistil--wwen-22B..
3.3
Analysis for Critic and Advantage Estimation
To further investigate the impact of our RI algorithm choices, particularly the preference for PPO over GRPO, conducted detailed analyses of the learned value function (i.e., critic), and its downstream
7

---

Averuge Adv.
book
Frequency
0.2
off
20-30 PiE
0.1
221
20 20TE1 201100 201311
10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
Advantage Difference Repelitive Tokens
Visualization Approximations
Figure 5: Left. Advantage comparison between PPO and GRPO repetitive tokens. Our PPO more negative advantages to repetitive patterns than GRPO, demonstrating superior penalization of undesirable. Right. Visualization of value approximations showing how (s() assigns lower values (red) to repetitive patterns and higher values (green) to coherent text, reflecting how the critic effectively identifies undesirable generation patterns.
effects. Our analyses reveal how the accurate critic influences advantage estimations and eventually translates into more effective policy updates compared to GRPO without critic.
Qualitatively, we observed that the value function (a())) learned during PPO training effectively identifies repetitive patterns (i.e., excessive repetition), which consistently occurs when sudden collapse of vanilla GRPO as described in Section 2.2 As illustrated in Figure [ (Right), states st containing such repetitions are typically assigned lower values by V& (i.e., lower expected future returns) compared to states with coherent patterns, a phenomenon known as credit assignment.
To quantify how this precise credit assignment benefits policy optimization, we performed com- parative analysis of advantage estimations between our PPO setup and hypothetical GRPO setup, focusing on repetitive tokens. Inspired by Kimi k we first identified all tokens that appear after the onset of the first noteworthy repetitive pattern within a generation, which we designate as tokens with repetitive patterns. We then calculated the average advantage assigned to these specific tokens by our PPO GAE (A = 7, = 1) setting (which includes batch-level advantage normalization). This was contrasted with the average advantage that GRPO would have assigned to the exact same tokens.
Figure demonstrate during our ORZ-7B training. advantage assignments of our PPO configuration are consistently lower (i.e., more negative) to these tokens with repetitive patterns compared to GRPO across the majority of training iterations. This finding reveals PPO's superior ability to penalize undesirable patterns, thereby fostering more precise and robust learning signal that actively discourages degenerate outputs and promotes higher-quality responses, consistent with our empirical observations in large-scale RL. More detailed analyses are provided in the Appendix.
3.4 Evaluation Results
We present a comprehensive analysis of our results, demonstrating the effectiveness of ORZ across different model scales and benchmarks. Our experiments evaluate both training efficiency and reasoning performance, highlighting the scalability and generalization capabilities of our approach.
In our experiments, ORZ-32B demonstrates significant advancements both efficiency and perfor- mance, shown in Figure []] The model achieves superior accuracies across all benchmarks, notably outperforming DeepSeck-R -eez--errooe22 training steps. Moreover, we also compare ORZ-32B with DAPO-32B, another recent Reasoner-Zero model in Ta ORZ achieves comparable performance on AIME while requiring fewer the training iterations used by DAPO. Note that ORZ remarkably outperforms DAPO on MATH500 and GPQA Diamond. We observed that the released DAPO tends to answer with integer numbers (e.g.. "Answer: 2") even in multiple-choice questions without integers the options. We hypothesize this behavior is related to their data curation and formatting approach, which transforms every answer into an integer for verification disambiguation. This limitation highlights the advantages of our data curation methodology, with high coverage and preprocessing handle diverse answer formats correctly.
while only requiring an order of magnitude fewer
8

---

Table 3: The ORZ recipe transfers seamlessly to mixed-domain setting without algorithmic changes. This approach yields substantial gains in coding (LCB), puzzle task (BBEH) and instruction following (IFEVAL) compared to the math-only training, yet no performance drop math benchmarks.
Model
AIME24
GPQA Dia.
LCB
IFEVAL
BBEH
Awen2.5-7B-Instrue
10.1
35.7
17.1
55.8
10.3
ORZ-7B
17.9
36.6
2.3
37.6
4.0
ORZ-7B-Mixee Mixed-Domain
18.2
39.0
22.2
71.9
12.4
Table 4: The ORZ recipe generalizes robustly across different model families, delivering strong gains on diverse benchmarks when applied to Llama-3. 1-8B Base model in the mixed setting.
Model
AIME24
GPQA Dia.
IFEVAL
MMLU
MMLU PRO
Llama-3.1 -lama-3.1-BB-Bsee
0.16
14.29 27.27
15,96
27.83
10.56
ORZ-Llama-3.1-4
0.31
68.67
60.02
33.51
We further illustrate the training dynamics of Reen-eeeaoner-7eeo models across various sizes in Figure 2 Training Reward and Response Length demonstrate consistent and steady growth across all scales, highlighting the scalability of our minimalist reinforcement learning approach. Interestingly, the Response Length curve of the ORZ-32B model exhibits noticeable fluctuations, yet these fluctuations do not negatively impact training stability or the continuous growth of reward. This phenomenon indicates the robusmess of method against temporary variations in generated sequence lengths and motivates further investigation into understanding and leveraging this behavior.
Finally, we present the generalization capabilities of our models on comprehensive benchmarks like MMLJ and MMI.U MMI.U_PRO. As shown in Table 2 ORZ-32B models demonstrate strong generalization capabilities, significantly outperforming @wenn.5--Insrruct-0 on MMLU, MMLU __RO through pure scaled-up RL training on reasoning-orientee tasks, without any additional instruction tuning.
Related Work
Scaling RL on Base Models for Reasoning The approach that applies RI. directly to base models to master complex reasoning skills, referred to as Reasoner-Zert training, has gain far-reaching attention (2). While several recent works [6 [7] have proposed detailed training recipes for Reasoner-Zero approaches in pilot studies, ORZ stands as the first fully open-source implementation of large-scale reinforcement learning applied directly to base language models for reasoning.
Concurrent efforts have also explored Reasoner-Zero training at scale. DAPO [5] matches ORZ's AIME performance, but uses roughly fivefold more training iterations and underperforms on other benchmarks, potentially due to its data processing strategies. While VAPO [8] reports stronger AIME2024 accuracy with similar iteration budget as DAPO, it scales less efficiently compared to ORZ, reaching only about 60% of ORZ's score at the same iteration budget. Notably, ORZ employs a simpler algorithm design avoiding the value function learning challenges faced by others, establishing it as a more effective and accessible baseline for future research.
Scaling RL on endeavors to scale up RL training on reasoning-enhancee Models. Theses models typically first acquire advanced reasoning patterns from existing reasoning models or high-quality human-labelee
Enhaanee Models Another important line of work 2 g
19
20
data (2 22 23 24 25) through techniques such as SFT distillation or other cold-start approaches. A key advantage of this pre-instruction is that the subsequent RL training more stablem and these models can achieve superior performance under the same RL compute budget compared to the Reasoner-Zero training. We validate that ORZ recipe is also highly effective for such distilled models.
Our Contributions In contrast to previous work, ORZ offers the most comprehensive open framework for Reasoner-Zero training. Our main contributions include: (1) simple yet scalable RI. algorithm implementation that serves as strong and accessible baseline for future research; (2) extensive configurations and benchmarks spanning models from 0.5B to 32B parameters; (3) the

---

ORZ-Llama-3.1-88
PRZZ-Llama-31180
Reward
Response Length
0.15
600
0.10
0.05
200
0.00
200
400
600
200
400
600
Training Steps
Training Steps
Figure 6: Train-time scale up Train Reward and Response Length of mixed-domain setting. The application the ORZ recipe exhibits stable training dynamics on different model architecture, confirming its robustness and model-family-agnootie nature.
in the
Table 5: The ORZ training recipe enables the BeenSeek-RI--istill----eeee
model to grasp advanced reasoning patterns distilled from stronger reasoning models, substantially boosting its performance. This ORZ-RI-Distill 14B achieves strong results on reasoning benchmarks, even surpassing the larger DeepSsek-RI--Dsttll-------- model.
Model
AIME2024
AIME2025
MATH500
GPQA Dia.
DeepStek-RI-Distill---w-aa DeepSSekk-RI-DDssil--ww----
69.7
49.1
93.9
59.1
72.6
60.0
94.3
62.1
ORZ-RI-Dissill-wwenn-
75.2
60.0
95.6
60.4
largest verified reasoning dataset; and (4) state-of-the-ard training efficiency with minimal iterations. Together, these efforts establish foundational open framework for large-scale RL research on LLMs, enabling broader community participation in advancing reasoning capabilities.
Limitations
While ORZ demonstrates significant advancements in scaling reasoning-oriented RL training, acknowledge certain limitations in the current work. Firstly, our investigation primarily focuses on mathematical and general reasoning tasks. We have included results code generation other programming-relater. reasoning tasks. Exploring the efficacy of ORZ on code-based benchmarks, or investigating the potential benefits and challenges of combining mathematical and code reasoning within a unified training framework, remains critical aspect of developing comprehensive reasoning agents. Secondly, while we successfully implement training time scaling in RL, we have yet to fully explore test time scaling like OpenAl ol. Future work should investigate scaling test time computation through multi-tum interactions for contextual reasoning, value models for trajectory assessment, and multi- agent scenarios for developing more sophisticated reasoning strategies.
6 Conclusion and Discussions
We present OpeenRReasoner-Zeer (ORZ), the first comprehensive open-source implementation of large-scale maasoning-orientee RL training. Our experiments show that vanilla PPO with GAE (A = 1) and simple rule-based rewards, without KL regularization, effectively scales reasoning capabilities in language models. This minimalist approach achieves competitive results compared to DeepSeek-R1-22 1-Zero while using significantly fewer training iterations. Our work provides in-depth analysis training dynamics, model behaviors, and advantage estimation. These insights offer practical guidance on scaling RL for complex reasoning tasks, addressing common challenges in stability and convergence. By releasing our complete training resources-cods configurations, data, and model weights across various sizes-we aim to democratize access to reasoning-oriented RL and provide valuable insights for the community.
10

---

References
[1] OpenAI. Learning to reason with llms.
Barning-to--raae sesoon-wtth-lee 1imhh 2025.
[2] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944, 2025.
[3] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.0244 024450000028,, 2015.
[4] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.0000 777476347 2017.
[5] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Win-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025.
[6] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[7] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 12599, 2025.
[8] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -preview with a 1.5b model by scaling rl, 2025. Notion Blog.
[9] Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang. Weihan Cao, Jiangning Liu, Hongwei Liu, Junnan Liu, Songyang Zhang, Dahua Lin, and Kai Chen. Exploring the limit of outcome reward for learning mathematical reasoning, 2025.
[10] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, 2021.
[11] Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath, 2024.
[12] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers open language model post-training. 2025.
[13] Loubna Ben Allal, Lewis Tunstall, Anton Lozhkov, Elie Bakouch, Guilherme Penedo, and Gabriel Martín Blázquez Hynek Kydlicek. Open rl: Evaluating ilms on uncontaminated math competitions, February 2025.
[14] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thougee reasoning in 1lms. arXiv preprint
2025.

---

[15] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo. Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning, 2025.
[16] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-Like Training: A Critical Perspective. 2025.
[17] Weihao Zeng. Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma. and Junxian He. SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild, 2025.
[18] Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu. Mingxuan Wang. Yonghui Wu, and Lin Yan. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025.
[19] Roy Huang Ameen Patel Alpay Ariyak Qingyang Wu Xiaoxiang Shi Rachel Xin Colin Cai Maurice Weber Ce Zhang Li Erran Li Raluca Ada Popa Ion Stoica Michael Luo, Sijun Tan. Deepeoder: fully open-source 14b coder o3- mini level, 2025. Notion Blog.
[20] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang. Peng Cheng, Xi- aoyu Zhang. Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li. Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series.
http:://caarrcccoou--yyrrrreeee 41c notion. site/ 2025.
Skywork- Seei--OOOpeRReoseerreeee
Notion Blog.
[21]
OpenThoughts Team. Open Thoughts. https://peentthoughee January 2025.
[22] Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-ar math- ematical reasoning models with openmathreasoning dataset. arXiv preprint 1688,1, 2025.
[23] Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran EI- Yaniv, Omri Puny, Ido Galil, 7ach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp. Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Ger- ald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang. David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran. Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhi- nav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor- Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramathh Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yuting Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Frick Galinkin, Michael Evans, Katherine Luna, Leon Derczynski, Nikki Pope, Fileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geifman, Eric Chung, and Chris Alexiuk. Llama-nemotron: Efficient reasoning models, 2025.
[24] Somshubra Majumdar Aleksander Ficek Siddhartha Jain Jocelyn Huang Vahid Noroozi Boris Ginsburg Wasi Uddin Ahmad, Sean Narenthiran. Openccderreasoning: Advancing data distillation for competitive coding. 2025.
12

---

[25] Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarin. Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces. http:://buggaaaaacccoooo// httpe///aaaateaeaeeeet///ooee/ee
2025.
[26] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
13

---

In this Supplementary Material, we provide more elaboration on the implementation details, experi- ment results, and qualitative results. Specifically, we present more evaluation results in Section thorough implementations of the model training in Section B and additional analyses and experi- ments in Section C and D These materials offer deeper insights into our methodology, experimental validation, and qualitative findings that support the conclusions presented in the main text.
A More Evaluation Results
In this section, we provide detailed results from evaluating ORZ models of varying parameter counts (0.5B, 1.5B, 7B, and 32B) across multiple reasoning-oriented benchmarks. Specifically, we report performance on AIME 2024, AIME 2025, MATH500, and GPQA Diamond. The results (see Table O clearly demonstrate consistent improvements in reasoning ability with increased model size, underscoring the strong scaling properties of our minimalist RL setup. We release these comprehensive evaluation results as reference to facilitate further research and reproducibility. We also provide the training performance curves for ORZ- (0.5,, 1,5B) in Figure 7
Table Reasoning-orienter benchmark performance across
model sizes.
Model
AIME 2024
AIME 2025
MATH500
GPQA Dia.
ORZ-0.5B
1.0
0.2
31.0
12.1
ORZ-115B
3.5
1.0
58.0
16.8
ORZ-7B
17.9
15.6
81.4
36.6
ORZ-32B
48.1
36.0
92.2
55.5
B Detailed Setting for Training
We initialize both our policy and critic networks with Qwen-2.5 base models (7B and 32B variants), where value head is randomly initialized from u(-v5, v5) with bias term. The policy and critic do not share weights during training. For both policy and critic networks, we employ Adam W optimizer with 5 [0.9, 0.95] without weight decay. The learning rates are set to 1 x 10-6 and 5 x 10 10-6 for the policy and critic networks, respectively. The learning rate schedulers are both constant learning with linear warm-up of 50 optimizer steps. We employ sample packing during training. Prompt for ORZ model training and evaluation are provided in Table
A conversation between User and Assistant. The asks question, and the Assistant solves it. The assistant first thinks about the reasoning process the mind and then provides the with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think>
<answer> answer </answer>. User: You must put your answer inside <answer> <lanswer> tags, i.e., <answer> answer </answer>, And your answer will extracted automatically by the \boxed() tag. [prompt))
Assistant: <think>
Table 7: Template for Open-Reasoner-Zerr. prompt will be replaced with the specific reasoning question during generation.
Each generation step contains 128 unique prompts sampled from the dataset, and policy generates 64 responses per prompt with temperature and top-p both set to 1.0. To maintain training stability, implement strict on-policy optimization for the policy network, where each generation corresponds to exactly one policy optimization step. The critic network, being less sensitive to off-policy updates, processes the experiences in 12 mini-batches, effectively performing 12 optimization steps per iteration. We apply batch level advantage normalization in the training. Notably, training process operates stably without any KL-related regularization terms or entropy bonuses, demonstrating that vanilla PPO can achieve stable training without these commonly used stabilization techniques.
For the 32B variant, we introduce an additional "annealing" training stage inspired by analogous practices in large language model pre-training [26). Specifically, we leverage the training process of
14

---

AIME2024
AIME2225
1.2
ORZ-1.58
ORZ-0.5B
ORZ-0058
1.9
Accuracy (%)
Accuracy (%)
0.8
0.5
0.4
0.2
6x110
10"
2 10
4x
6=100
Training Steps
Training Steps
MATH500
GPQA Diamond
ORZ-I.58
ORZ-L.58
ORZ-8.5B
ORZ-8.88
Accuracy(%)
40-
Accuracy(% (%))
10
30
4x10
5x110
10" Training Steps
2x100
4x100
6x10
10" Training Steps
1x
Figure Evaluation performance of Open-Reasoner-Zero (0.50, 1.5B). We report the average accuracy on the benchmark dataset for each question with 16 responses.
our 32B model itself to identify challenging and sigh-quality prompts for this annealing stage. We pinpoint 13k particularly difficult prompts, defined those where the model achieves fewer than correct answers out of total of 64 attempts during the first 1100 steps of training. These identified prompts then selectively used in final training stage of 100 additional steps and apply linear learning rate decay schedule, reducing to x 10-7. This targeted training phase is explicitly designed enhance the model's capability on more complex reasoning tasks.
For the training of the DIZ-RI--Ditill--wenn DeepSeek-R1-PD -eeeee-RR--isssll--ee------ 14B model. We utilize the mined 13k difficult prompts training data. All other hyperparameters follow the basic configuration of the ORZ model family. The reported results correspond to the checkpoint 300 training iterations.
model, we initialized its weights from the
C Additional Analyses and Experiments
C.1 More Analysis for Critic and Advantage Estimation
As noted in the main text, vanilla GRPO often suffers from significant training instability, phe- nomenon also observed in many community implementations. This instability typically manifests deterioration in generation quality midway through training, with models tending produce repetitive or incoherent text. Our large-scale experimental validation strongly corroborates these observations and highlights the superior training stability of PPO compared GRPO, as shown in Figure g
C.2 Ablation on Data Curation
Based our analysis of data quality issues, we conduct comprehensive ablation studies evaluate how different data curation strategies affect model training stability and performance. Motivated by OpenR1's finding [13] that SFT performance degradation on Chinese subsets was due simpler question patterns, we experiment with two data curation approaches: using English-on/ data versus using both English and Chinese data. As shown Figure the English-only dataset yields superior training stability and final model performance.
15

---

10'
PPO
PPO
PPO
0.8
Truncate Rate
10"
Repeat
Rewards
0.5
10
10
0.4
10
0.2
0.0
10
300
Tratning Stops
Steps
Steps
Figure 8: PPO vs. GRPO stability in the ORZ-7B setting. GRPO empirically demonstrates severe training instability around 240 training steps: its reward suddenly destabilizes, and responses degen- erate both Truncate Rate and Average Repeat Score surge 1.0. In contrast, PPO maintains stable rewards and low Truncate/Repeat Scores throughout.
Reward During Training
Response Length During Training
0.9k
0.5
Response Length
0.8k
Reward
0.2
Open Reasoner Zero 57k
Open Reasoner Zero 57k
0.1
0.5k
Open Reasoner Zero 57k+CN
Open Reasoner Zero 57k+CN
20
40
80
100
20
40
60
80
160
Training Steps
Training Steps
Figure 9: Data Curation Ablation. CN represents Chinese data and EN represents English data. English-only dataset yields superior training stability and final model performance.
D Derivation and Code for PPO with GAE(? = 1,A=1) 1)
This section provides detailed derivation for the GAE the specific case where y 1 and À We substitute = 1 and X = into the general GAE formulation (defined in the main text). Recall that 8.44 r=+h + [Vo(#+++11) Vo(s++)). With 7 1, this becomes deta T*+
Vo (5t+kt+1) - Vo($++)) Thus:
T-1-
A AGAE(1=LA=1)
Σ
(1 1) S54k
k=
T-t-1
(re*k + Vol+442+1) Vo(s441))
T-1-1
T-1-1
Σ
retk +
Σ
(Vo(6+4442) - (o(
(6)
K=
0=0
R Vi((s)
(7)
The step from 6 to 7 follows because: (i) the sum of rewards T-1-1 ri+h equals the single terminal reward R for the trajectory, as intermediate rewards are zero; and (ii) the second sum
ΣI-I1 (Vo(+++441) (o(()+)) is telescoping series that evaluates Vo(ST) Vo(st), where ST is the terminal state, and (a()) 0.
16

---

Now we derive the simplified form for the value target V.tagst As defined in the main text, we have:
yaugee A AGAE(L.) + VV(ss)
(R Vo(ss)) Vo(s))
(8)
Substituting this simplified target V main text):
Janget
R into the general value loss formulation (defined in the
Tvalue (Ø) BT m
Σ (Vo(s)) - R)²
(9)
We also provide detailed algorithm implementation
Algorithm 1 PPO with GAE(y 1)
Require: Initial policy parameters 0u, initial value parameters Φu, prompt dataset D. Require: Hyperparameters: clip range c, trajectories per prompt n, minibatch size M. I: Initialize policy Ny <- #gg value function V Voo.
2: Initialize Oola <- 0o Oold f- Ø0-
3: for iteration 1, 1, do
4: Initialize experience buffer B <-
5:
Rollout Phase
6: Sample batch of prompts {qj from D. for all prompts qj in the batch do Generate trajectory т = (So. Ao. ...,, ST -, 0r-1) using policy Tow-
7:
8:
9:
Compute terminal reward R E {0. 13 for
10:
Compute value estimate V/oldd = Venn
11:
Compute advantage Å, R Viold
Using 1,
Store tuple (r, log Te.. (a.(st). ) R. As) in buffer B.
13:
end for
Update Phase > Update critic model
15:
for all minibatches (r, R) from B do
17:
Compute current critic value (s(). Lvr(Ø) (V)(s) R)².
18:
19:
Backward and update Φ
20:
end for
Update policy model
22:
for all minibatches (r, log (a)s, Rt A) from do
23:
Compute current policy log- probability log (a(a))).
24:
Calculate probability ratio p(0) = exp(log no (a
log Tatu(a|s))
LCLIP (0) = min clip((p()) -
26:
Backward and update 0 end for
27:
28:
Update old parameters: lus +- 0, dold t Φ.
29:
end for
Ensure: Final policy parameters 0.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section as introduction reflects main contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses the limitation our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Our paper builds upon established algorithms (PPO and GAE) whose theoreti- cal foundations and proofs are well-documented in prior literature. We do not introduce new theoretical results requiring formal proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We release all our training resources, including training code, curated data, detailed settings, and various model weights.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release all our training resources, including training code, curated data, detailed settings, and various model weights.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We release all our training resources, including training code, curated data, detailed settings, and various model weights.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our main results are evaluated using the avg@ 16 metric, which aggregates performance across multiple runs. This statistical reporting ensures the reliability and reproducibility of our findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: Our experiments were conducted across various computational environments due to resource constraints. The performance of our models can be verified through our released checkpoints rather than through exact hardware specifications.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///waapppppppoo/oo//cccaaaa
Answer: [Yes]
Justification:
This work aligns with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work does not have potential negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All code and data are following licenses properly. And all related works cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release all training resources, including training code, curated data, detailed settings, and various model weights.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects in this
paper.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not involve crowdsourcing research with human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do not use L.M for any core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
24


Reviews:
[{"review_id": "1H12WxWkRH", "reviewer": "Reviewer_am8n", "summary": "The paper builds on insights from DeepSeek-R1-Zero and introduces an open source and scalabe approach for large scale reinforcement learning from base models. The approach proposed in the work is minimal and demonstrates that vanilla PPO with GAE and simple rule based rewards is sufficient for good performance. The proposed recipe also avoids KL regularization, which simplifies it further. The strengths of the work include open source code, hyper-parameters and training data, instead from model weights alone, and the simplicity of the approach. The main weakness is limited demonstration on math and reasoning benchmarks, and overlap with previous works which have attempted to reproduce DeepSeek-R1-Zero.", "questions": "Have you experimented on non Qwen models to explore generalizability?\nWhat modifications would be needed to generalize to code and other domains?\nAre there any emergent patterns that are interesting to highlight which haven't been shown by previous works?\nDid you use multilingual data in your training set?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The technique is simple and scalable, doing away with complexities like KL penalty and complex reward function design.\n2. The authors have open sourced code, weights and also training data, which fosters reproducibility.\n3. The results demonstrate that their recipe is scalable and also more efficient compared to previous works - achieving strong performance with fewer training steps. \n\nWeaknesses:\n1. Narrow focus on reasoning and math tasks.\n2. While the empirical results are solid, the authors have not explored from a theoretical standpoint why the minimalist approach works so well, and whether it is really superior to previous approaches.", "was_revised": true, "final_justification": "The paper is well suited for the conference, and the results presented here would be of interest to the wider community. I recommend acceptance.", "rebuttal": "Thanks to the reviewer for the careful reading and constructive feedback. We appreciate the acknowledgement of our open-source philosophy and the paper’s practical value, and we address the raised points below.\n\n### Extension to domains beyond math and reasoning\nOur recipe transfers to new domains (e.g. code, puzzle, instruction-following dialogues) without any modifications. Note that the original submission has already included puzzle-style tasks such as table-join, table-reformat, zebra, and connections alongside math problems. To further demonstrate generalizability, we conducted additional experiments on Qwen-2.5-7B base model with a mixed-domain training distribution (math:code:puzzle:instruction-following = 40%:40%:15%:5%) using the identical ORZ recipe. Training progressed smoothly with consistent increases in both reward and response length. The table below compares benchmark performance between this mixed-domain model and our math-focused model across multiple evaluations. \n| init_model | training domain | train_iter | AIME24 | AIME25 | MATH500 | GPQA_DIA | LCB_V5 | LCB_V6 | IFEVAL | LIVEBENCH | MMLU_TEST | MMLU_PRO | NAIVE_AVG |\n|------------|-----------------|------------|--------|--------|---------|----------|--------|--------|--------|-----------|-----------|----------|-----------|\n| Qwen2.5-7B-Instruct | - | - | 10.10% | 6.04% | 73.80% | 35.70% | 12.70% | 17.14% | 55.82% | 28.83% | 72.66% | 50.91% | 36.37% |\n| Qwen2.5-7B-Base | Math+Code+Puzzle+IF | 500 | 16.46% | 14.69% | 79.00% | 35.89% | 20.79% | 23.14% | 66.27% | 40.51% | 73.81% | 54.06% | 42.46% |\n| Qwen2.5-7B-Base | Math | 500 | 16.20% | 13.13% | 79.60% | 36.71% | 1.30% | 1.72% | 40.27% | 33.99% | 74.67% | 56.08% | 35.37% |\n| Qwen2.5-7B-Base | Math | 700 | 17.50% | 15.00% | 81.00% | 37.85% | 2.15% | 2.29% | 37.62% | 35.97% | 75.04% | 57.80% | 36.22% |\n\nThe enlarged task mix delivers substantial gains beyond pure mathematics: IFEVAL, LIVEBENCH, and the logic/code-heavy LCB suites improve sharply, yet the minimalist PPO recipe remains untouched, clear evidence of stronger reasoning transfer.\nThis confirms that our recipe generalizes seamlessly to diverse domains without any algorithmic changes.\n\n### Generalization to non-Qwen model families\nWe successfully applied our ORZ recipe to Llama-3.1-8B base model with minimal adaptation (in the above mixed-domain training setting, only adjusting the learning rate from 1e-6 to 5e-7). Training metrics show steady increases in both reward and response length, as shown in the table below:\n| train_iters | 0 | 100 | 200 | 300 | 400 | 500 |\n|-------------|---|-----|-----|-----|-----|-----|\n| response_len | 458 | 230 | 307 | 356 | 546 | 496 |\n| reward | 1.10% | 3.49% | 8.05% | 9.20% | 11.91% | 12.31% |\n\nOffline evaluations show consistent performance improvements across benchmarks as table below. The ORZ-Llama3.1-8B model in the table below corresponds to the checkpoint at iteration 500 (iter500) from applying our ORZ recipe to the Llama3.1-8B-Base model. The NAIVE_AVG is benchmark level of arithmetic mean of all listed benchmarks. \n| model | AIME24 | AIME25 | MATH500 | GPQA_DIA | IFEVAL | LIVEBENCH | MMLU_TEST | NAIVE_AVG |\n|-------|--------|--------|---------|----------|--------|-----------|-----------|-----------|\n| Llama-3.1-8B-Instruct | 2.71% | 0.42% | 39.60% | 27.68% | 54.21% | 19.36% | 62.26% | 29.46% |\n| Llama3.1-8B-Base | 0.16% | 0.00% | 0.29% | 14.29% | 15.96% | 3.56% | 27.83% | 8.87% |\n| ORZ-Llama3.1-8B | 0.52% | 0.16% | 21.93% | 26.70% | 66.96% | 27.17% | 61.21% | 29.24% |\n\n### Emergent patterns. \nWe observe two interesting emergent patterns not well documented previously. First, during the final phases of ORZ-32B training, we observe the emergence of a distinctive reflection pattern where the model spontaneously revisits the problem statement from the beginning when encountering difficulties. Unlike local reflection patterns (e.g., correcting a single step), this global revisiting behavior allows the model to escape local minima by fundamentally reconsidering its approach. Second, while our training focuses on mathematical reasoning, ORZ-32B demonstrates unexpected transfer to logical reasoning in unrelated domains. Without explicit training, the model exhibits structured reasoning patterns in tasks such as academic writing, poetry composing, and general domain quesion and answering. \n\n### Multilingual data\nOur training data includes multilingual content, with approximately 5% Chinese material from CN K12 datasets in Numina.\n\n### Theoretical perspective\nWe acknowledge the reviewer's interest in theoretical foundations. Crafting a rigorous framework that explains why our minimalist recipe scales reinforcement learning so effectively is an important—but substantial—undertaking. A full treatment would require dedicated analysis beyond the present submission, and we view it as a promising direction for future work.\n\nWe will include expanded ablation results and full training-curves in the revision.  We hope these additions address the reviewer’s concerns and clarify the generality, empirical strength, and transparency of Open-Reasoner-Zero."}, {"review_id": "ZvG7SCSPzO", "reviewer": "Reviewer_5KN5", "summary": "The authors provide an open-source reproduction of large-scale RL reasoning results, matching the performance of DeepSeek-R1-Zero over four benchmarks. They also provide analysis over design decisions and simplifications the method which reduce the number of training steps.", "questions": "Code release would be important.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Provides open-source experimental methodology to match the performance of a well-known agent. This provides a clear value to the community, in terms of reproducibility and providing a clear framework for further iteration.\n- The authors simplify aspects/ablate over the original approach, which meaningfully reduces the number of training steps and may provide valuable insights for practitioners. \n\nWeaknesses\n- Organization: In general, while I found the paper clearly written, I do think the paper suffers from poor structure and organization of results. I often had to scroll back and forth to find relevant figures, or explanations of certain results. For example, while Figure 3 is clearly relevant to the design choices in 2.2, it only gets explained 3.2, almost two pages later. \n- Although it’s interesting that PPO allows for better advantage estimation on repetitive tokens, the authors do not present any explanation as to why this might be the case. While this observation is clearly useful, understanding this phenomenon might allow for better algorithm design of future work and would strengthen the paper. \n- It should not go unmentioned that the novelty is low, given it is mainly centered around a reproduction of prior work. Regardless, I think the open-sourcing and simplifications outweigh this negative. \n- No code included with submission.", "was_revised": true, "final_justification": "The paper offers a valuable contribution to the field in terms of open-sourcing. The authors have promised to correct my issues with organization and to actually release the code. I think there is an opportunity for further strengthening the paper with additional analysis and experiments, but this should not be a blocker for acceptance.", "rebuttal": "We sincerely thank the reviewer for the positive assessment and valuable feedback. We appreciate the recognition of our work's contribution to the community through open-source methodology and meaningful simplifications. We address each concern below.\n### Organization and Structure\nThanks for your valuable feedback on the structure and organization of our results. We fully agree that improving the alignment between figures, their explanations, and relevant sections is crucial for readability. We will reorganize the manuscript in the camera-ready version to ensure better flow between related content. Specifically, we will move the explanation of Figure 3 closer to Section 2.2 where the design choices are introduced, eliminating the need for readers to scroll between sections. We will also improve cross-referencing throughout the paper to enhance readability.\n\n### Understanding PPO's Superior Advantage Estimation on Repetitive Tokens\nWe appreciate the reviewer's interest in understanding why PPO achieve better repetitive token advantage estimation. Our analysis suggests the following mechanism:\nDuring training, we find that repetitive token sequences reliably signal wrong reasoning paths and failed solutions. The value function quickly learns to give such states low scores, so when the model slips into repetition, the critic usually flags the state with negative advantages and pushes the policy out of the lower value states. Critic-free methods like GRPO lack this salient feedback and miss the cue. \n\n### Novelty\nWhile the core algorithm is purposefully minimal, the contribution is the first fully open-source, greatly simplified, efficient reproduction of DeepSeek-R1-Zero. Matching its accuracy with 10× fewer steps, dissecting which components are dispensable, and providing a reusable training harness collectively advance the field's understanding and accessibility of reasoning-oriented RL training. \n\n### Full open-source release. \nWe understand the reviewer's concern about code availability. Unfortunately, NeurIPS 2024 submission guidelines prohibit  sharing URLs or de-anonymizing information during the review period. \nTo ensure reproducibility,  we commit to releasing our complete codebase, training data, model checkpoints, and detailed reproduction instructions immediately after double-blind review period. Our submission includes a clear reproducibility statement affirming this commitment.\n\nWe hope these clarifications address the reviewer's concerns and reinforce the value of our contributions to the community."}, {"review_id": "QIXPEuVlXD", "reviewer": "Reviewer_wfie", "summary": "The paper introduces Open-Reasoner-Zero-32B (ORZ-32B), an open-source large-scale reasoning model trained with PPO and Generalized Advantage Estimation (GAE) using Qwen 2.5-32B base model. By dissecting the contribution of individual algorithmic components and their hyperparameters, the authors show that vanilla PPO, when paired with their recommended hyperparameter settings, outperforms the GRPO variant of Guo et al. (2025). Despite operating with only 10 % of DeepSeek-R1-Zero’s total training budget, ORZ-32B achieves higher average scores on a suite of mathematical and general-reasoning benchmarks.\n\nDaya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi ,et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,2025.", "questions": "1. Beyond benchmark accuracy, does ORZ-32B exhibit issues such as poor readability, mixed-language outputs, or over-compression?\n2. Have you identified any persistent weaknesses (e.g., hallucination on domain-specific tasks, sensitivity to prompt length) in ORZ-32B?", "limitations": "The authors have adequately addressed tthe limitations of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- Thorough experimental design with ablation studies that isolate the impact of each component.\n- Transparent reporting of hyperparameters, training budget, and evaluation setup.\n- Writing is concise and well structured; figures and tables are easy to follow.\n- Provides the first comprehensive head-to-head comparison of vanilla PPO versus GRPO in this setting.\n### Weaknesses\n-  Limited analysis of downstream failure modes or undesirable behaviors beyond benchmark scores.\n- The proposed ORZ pipeline resembes prior PPO-finetuning pipelines; novelty lies more in systematic analysis than in algorithmic invention.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the encouraging evaluation and for highlighting areas where a deeper analysis could strengthen the paper. We respond to each point below.\n\n### General-scene quality\nTo go beyond reasoning benchmark scores and examine the quality of answer part (not the thinking part) of model output, we now include results on ArenaHard v2, an open-ended dialogue benchmark that exhibits strong correlation and separability with LMArena “judge-vote” rankings. Notably, ORZ-32B outperforms the strong Qwen-2.5-32B-Instruct baseline on ArenaHard v2—by +0.7 % in overall win rate (6.50 % vs 5.81 %) and an even larger +3.4 % in style-control win rate (11.40% vs 7.96%). These margins confirm that ORZ-32B generates generally fluent, well-controlled dialogue without mixed-language artifacts or over-compression in the answer part of model's output. \n| model | win_rate | style_control_win_rate |\n|-------|----------|------------------------|\n| Qwen2.5-32B-Instruct | 5.81% | 7.96% |\n| ORZ-32B | 6.50% | 11.40% |\n\nHowever, after careful investigation, we find that ORZ-32B develops highly flexible patterns in the thinking part of the model's output. The model sometimes exhibits free-form thinking behaviors including: (1) dynamic use of multiple thinking markers (e.g., nested <think></think> blocks), and (2) spontaneous code-switching between languages when conducting reasoning. \n\n### Novelty of the recipe\nOur pipeline, while retaining the simplicity of vanilla PPO, introduces three scale-critical components that depart markedly from prior PPO implementations—and these deviations are uniquely critical for large-scale reasoning-oriented RL.\n\n### Hallucination and prompt-length sensitivity\nWe benchmark ORZ-32B on MATHIF, which measures instruction-following in mathematical reasoning.\nCorrectness is reported without and with explicit constraints (w/o const. vs. w/ const.); the final column gives the relative change when constraints are applied.\nBy contrast, all other models in the table lose 4–40% under the same test, showing that ORZ’s alignment maintains and even improves correctness where others falter.\n| model | Correctness w/o const. | Correctness w/ const. | Diff.(%) |\n|-------|------------------------|----------------------|----------|\n| DeepSeek-R1-Distill-Qwen-7B | 65.24 | 48.57 | -25.55 |\n| DeepSeek-R1-Distill-Llama-8B | 59.76 | 36.43 | -39.04 |\n| **Open-Reasoner-Zero-7B** | 52.86 | 51.9 | -1.82 |\n| Qwen3-32B | 72.62 | 70 | -3.61 |\n| DeepSeek-R1-Distill-Qwen-32B | 71.43 | 57.62 | -19.33 |\n| DeepSeek-R1-Distill-Llama-70B | 71.19 | 54.05 | -24.08 |\n| **Open-Reasoner-Zero-32B** | 65.48 | 67.62 | +3.27 |\n\nWe hope these additions address the reviewer’s questions about broader behaviour, clarify the source of our efficiency gains, and further demonstrate ORZ-32B’s value to the field."}, {"review_id": "DIu2StefGD", "reviewer": "Reviewer_2p6y", "summary": "This paper presents Open-Reasoner-Zero, an open-source implementation of large-scale reasoning-oriented reinforcement learning (RL) directly on base LLMs. The work advocates for a minimalist pipeline based on vanilla PPO with simple rule-based rewards and demonstrates scalability across models (0.5B to 32B) and benchmarks (AIME, MATH500, GPQA). The authors compare their method against DeepSeek-R1-Zero and QwQ models, claiming superior performance with significantly fewer training steps.", "questions": "1. Can you provide results or qualitative insights on whether similar improvements are seen when applying your pipeline to non-Qwen base models?\n\n2. Have you tested or considered other RL algorithms to verify if PPO is indeed optimal for this setting?\n\n3. What is the expected failure mode or limitation of this minimalist RL approach when scaling to other domains or tasks beyond reasoning?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n\n1. Open-source contribution: The release of code, weights, and data contributes to the reproducibility and accessibility of RL for LLMs, enabling further experimentation and community participation.\n\n2. Empirical findings on minimalist design: The authors provide insightful observations that vanilla PPO without KL regularization and with basic reward functions can scale well, which may simplify the design of future RL pipelines for LLMs.\n\n3. Efficient performance scaling: The reported results show strong gains in benchmark performance and response length with significantly fewer steps, indicating the practical value of the proposed pipeline.\n\nWeaknesses:\n\n1. Limited novelty in methodology: The paper primarily repackages existing techniques (e.g., vanilla PPO, GAE, rule-based rewards) and applies them to a specific training setting. There is no substantive methodological innovation in RL algorithms, reward modeling, or data curation that would be considered novel in the NeurIPS community.\n\n2. Evaluation restricted to Qwen-based models: The experiments focus solely on Qwen2.5 models. It remains unclear whether the observed benefits and scalability extend to other foundation models (e.g., LLaMA, Mistral, or GPT-like architectures). Broader evaluation would better support the claimed generality of the pipeline.\n\n3. No comparison of alternative RL methods: PPO is the sole algorithm evaluated. The authors do not justify why PPO is preferable over other modern RL methods (e.g., TRPO, REINFORCE++, DPO, P3O). It would strengthen the contribution to examine whether PPO’s stability is unique or if similar trends arise with other approaches.", "was_revised": true, "final_justification": "The response from the author has partially address my previous concerns (e.g., evaluation on non-Qwen models and coding benchmark, discussion on other RL approach). Even though the novelty for the broader LLM research might be limited, the contribution of open sourcing outweighs such concern, and I would adjust the rating based on these considerations.", "rebuttal": "Thanks to the reviewer for the detailed assessment and for flagging areas where broader evidence would help.\n### Novelty\nWhile the core algorithm is purposefully minimal, the contribution is the first fully open and efficient reproduction of DeepSeek-R1-Zero. Matching its accuracy with 10x fewer steps, dissecting which components are dispensable, and providing a reusable training harness help the field move beyond opaque, resource-intensive recipes.\nOur pipeline, while retaining the simplicity of vanilla PPO, introduces three scaling-critical components that depart markedly from originally widely-adpoted PPO implementations—and these deviations are uniquely critical for large-scale reasoning-oriented RL.\n\n### Extension to domains beyond math and reasoning\nOur recipe transfers to new domains (e.g. code, puzzle, instruction-following dialogues) without any modifications. Note that the original submission has already included puzzle-style tasks such as table-join, table-reformat, zebra, and connections alongside math problems. To further demonstrate generalizability, we conducted additional experiments on Qwen-2.5-7B base model with a mixed-domain training distribution (math:code:puzzle:instruction-following = 40%:40%:15%:5%) using the identical ORZ recipe. Training progressed smoothly with consistent increases in both reward and response length. The table below compare benchmark performance between this mixed-domain model and our math-focused model across multiple evaluations, where the final column NAIVE_AVG is benchmark level arithmetic average for all listed benchmarks.\n| init_model | training domain | train_iter | AIME24 | AIME25 | MATH500 | GPQA_DIA | LCB_V5 | LCB_V6 | IFEVAL | LIVEBENCH | MMLU_TEST | MMLU_PRO | NAIVE_AVG |\n|------------|-----------------|------------|--------|--------|---------|----------|--------|--------|--------|-----------|-----------|----------|-----------|\n| Qwen2.5-7B-Instruct | - | - | 10.10% | 6.04% | 73.80% | 35.70% | 12.70% | 17.14% | 55.82% | 28.83% | 72.66% | 50.91% | 36.37% |\n| Qwen2.5-7B-Base | Math+Code+Puzzle+IF | 500 | 16.46% | 14.69% | 79.00% | 35.89% | 20.79% | 23.14% | 66.27% | 40.51% | 73.81% | 54.06% | 42.46% |\n| Qwen2.5-7B-Base | Math | 500 | 16.20% | 13.13% | 79.60% | 36.71% | 1.30% | 1.72% | 40.27% | 33.99% | 74.67% | 56.08% | 35.37% |\n| Qwen2.5-7B-Base | Math | 700 | 17.50% | 15.00% | 81.00% | 37.85% | 2.15% | 2.29% | 37.62% | 35.97% | 75.04% | 57.80% | 36.22% |\n\nThe enlarged task mix delivers substantial gains beyond pure mathematics: IFEVAL, LIVEBENCH, and the logic/code-heavy LCB suites improve sharply, yet the minimalist PPO recipe remains untouched, clear evidence of stronger reasoning transfer.\nThis confirms that our recipe generalizes seamlessly to diverse domains without any algorithmic changes.\n\n### Generalization to non-Qwen model families\nWe successfully applied our ORZ recipe to Llama-3.1-8B base model with minimal adaptation (in the above mentioned mixed domain training setting, only adjusting the learning rate from 1e-6 to 5e-7). Training metrics show stable progression with both reward and response length increasing steadily throughout training, as shown in the table below:\n| train_iters | 0 | 100 | 200 | 300 | 400 | 500 |\n|-------------|---|-----|-----|-----|-----|-----|\n| response_len | 458 | 230 | 307 | 356 | 546 | 496 |\n| reward | 1.10% | 3.49% | 8.05% | 9.20% | 11.91% | 12.31% |\n\nOffline evaluations also show significant performance improvements across benchmarks as table below. The ORZ-Llama3.1-8B model in the table below corresponds to the checkpoint at iteration 500 (iter500) from applying our ORZ recipe to the Llama3.1-8B-Base model. The NAIVE_AVG is benchmark level of arithmetic mean of all listed benchmarks. \n| model | AIME24 | AIME25 | MATH500 | GPQA_DIA | IFEVAL | LIVEBENCH | MMLU_TEST | NAIVE_AVG |\n|-------|--------|--------|---------|----------|--------|-----------|-----------|-----------|\n| Llama-3.1-8B-Instruct | 2.71% | 0.42% | 39.60% | 27.68% | 54.21% | 19.36% | 62.26% | 29.46% |\n| Llama3.1-8B-Base | 0.16% | 0.00% | 0.29% | 14.29% | 15.96% | 3.56% | 27.83% | 8.87% |\n| ORZ-Llama3.1-8B | 0.52% | 0.16% | 21.93% | 26.70% | 66.96% | 27.17% | 61.21% | 29.24% |\n\n### Alternative RL algorithms.\nWe have already benchmarked against GRPO—the most widely used variant in this setting—and found PPO to be both more sample-efficient and more stable. TRPO is seldom applied to LLMs because its second-order updates are computationally expensive, while REINFORCE++, a critic-free method akin to GRPO, shows similar convergence behaviors without offering clear advantages.\n\nWe hope these additions clarify the generality of the approach, justify our focus on PPO, and acknowledge its current limits."}, {"review_id": "XFfZMsSqQr", "reviewer": "Reviewer_Bk7x", "summary": "This paper proposes an open-source framework that enables DeepSeek-R1-Zero-style scaling in both performance and response length, using PPO combined with a simple exact-matching rule-based reward. The method involves PPO with multiple rollouts per prompt, no KL regularization, careful data curation, and GAE with γ=1 and λ=1 (collapses to the Monte-Carlo return minus value baseline). Experiments on mathematical reasoning tasks demonstrate that this PPO setup achieves efficient and stable training dynamics, outperforming GRPO and DAPO baselines.", "questions": "1. The increase in response length and reward does not necessarily indicate improved reasoning ability, it may also result from reward hacking. How do the authors rule out this possibility?\n2. In Fig.1 and Fig.2, the training steps appears to scale with model size. What is the rationale behind this? What would happen if training continued further?\n3. Why does ORZ-32B exhibit noticeable fluctuations in response length (Fig.2), while smaller models (e.g., ORZ-7B, ORZ-1.5B, ORZ-0.5B) do not?\n4. How is the threshold between low-value and high-value regions defined (Fig.5)?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strength\nThis work introduces a valuable framework with insightful empirical observations, which could be impactful for the LLM reasoning and RL post-training communities. The analysis of the critic and advantage estimation is particularly helpful, showing the clear benefits of using a learnable critic.\n\n### Weakness\nThe comparisons and analysis of ORZ primarily focus on performance and response length curves. However, merely presenting the curves does not rule out the influence of reward hacking, especially after the KL penalty has been removed. Providing additional qualitative comparisons of the responses can make the paper more convincing.", "was_revised": true, "final_justification": "My concerns have been addressed. I will keep my original score, which supports acceptance.", "rebuttal": "We sincerely thank the reviewer for the positive assessment and thoughtful questions about our work. We appreciate the recognition of our framework's value and the insightful analysis on critic and advantage estimation. We address each concern below.\n### On Reward Hacking Concerns\nWe appreciate the reviewer's concern about potential reward hacking. However, our experimental setup has taken this into consideration, and greatly reduce the potential space for reward hacking, fundamentally different from RLHF settings where this issue commonly arises: \n- We focus exclusively on reasoning tasks with verifiable ground truth answers\n- Our reward function is a simple exact-match criterion against these ground truths\n- Unlike RLHF with learned reward models that can be exploited, our binary exact-match reward cannot be \"gamed\" through superficial response modifications\nThe traditional reward hacking in RLHF manifests when models optimize for high reward model scores while producing qualitatively worse outputs. In our setting, achieving higher rewards directly corresponds to solving more problems correctly. \n\n### Evidence of Genuine Reasoning Improvement\nBeyond the reward and response length curves, we want to share more evidence that ORZ develops genuine reasoning capabilities. First, the model shows consistent performance improvements across a wide range of held-out evaluation sets, indicating true generalized intelligence improvement. We also conducted careful manual inspection of model outputs throughout training. Here's a representative example from ORZ-32B: \n- Question: Find the greatest integer value of $b$ for which the expression $\\frac{9x^3+4x^2+11x+7}{x^2+bx+8}$ has a domain of all real numbers. \n- ORZ-32B's approach (summarized): The model correctly identifies that the domain constraint requires the denominator to never equal zero → formulates this as requiring the discriminant $b^2-32<0$ → solves to get $|b|<\\sqrt{32}=4\\sqrt{2}\\approx 5.656$ → systematically checks boundary cases to conclude $b=5$ is the maximum integer value → performs multiple rounds of self-reflection to verify the answer. \n\nThis example illustrates the model's ability to: (i) correctly interpret mathematical constraints, (ii) apply appropriate algebraic techniques, (iii) perform systematic case analysis, and (iv) engage in self-verification. These are hallmarks of genuine mathematical reasoning, not reward exploitation. We will include additional qualitative examples and more detailed analysis in the revised version.\n\n### Training Duration for Different Model Scales\nThe reviewer correctly observes that training steps are different across model sizes. We clarify that this was not a deliberate design choice but rather a practical constraint: experiments on smaller scale models are not of top priority in our plan, and it gets terminated early due to limited computation and time constraints. We are currently resuming experiments to explore longer training horizons for these smaller models and will update the paper with extended training curves.\n\n### Response Length Fluctuations in ORZ-32B\nThis is indeed an interesting phenomenon that we've investigated further since submission. Our additional experiments reveal that learning rate tuning is the key factor:\n- When we reduced the 32B model's learning rate from 1e-6 to 5e-7, length growth became much more stable.\n- Conversely, increasing the 7B model's learning rate from 1e-6 to 2e-6 induced similar fluctuations.\nWe will include these findings in the appendix of the revised version.\n\n### Clarification on Figure 5 Visualization\nWe apologize for any confusion in the figure caption. We actually use a continuous color map rather than discrete thresholds in text visualization: values closer to 0 appear redder, and values closer to 1 appear greener. And as you can see in the original submission's Figure 5, the critic learns to assign lower values to repetitive/degenerate text and higher values to meaningful reasoning steps. We will revise the caption to make this color mapping logic clearer."}]

Author rebuttal:

Meta-review: The paper is an insightful reproduction of DeepSeek-R1-Zero. The reviewers and I unanimously agree it is a value contribution to the community providing a clear study of the key design decisions and hyper-parameter choices behind attaining these results with a variation on PPO.
Decision: Accept (poster)