Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement
  Learning on the Base Model
Authors: name: Jingcheng Hu; name: Yinmin Zhang; name: Qi Han; name: Daxin Jiang; name: Xiangyu Zhang; name: Heung-Yeung Shum
Published: 2025-03-31 16:36:05

Abstract: We introduce Open-Reasoner-Zero, the first open source implementation of
large-scale reasoning-oriented RL training focusing on scalability, simplicity
and accessibility. Through extensive experiments, we demonstrate that a
minimalist approach, vanilla PPO with GAE (lambda=1, gamma=1) and
straightforward rule-based rewards, without any KL regularization, is
sufficient to scale up both response length and benchmark performance, similar
to the phenomenon observed in DeepSeek-R1-Zero. Using the same base model as
DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance on
AIME2024, MATH500, and the GPQA Diamond benchmark while demonstrating
remarkable efficiency -- requiring only a tenth of the training steps, compared
to DeepSeek-R1-Zero pipeline. In the spirit of open source, we release our
source code, parameter settings, training data, and model weights across
various sizes.