An Empirical Study on Eliciting and Improving R1-like Reasoning Models
Authors: name: Zhipeng Chen; name: Yingqian Min; name: Beichen Zhang; name: Jie Chen; name: Jinhao Jiang; name: Daixuan Cheng; name: Wayne Xin Zhao; name: Zheng Liu; name: Xu Miao; name: Yang Lu; name: Lei Fang; name: Zhongyuan Wang; name: Ji-Rong Wen
Published: 2025-03-06 15:34:27

Abstract: In this report, we present the third technical report on the development of
slow-thinking models as part of the STILL project. As the technical pathway
becomes clearer, scaling RL training has become a central technique for
implementing such reasoning models. We systematically experiment with and
document the effects of various factors influencing RL training, conducting
experiments on both base models and fine-tuned models. Specifically, we
demonstrate that our RL training approach consistently improves the Qwen2.5-32B
base models, enhancing both response length and test accuracy. Furthermore, we
show that even when a model like DeepSeek-R1-Distill-Qwen-1.5B has already
achieved a high performance level, it can be further refined through RL
training, reaching an accuracy of 39.33% on AIME 2024. Beyond RL training, we
also explore the use of tool manipulation, finding that it significantly boosts
the reasoning performance of large reasoning models. This approach achieves a
remarkable accuracy of 86.67% with greedy search on AIME 2024, underscoring its
effectiveness in enhancing model capabilities. We release our resources at the
STILL project website: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.