DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open
  Language Models
Authors: name: Zhihong Shao; name: Peiyi Wang; name: Qihao Zhu; name: Runxin Xu; name: Junxiao Song; name: Mingchuan Zhang; name: Y. K. Li; name: Y. Wu; name: Daya Guo
Published: 2024-02-05 18:55:32

Abstract: Mathematical reasoning poses a significant challenge for language models due
to its complex and structured nature. In this paper, we introduce DeepSeekMath
7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B
math-related tokens sourced from Common Crawl, together with natural language
and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the
competition-level MATH benchmark without relying on external toolkits and
voting techniques, approaching the performance level of Gemini-Ultra and GPT-4.
Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH.
The mathematical reasoning capability of DeepSeekMath is attributed to two key
factors: First, we harness the significant potential of publicly available web
data through a meticulously engineered data selection pipeline. Second, we
introduce Group Relative Policy Optimization (GRPO), a variant of Proximal
Policy Optimization (PPO), that enhances mathematical reasoning abilities while
concurrently optimizing the memory usage of PPO.