AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward
  Modeling
Authors: name: Zihan Liu; name: Yang Chen; name: Mohammad Shoeybi; name: Bryan Catanzaro; name: Wei Ping
Published: 2024-12-19 17:29:44

Abstract: In this paper, we introduce AceMath, a suite of frontier math models that
excel in solving complex math problems, along with highly effective reward
models capable of evaluating generated solutions and reliably identifying the
correct ones. To develop the instruction-tuned math models, we propose a
supervised fine-tuning (SFT) process that first achieves competitive
performance across general domains, followed by targeted fine-tuning for the
math domain using a carefully curated set of prompts and synthetically
generated responses. The resulting model, AceMath-72B-Instruct greatly
outperforms Qwen2.5-Math-72B-Instruct, GPT-4o and Claude-3.5 Sonnet. To develop
math-specialized reward model, we first construct AceMath-RewardBench, a
comprehensive and robust benchmark for evaluating math reward models across
diverse problems and difficulty levels. After that, we present a systematic
approach to build our math reward models. The resulting model, AceMath-72B-RM,
consistently outperforms state-of-the-art reward models. Furthermore, when
combining AceMath-72B-Instruct with AceMath-72B-RM, we achieve the highest
average rm@8 score across the math reasoning benchmarks. We will release model
weights, training data, and evaluation benchmarks at:
https://research.nvidia.com/labs/adlr/acemath