VerifyBench: Benchmarking Reference-based Reward Systems for Large
  Language Models
Authors: name: Yuchen Yan; name: Jin Jiang; name: Zhenbang Ren; name: Yijun Li; name: Xudong Cai; name: Yang Liu; name: Xin Xu; name: Mengdi Zhang; name: Jian Shao; name: Yongliang Shen; name: Jun Xiao; name: Yueting Zhuang
Published: 2025-05-21 17:54:43

Abstract: Large reasoning models such as OpenAI o1 and DeepSeek-R1 have achieved
remarkable performance in the domain of reasoning. A key component of their
training is the incorporation of verifiable rewards within reinforcement
learning (RL). However, existing reward benchmarks do not evaluate
reference-based reward systems, leaving researchers with limited understanding
of the accuracy of verifiers used in RL. In this paper, we introduce two
benchmarks, VerifyBench and VerifyBench-Hard, designed to assess the
performance of reference-based reward systems. These benchmarks are constructed
through meticulous data collection and curation, followed by careful human
annotation to ensure high quality. Current models still show considerable room
for improvement on both VerifyBench and VerifyBench-Hard, especially
smaller-scale models. Furthermore, we conduct a thorough and comprehensive
analysis of evaluation results, offering insights for understanding and
developing reference-based reward systems. Our proposed benchmarks serve as
effective tools for guiding the development of verifier accuracy and the
reasoning capabilities of models trained via RL in reasoning tasks.