Pitfalls of Rule- and Model-based Verifiers -- A Case Study on
  Mathematical Reasoning
Authors: name: Yuzhen Huang, name: Weihao Zeng, name: Xingshan Zeng, name: Qi Zhu, name: Junxian He

Trustworthy verifiers are essential for the success of reinforcement learning
with verifiable reward (RLVR), which is the core methodology behind various
large reasoning models such as DeepSeek-R1. In complex domains like
mathematical reasoning, rule-based verifiers have been widely adopted in
previous works to train strong reasoning models. However, the reliability of
these verifiers and their impact on the RL training process remain poorly
understood. In this work, we take mathematical reasoning as a case study and
conduct a comprehensive analysis of various verifiers in both static evaluation
and RL training scenarios. First, we find that current open-source rule-based
verifiers often fail to recognize equivalent answers presented in different
formats across multiple commonly used mathematical datasets, resulting in
non-negligible false negative rates. This limitation adversely affects RL
training performance and becomes more pronounced as the policy model gets
stronger. Subsequently, we investigate model-based verifiers as a potential
solution to address these limitations. While the static evaluation shows that
model-based verifiers achieve significantly higher verification accuracy,
further analysis and RL training results imply that they are highly susceptible
to hacking, where they misclassify certain patterns in responses as correct
(i.e., false positives). This vulnerability is exploited during policy model
optimization, leading to artificially inflated rewards. Our findings underscore
the unique risks inherent to both rule-based and model-based verifiers, aiming
to offer valuable insights to develop more robust reward systems in
reinforcement learning.