xVerify: Efficient Answer Verifier for Reasoning Model Evaluations
Authors: name: Ding Chen, name: Qingchen Yu, name: Pengyuan Wang, name: Wentao Zhang, name: Bo Tang, name: Feiyu Xiong, name: Xinchi Li, name: Minchuan Yang, name: Zhiyu Li

With the release of the o1 model by OpenAI, reasoning models adopting slow
thinking strategies have gradually emerged. As the responses generated by such
models often include complex reasoning, intermediate steps, and
self-reflection, existing evaluation methods are often inadequate. They
struggle to determine whether the LLM output is truly equivalent to the
reference answer, and also have difficulty identifying and extracting the final
answer from long, complex responses. To address this issue, we propose xVerify,
an efficient answer verifier for reasoning model evaluations. xVerify
demonstrates strong capability in equivalence judgment, enabling it to
effectively determine whether the answers produced by reasoning models are
equivalent to reference answers across various types of objective questions. To
train and evaluate xVerify, we construct the VAR dataset by collecting
question-answer pairs generated by multiple LLMs across various datasets,
leveraging multiple reasoning models and challenging evaluation sets designed
specifically for reasoning model assessment. A multi-round annotation process
is employed to ensure label accuracy. Based on the VAR dataset, we train
multiple xVerify models of different scales. In evaluation experiments
conducted on both the test set and generalization set, all xVerify models
achieve overall F1 scores and accuracy exceeding 95\%. Notably, the smallest
variant, xVerify-0.5B-I, outperforms all evaluation methods except GPT-4o,
while xVerify-3B-Ib surpasses GPT-4o in overall performance. These results
validate the effectiveness and generalizability of xVerify.