VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models
Authors: Yuchen Yan
Jin Jiang
Zhenbang Ren
Yijun Li
Xudong Cai
Yang Liu
Xin Xu
Mengdi Zhang
Jian Shao
Yongliang Shen
Jun Xiao
Yueting Zhuang
Venue: ICLR 2026 Poster (2026)
Keywords: Reference-based reward bench
Reward for reinforcement learning
Primary area: datasets and benchmarks
TL;DR: We propose VerifyBench(-Hard) to test reference-based reward systems for LLMs, and find current verifiers perform poorly on hard cases, highlighting key failure modes.

Abstract: Large reasoning models such as OpenAI o1 and DeepSeek-R1 have demonstrated remarkable performance in complex reasoning tasks. A critical component of their training is the incorporation of reference-based reward systems within reinforcement learning (RL), where model outputs are evaluated against ground truth references.  However, existing reward benchmarks focus on preference comparisons between responses rather than evaluating verification against ground truth references, leaving a critical gap in our ability to evaluate verification systems used in reasoning model training. In this paper, we introduce VerifyBench and its challenging variant VerifyBench-Hard, two benchmarks specifically designed to assess reference-based reward systems. These benchmarks are constructed through meticulous data collection and curation, followed by careful human annotation to ensure high quality. Our comprehensive evaluation reveals that while larger model-based verifiers show promise on standard cases, all current systems demonstrate substantial room for improvement on challenging instances. Through systematic analysis of performance patterns across reasoning tasks and error categories, we provide insights for advancing reference-based reward systems. These benchmarks establish a standardized framework for improving verification accuracy, ultimately enhancing reasoning capabilities in models trained via RL.

Published as a conference paper at ICLR 2026
VERIFYBENCH: BENCHMARKING REFERENCE-BASED REWARD SYSTEMS FOR LARGE LANGUAGE MODELS
Yuchen Yan Jin Jiang Zhenbang Ren Yijun Lit, Xudong Cai', Yang Liu?, Xin Xu', Mengdi Zhang". Jian Shao'; Yongliang Shen",, Jun Xiao', Yueting Zhuang' 'Zhejiang University "Meituan Group *Peking University *University of Electronic Science and Technology of China 'The Hong Kong University of Science and Technology [yanyuchen, syl, jshao )6 eudu..uu. edu. cn
ABSTRACT
Large reasoning models such as OpenAl ol and DeepSeek-RI have demonstrated remarkable performance in complex reasoning tasks. A critical component of their training is the incorporation of reference-based reward systems within rein- forcement learning (RL), where model outputs are evaluated against ground truth references. However, existing reward benchmarks focus on preference compar- isons between responses rather than evaluating verification against ground truth references, leaving a critical gap in our ability to evaluate verification systems used in reasoning model training. this paper, we introduce VerifyBench and its challenging variant VerifyBench-Hard, two benchmarks specifically designed to assess reference-based reward systems. These benchmarks are constructed through meticulous data collection and curation, followed by careful human an- notation to ensure high quality. Our comprehensive evaluation reveals that while larger model- based verifiers show promise standard cases, all current systems demonstrate substantial room for improvement on challenging instances. Through systematic analysis of performance patterns across reasoning tasks and error cate- gories, we provide insights for advancing reference-basee reward systems. These benchmarks establish standardized framework for improving verification accu- racy, ultimately enhancing reasoning capabilities in models trained via RL.
Project Page https///juuraaggghbbbbooooeereer
Benchmark Code
http:///ttbbbcomm/////EEAA///VVD
INTRODUCTION
In recent years, large language models (LLMs) have exhibited remarkable capabilities, significantly assisting humans across diverse practical domains (DeepSeek-AL et al.) 2025b) Grattafiori et al,
2024 Yang et al. 2025). Reinforcement learning from human feedback (RLHF) has been crucial to this progress, with reward models playing a central role by evaluating and scoring model-generated responses guide training. This approach has led to the development of numerous benchmarks
[Lambert et al. 2025 Liu et al. 2024| Zhou et al.) (202)) for systematic reward model evaluation, focusing primarily on pairwise preference judgments between competing responses.
The emergence of specialized large reasoning models (LRMs) (DeepSeek-AL et al., 2025a Owen Team 2024, Kimi Team al.) 2025) such as OpenAl's (OpenAl] (024)) and DeepSeek- RI Rl(DeepSeek-Al et al.) (025a) has fundamentally changed this landscape. These models achieve unprecedented performance on reasoning tasks through specialized reinforcement learning tech- niques that differ from standard RLHF approaches. A key distinction in training methodologies for LRMs is their reliance on reference-basee reward systems, where rewards are assigned based on alignment between model-generated responses and authoritative reference answers. This approach
"Contribution during internship at Meituan Group.
'Corrsponding authors.

---

Published as a conference paper at ICLR 2026
Chosen Completion
0.8
Question
Reward System
Chosen Rejected?
Rejected Completion
0.2
Reference Answer
Golden Label
Question
Reward System
True
Correct / Wrong?
Completion
Figure 1: The core distinction between VerifyBench and existing reward benchmarks (Lambert al., 2025 Liu et al. 2024) is illustrated as follows. Upper panel: Existing reward benchmarks assess the accuracy reward system by comparing the ranking of two completions for the same question. Lower panel: In contrast, our proposed VerifyBench evaluates the accuracy of a reward system by determining the correctness of a single completion using a reference answer.
has been implemented variously across leading models, with Deepseek-Ro employs a rule-based reward to prevent reward hacking, whereas models like Seed1.5. T-inkkin (Seed et al. (2225) adopt model-based reward systems to generate more precise and robust signals.
Despite the widespread adoption of reference-based reward systems in training state-of-the-art rea- soning models, significant gap exists our ability to evaluate these systems systematically. Cur- rent benchmarks focus almost exclusively on preference-basee evaluation, assessing rewards on their ability to rank competing responses correctly. This approach fails to capture the requirements of reference-based verification, where responses must be judged against objective ground truths rather than relative preferences (Kim et al., (0044). The absence of dedicated benchmarks for reference- based reward systems has limited researchers' ability to assess, compare, and improve their verifi- cation methodologies effectively, potentially impeding progress in reasoning model development.
To address this critical gap, introduce VerifyBench, benchmark specifically designed to evaluate the accuracy of reference-based reward systems. VerifyBench differs fundamentally from existing reward benchmarks by focusing on absolute correctness judgments rather than relative preference assessments. While traditional benchmarks ask reward models to determine which of two responses is better, VerifyBench challenges systems verify whether single response correctly aligns with reference answer, more accurately reflecting the actual use case in reasoning model training.
In this paper, we present VerifyBench, a benchmark specifically designed evaluate the accuracy of reference-based reward systems. To create VerifyBench, we curated diverse collection of in- structions paired with reference answers sourced from existing open datasets. Responses these instructions were generated by multiple open-source and proprietary LLMs. The correctness of each response was assessed using both automated model judgments and human evaluations. Each instance in VerifyBench was verified by at least two human annotators to ensure label consistency and reliability, thereby producing a high-quality benchmark for the evaluation of reward systems.
Recognizing the need to differentiate between various verification techniques and to push the bound- aries of current capabilities, we further developed VerifyBench- Hard, a more challenging variant of benchmark. This dataset focuses contentious cases where leading models produce highly con- flicting judgments, providing a more stringent test for reward system accuracy. Verify Bench-Har samples were carefully selected based disagreement patterns among high-performing models, then subjected to thorough human annotation to ensure label quality.
Our contributions summarized follows:
2

---

Published as a conference paper at ICLR 2026
To better reflect realistic reinforcement learning (RL) scenarios for reasoning models, con- struct VerifyBench, a benchmark derived from existing models and datasets, to provide an ob- jective evaluation of the accuracy of ffference-baset reward systems.
We further develop VerifyBench-Hard, a more challenging benchmark curated from cases ex- hibiting high disagreement among multiple models. This dataset contains larger proportion of difficultto-verrfyy samples, highlighting substantial potential for improvement in current models.
We conduct a comprehensive empirical analysis of model performance on both VerifyBench and Verify Bench-Hard offering actionable insights to advance the accuracy of reference-based reward systems and enhance RI. training reasoning tasks.
PRELIMINARIES
Reference-free Reward Models In reinforcement learning (RL) for large language models (LLMs), the reward model plays a crucial role by approximating real-world reward signals asso- ciated with model-generated outputs. A typical reward model takes as input user's query along with the corresponding I.LM-generated response r, and produces a reward signal, formally defined
reward = r))
(1)
where q represents the user's query, denotes the response generated by the LLM, and ip encapsu- lates either the learned parameters of the reward model or the heuristic criteria used to evaluate the quality of the response given g and r.
Evaluation of Reference-free Reward Models Generally, reward models produce scalar outputs whose scales can vary significantly across different implementations, complicating direct numerical comparisons. Consequently, current benchmarks evaluate reward models using a pairwise compara- tive approach. Formally, given a dataset D comprising tuples (4, Tie r2). where g represents user's s query, and Fw and rı denote two candidate responses with r. considered superior to ri, the accuracy of reward model is quantified as the proportion of instances in which the model correctly assigns a higher score to Tw than to ri. Mathematically, this accuracy metric is defined as:
Accuracy = D
Σ
[( > Rp(g,ri)
(2)
(q.Tw.ri)ED
where I(.) is the indicator function, and Rg denotes the reward model parameterized by p.
Reference-based Reward Models With the emergence of advanced reasoning models such as DeepSeek-R1, reference-based reward systems have been integrated into reinforcement learning (RL) frameworks for large reasoning models (LRMs). These models require training on extensive datasets, which typically include authoritative reference answers. Consequently, the reward assign- ment task shifts towards evaluating the alignment between the model-generated outputs and their corresponding reference answers. Formally, this reward calculation can be expressed as:
reward Re gl,
where y denotes the user-issued query, gl denotes the ground-truth reference answer, represents the generageee response, and yp encapsulates either the learned parameters of the reward model or the established evaluation criteria used to assess the alignment among gt, and r.
Evaluation of Reference-based Reward Models In this paper, we propose a reference-basee reward benchmark designed to systematically evaluate reward models within reinforcement learn- ing (RL) frameworks for large reasoning models (LRMs). Unlike traditional reward evaluation benchmarks, which rely on pairwise comparisons, our approach leverages explicit reference an- mener-eeeeeated responses. Concretely, given
swers to directly assess the correctness of individual a dataset D consisting of instances (g, gl, r, v). where q denotes the user-issued query, gl represents the ground-truth reference answer, r is the model-generatee response, and y is the binary correctness label assigned to the response, we evaluate the reward model by measuring its accuracy in correctly predicting these labels. Formally, the accuracy metric is defined as:
Accuracy
Σ
T (R. gt, =
(4)
D
(g.g
3

---

Published as a conference paper at ICLR 2026
Query Curation
Answer Type Labeling
Completion Generation Pre-annotation
Annotation
VerifyBench
Query Pool
Human Annotate
Balance
Balance Answer Type
Correctness
Content
foo
Question
Ref Answer
Answer Type Classification
Generate Completions
Verify
LLMs
Yi
Human Annotators
Generate
Verify
Domain
Completions
Filtered Verification Diff
General
X
x
Logic
Math
X
Human Annotate
Query Caration
Completion Generation
Difficutty Filtering
Iuama Annotation
VerityBench Hard
Overview of the benchmark construction process. The upper section outlines the pipeline used to construct VerifyBench, whereas the lower section details the pipeline for VerifyBench-Hae The components highlighted by black boxes denote the final entries included in the benchmark.
Figure 2:
where (g. gt. r) denotes the reward model parameterized by y defined by heuristic verifica- tion rules, producing predictions indicative of the correctness of response r relative to the provided reference answer gl. The function B(-) represents operation (e.g., thresholding discretization) mapping continuous reward scores into discrete correctness predictions suitable for direct compari- son with the ground-truth labels y.
BENCHMARK CONSTRUCTION
In this paper, introduce two benchmarks, VerifyBench and VerifyBench--ard, to evaluate reference-based reward systems. The VerifyBench benchmark (Section (.1]) is designed to reflect naturally distributed data, whereas VerifyBench-HHade (Section 3.2) comprises samples exhibiting high levels of disagreement among models, thereby assessing model's ability to provide reliable judgments in ambiguous or challenging scenarios.
3.1 CONSTRUCTION OF VERIFYBENCE
To emulate realistic reinforcement learning (RL) scenarios involving reference- based reward systems, we curate a comprehensive collection open-source reasoning problems paired with corresponding reference answers. These problems encompass three primary categories, general reasoning, logical reasoning and mathematical reasoning, and are aggregated from 41 dis- tinct sources. A complete list of these data sources provided in Appendix D
Query Curation
Answer Type Labeling To comprehensively evaluate model performance across diverse an- swer formats, we define four canonical answer types: numerical values, algebraic expressions, multiple-choice selections, and free-form strings. Utilizing general-purpose LLM L.ama-3.3-7000 Instruct (Grattafiori al. 2024). we performed automatic answer-type classification with prompt (Appendix G.I.. Questions that fall outside these categories, such as proof-based or open-ended prompts, were excluded from further analysis. Following classification, we randomly sampled 2,000 instances per answer type, resulting in final candidate pool of 8,000 questions.
Completion Generation and Pre-annotation We subsequently employed 22 widely used open- source and proprietary models (see Appendix B generate single-shot completions for the curated
4

---

Published as a conference paper at ICLR 2026
set of 8,000 questions, resulting in total of 176,000 completions. To assign initial correctness labels, we utilized Llama-3.3-7088 -lamaa333770--msstuuttt (Grattafiori et al. (024) within a prompt- based judgment framework. For each question, randomly selected four completions, two labeled correct and two labeled as incorrect by the model, and retained them for subsequent human annotation.
Human Annotation We conducted human annotation for the aforementioned questions and their associated completions. The annotation procedure comprised two primary tasks: (1) identifying the most appropriate answer type for each question based on its formulation and corresponding ground- truth answer, and (2) evaluating the correctness of each of the four completions. Each question was independently annotated by at least two annotators. If their annotations were consistent, the labeling was finalized; otherwise, meta-annotaton resolved disagreements to ensure consistency and finalize the labels, A detailed description of the human annotation process is provided in Appendix H
Benchmark Construction Following human annotation, we identified notable biases in the mod- els' predictions regarding both answer types and completion correctness, leading to imbalanced data distributions. To mitigate this issue, we performed controlled downsampling to ensure uni- form category-level representation and balanced correctness labels. Specifically, retained 250 questions per answer type, resulting in a total of 1,000 questions. Each question is paired with exactly two completions, one correct and one incorrect, aimed to minimize bias arising from class imbalance. In this way, we maintain an equal number of samples across these 4 answer types in VerifyBench, enabling us to assess model's verification accuracy under different scenarios. The resulting dataset, VerifyBench, thus comprises 2,000 well-balanced question-answee aaweo-aaawr-oooplction correctness tuples. Detailed statistics for VerifyBench are provided in Table and Appendix [F] and illustrative examples are presented in Appendix K and L
3.2 CONSTRUCTION OF BENCH-
To construct VerifyBench-Hann we employed specialized data generation pipeline consisting of the following key steps:
Table I: Benchmark statistics of VerifyBench
(VB) and VerifyBench-Hard (VB-H).
Statistics for VerifyBench and VerifyBench Hard
Completion Generation To construct the dataset, we first generated single-turn comple- tions for the queries described in Section 3.1 using a collection of 18 open-source mod- els (Appendix [E). Due to the substantial vol- ume of generations and the associated compu- tational costs, closed-source models were ex- cluded from this stage. In total, we produced approximately 1.45 million completions.
Statistics
VB
VB-H
of unique questions
1000
945
of unique completions
2000
1000
of correct completions
1000
291
of wrong completions
1000
709
Answer Type Distribution
Numeric Values
500
252
of Expressions
500
88
Difficulty Filtering Next, we employed five top-performing LLMs on VerifyBench (LJama-3.3-70B-Inntmnc (Grattafiori et al.
of Multi-choice
500
430
of String
500
230
Domain Distribution
(Meta (Qwen cct al. 2025). Qwen3-30B-An and Qwen3-
2024 1aama-4-Scuut-17B-666----
A1
2025). Dwee2.5--2B-IInttrn
General Reasoing
404
303
of Logic Reasoning
498
315
Yang al. 2025)), which span diverse range of model sizes and architectures, to evaluate the correctness of the generated completions. Based their judgments, we identified inestion-ansee-ccopllttooo tuples exhibiting model disagreement, specifically those for which two models' assessments diverged from the other three. To ensure balanced and comprehensive representation, we applied stratified sampling across data domains and sources, ultimately selecting 2,000 examples for human annotation.
32B
Math Reasoning 1098
382
Human Annotation We subsequently subjected the selected samples to human annotation, fo- cusing on two key aspects: identifying the answer type and determining the correctness of each completion. Each instance was annotated independently by least annotators. In cases where
5

---

Published as a conference paper at ICLR 2026
Table Overall performance(%) of VerifyBench and Verify Num stands for Numeric Values, Exp stands for Expressions, MC stands for Multi-choice and Str stands for String.
VerifyRench
Verifyyench--aan
Model/Method
Num
Exp
MC
Str
AVG
Num
Exp
MC
AVG
Rule-based functions
funnttion
math-verify
85.60
75.60
55.00
51.60
66.95
84.52
82.95
68.37
78.26
76.00
LiM-ax-a-judge
ppX-4o-022-11-220
94.80
90.20
96.80
90.80
93.15
71,43
65.91
75,35
71.30
72.60
gpt-4o-mini
95.80
89.80
95.80
90.00
92.85
69.05
72.73
74.19
72.17
72.30
DeepSeck-V3-0324
96.80
93.00
97.60
91.60
94.75
81.35
77.27
81.63
75.22
79.70
DeepSeck-R1-0528
98.00
92.60
98,00
92.00
95.15
82.14
81.82
90.93
85.22
86.60
gpt-oss-20b
97.40
94.20
98.40
89.20
94.80
82.54
79.55
90.23
76.96
84.30
gpt-oss-120b
98.00
94.80
99.20
91.40
95.85
84.13
80.68
92.56
86.09
87.90
1aama-4-17-116-Immsae
94.20
86.80
89.80
89.25
90.01
48.02
39.77
46.98
55.22
48.50
Llama-3,3-700-Imm
88.80
77.80
88.40
78.00
83.25
54.37
45.45
60.70
47.39
54.70
Llama-3,1-88
Inst-nrrrettt
72.20
70.60
77.00
72.40
73.05
51.19
35.23
45.12
33.91
43.20
65.80
63.60
56.80
57.60
60.95
33.33
28.41
38.84
27.39
33.90
1aama-3.2-1B-Instruu
44.40
41.00
37.60
53.60
44.15
22.22
13.64
29.07
27.39
25.60
Qwen3-235B-A221
96.40
92.40
97.00
89.40
93.80
70.24
72.73
70.93
69.57
Qwen3-30B-A335
96.60
91.80
97.40
90.20
94.00
64.68
70.45
69.53
56.52
65.40
Owen2.5-72B-Insee
95.40
95.60
88.60
92.35
60.23
61.40
62.40
Qwen3-32B
97.60
94.00
99.00
92.60
95.80
81.82
68.14
77.83
Qwen3-8B
96.40
93.00
90.40
94.00
68.65
78.41
73.02
66.52
70.90
Qwen3-4B
95.20
91.60
87.60
92.00
71.03
62.50
75.58
71.74
72.40
Qwen3-1.7B
83.20
81.00
80.60
79.60
81.10
48.81
38.64
60.93
41.74
51.50
phi-4
92.60
86.40
93.00
85.40
89.35
59.52
57.95
54.19
57.39
56.60
11-1.5-9B-Chat-Bl
90.40
87.40
88.00
85.00
87.70
65.48
63.64
62.09
54.78
61.40
gemma-3-1b-it
55.40
56.20
43.00
56.00
52.65
32.14
19.32
33.72
40.87
33.70
Model-based Verifier
VVriff--.5BB-
79.60
74.40
88.80
87.20
82.50
60.32
62.50
91.86
77.83
78.10
XVerily-BB-Ia
89.80
86.40
95.00
90.20
90.35
73.81
78.41
93.49
73.91
82.70
x VVerify-BB-1
93.80
89.60
95.20
89.20
91.95
69.05
76.14
93.49
81.74
83.10
Tencent-
Qwenn.5-7B-RR..
96.80
95.40
93.00
90.80
94.00
75.43
75.00
61.36
63.64
72.20
CompassVerifier::
92.00
84.80
94.60
88.60
90.00
77.38
84.09
93.95
75.22
84.60
CompassVerifier-7B
92.80
88.60
97.20
88.40
91.75
78.97
85.23
95.35
76.09
85.90
CompassVeriffer-22R
94.20
89.80
96.80
89.40
92.55
80.56
82.95
95.58
83.48
87.90
both annotators agreed, the annotation was finalized; when disagreement occurred, aecta-nnntttor was consulted resolve the conflict. detailed description of the human annotation process is pro- vided in Appendix H
Benchmark Construction Following human annotation, we excluded samples identified as un- suitable for inclusion in benchmark. This filtering resulted in final set of 1,000 question- mnswer-coplletioncorreeett
tuples. In contrast to VerifyBench, which enforces a balanced struc- ture with correct and one incorrect completion per question, VerifyBench-Haut is derived through natural sampling. We observed that larger models are more likely to erroneously accept incorrect answers correct, resulting in natural skew towards incorrect completions within the dataset. Detailed statistics for SerifyBench-Harde are provided in Table
4 EVALUATION RESULTS
This section presents the evaluation results and analyses of our proposed benchmark. Section 4.1 reports the primary evaluation outcomes. In Section 4.2 we investigate the impact of reference an- swers the verification process. Section 4.3 provides comparative analysis between bench- mark and existing reward benchmarks, as well as the performance several general purpose reward models on VerifyBench and
ceneral-purppoee

---

Published as a conference paper at ICLR 2026
4.1 OVERALL PERFORMANCE
We evaluate the performance of various verification approaches on both VerifyBench and VerifyBench- Hardd For rule-based baselines, we adopt the widely used math-verify (Kydlíček) (025) method" In the LLM-as-a-judge setting, prompt general LLMs to perform verification; detailed prompt templates are provided in Appendix G.2 Additionally, we conduct evaluations of exist- ing verification-spciialiee model-based verifiers, which have been carefully trained verification tasks, employing prompts consistent with their provided templates to ensure faimesss (Chen et al.) 2025
Su et al. 2025
Liu al. 2025a Our evaluation yields several key findings:
Existing models perform well on Verify Berify The primary objective in constructing Verify- Bench is to establish benchmark for the objective evaluation of reference-based reward systems. To this end, we designed the dataset with a balanced distribution across diverse domains and answer types, pairing each question with both a correct and an incorrect completion. This structure facili- tates a rigorous and fair assessment of reward model performance. Notably, bfate-of-hhe-r the LLMs already demonstrate strong performance on this benchmark: GPT-4o-mini achieves an average ac- curacy of 92.85% while Qwen3-32B reaches 95.8%, highlighting the high reliability of LLMs as verifiers in this context.
Verify Berth-Been is challenging. To more effectively differentiate the performance of various models, we constructed Verify by selecting cases in which multiple LLMs exhibited substantial disagreement in their verification outputs. Evaluation results demonstrate that model performance on Verify Bench-Hard is significantly lower than on VerifyBench. The highest accuracy achieved is 72.4%, representing a 20% decrease compared to performance on VerifyBench. This performance gap underscores substantial opportunities for improvement the precise verification capabilities of current LLMs.
Small-scale models still have room for development. In practical reinforcement learning sce- narios, the inference efficiency of the reward system significantly impacts the overall training speed. Since such verification tasks typically involve generative inference, their computational cost is com- parable to that of the rollout process itself. Thus, efficiently leveraging smaller models to perform verification is a practical concern worth exploring. According to our results, models with smaller pa- rameters (<3B parameters) exhibit notably poorer performance on VerifyBench, achieving 80.10% accuracy with Qwen3-1.7B and only 60.95% accuracy with 3.2ma--.2-3B-Insse scale models can achieve over 90% accuracy. Therefore, enhancing the capability of smaller models on these verification tasks represents a valuable direction for future research.
while larger-
4.2 REFERENCE-ANSWER PLAY AN IMPORTANT ROLE IN VERIFICATION
The benchmark proposed this work funda- mentally differs from existing reward bench- marks by explicitly incorporating reference answers, thereby aligning more closely with the training setups of contemporary reasoning LLMs. To isolate the impact of reference an- swers on verification performance, we conduct an ablation study in which models are evalu- ated without reference answers provided in the prompt; the prompt format used is detailed in Appendix G.3
Table 3: Evaluation results about how includ-
ing the reference answer in the prompt influences the performance of LLM-as-a-judge.
Verify Bench
Model
Ref
w/o Ref
1ire44--cout-77--6-2 1nE-tnstt
90.01
73.95 16.05
Llama-3.3-70P8 Innstruct
83.25
75.00
Llama-3.1-88 1-.1----nnstaae
73.05
64. 10 8.95
1lama-3.2-3B-Insttt
60.95
55.35-sA0
Llmma-3.2-1B-Instrru
44.15
44.50-0.35
Qwen3-235B-ATT
93.80
Experimental results, summarized in Table 3 reveal performance degradation of approxi- mately 5-18% when reference answers are ex- cluded. These findings underscore the crucial role of reference answers in masooing--oriente
Qwen3-30B-A3B
94.00
78.25-15.75
Qwen2.5-77BBInstrmut
92.35
77.30-15.05
Qwen3-32B
95.80
78.90
Qwen3-8B
94.00
75.75 18.25
Qwen3-4B
92.00
74.40-17.0
Qwen3-1.7B
81.10
62.10 19.00
For math-verify, we report the accuracy of version v0.8.0.

---

Published as a conference paper at ICLR 2026
Table The performance(%) of existing reward models VerifyBench without access reference answers, as well as comparison with existing reward benchmarks.
Verify BerfhyBench
Model
R.M-Bench
Reward Bench
Num
Exp
MC
Str
AVG
General Reward Models
Skywork- Reeerk--ewwar----d- Reward-L.lam--3 1lama-..1-42 1-1-B
72.29
93.33
60.80
64.80
59.60
68.80
63.48
(iermlm2-200-rewaal
72.06
92.16
65.60
64.80
61.20
70.00
65.40
GRM-llama3-8B-sfute mntemlm2-7b-rrwaam
71.33
88.50
64.80
58.40
58.80
67.60
62.40
72.42
90.02
73.20
68.00
66.80
70.40
Domain speciific Reward Models
Qwen2.5- Maann55--M Qwen2-Malte Math--M--h-M
76.28 62.61
82.11
83.60
79.20
73.60
75.60
78.00
75.54
79.20
78.40
73.20
72.80
75.90
RL, suggesting they provide more reliable and informative supervision signal during reward mod- eling.
4.3 PERFORMANCE OF REFERENCE- REWARD MODELS
To enable comprehensive evaluation of existing reward models, we additionally assessed several reference-frre reward models and benchmarked their performance on conventional pairwise reward evaluation datasets for comparison. Notably. each question in our proposed VerifyBench consists of one correct and one incorrect completion, enabling straightforward reformulation into standard pairwise evaluation instances. The experimental results are summarized Table
Our experimental results show that VerifyBench introduces level of challenge comparable to ex- isting reward benchmarks, with the absence reference answers. Reference-free reward models achieve sub-80% accuracy VerifyBench, highlighting its difficulty. Furthermore. domain-specifte reward models exhibit inferior performance on general reward benchmarks (Liu et al. 2024 Lam-
bert et al. 2025) compared to VerifyBench, validating the benchmark's 's design objectives.
ANALYSIS
5.1
ERROR ANALYSIS
To gain deeper insights from VerifyBench, we introduce a more fine-grained taxonomy for each answer type and analyze model performance across these subcategories. This detailed analysis helps identify specific reasoning tasks or answer formats where models are particularly error-prone. We subdivide the Numeric Values category into 8 subtypes, Expressions into 5 subtypes, Multi- choice into 3 subtypes, and String into 2 subtypes. Table presents the comparative performance of different models across these detailed categories. The complete taxonomy of VerifyBench and VerifyBench-Hurn Hard is provided in Appendix[]
We further analyze subcategories within each major category that exhibit below-average accuracy. The following error-prone subtypes are identified as the most frequent sources of incorrect judg- ments:
Numerie Values: Complex numbers and answers containing multiple numerical values;
Expressions: Formulas and equations;
Multi-choiee: Multiple-choice problems;
String: Strings requiring semantic consistency verification.
We analyzed the samples most prone to errors and identified a common underlying issue: mod- els frequently fail fully comprehend the question clearly recognize the intended objective. For instance, in cases involving multi-value answers, the ordering of values is typically irrelevant. However, if the sequence of values in the model's output differs from the golden answer, models
8

---

Published as a conference paper at ICLR 2026
MATH500
AMC23
MinervaMath
68
28
66
Accuracy
40
26
64
38
36
24
34
22
32
50
100
150
200
50
100
150
200
50
100
150
200
Training Step
Training Step
Training Step
1lmma-32:30--nmae (60.95)
Qwen3-1.7B (81.10)
Qwen3-4B (92.00)
opt-sss-20b (94.8)
math-verify
Figure 3: The performance(%) of RL across different LLM judges which have various performance on VerifyBench.
often incorrectly classify the response as erroneous. Similarly, errors within the Expressions cate- gory, particularly involving algebraic formulas and equations, predominantly result from inadequate mathematical comprehension. Specifically, when model outputs an unsimplified expression, su- perficial textual discrepancies compared the ground-truth answer can be significant. Rather than evaluating whether the expression is mathematically equivalent upon simplification, models prema- turely deem the output incorrect, thereby leading to verification failures. We provide, in Appendix M examples of misjudgments made by gpt-oss-120b, the best pest-performing open-source model on Verify within each category. These examples span different answer types and are intended to help better understand why LLM verifiers make errors.
5.2 CORRELATION ANALYSIS
We constructed Verify VerifyBench and VerifyBench- Hard with the goal of improving the effective- ness of RL for reasoning models by enhanc- ing the accuracy of reference-baseee reward sys- tems. To evaluate the practical utility of our benchmark, we performed correlation anal- ysis between VerifyBench and real-world RI. performance.
Table 5: Model performance(%o) across the fine- grained taxonomy on VerifyBench. Q32B stands Qwen3-32B, g4o stands for gpt-4o-2024-1 20, L70B stands for 1i-mma-..--70--nnee and L3B stands for J.ama-3.2-3B-Im
Answer Type
Q32B
g4o
L70B
L3B
Numeric Values
97.60
94.80
88.80
65.80
We conducted practical RL training on Qwen2.5-3B-Instruct (Qwen et al.] (025). using LLMs with varying performance on VerifyBench LLM judgers to serve the reward system. We then compared their improvements downstream tasks. Specif- ically, we adopted math-verify (Kydliček) LLamma-.2-3B--nsttrr
Integer
96.88-022
96.88
93.75
65.62-6.1
Constant
96.88 0.2
92.19
70.31
Float
98.39
96.77
90.32
61.29
Radical
98.39
95.16
87.10
75.81
Complex
96.77-aas
96.77
85.48-3.22
59.68-6.12
Angle
96.77-aas
96.77
93.55
66.13
Non-decimal
93.55--125
88.71
64,52-1.28
Multiple values
96.77.ass
87.10.2.0
79.03
9:703-9.77
62.90
Expressions
94.00
90.20
77.80
63.60
2025
(Grattafiori
Formula
91.54-2.46
84.62.s.s
67.69.10
56.92-6.88
et al.
2024), Qwen3-1.7, Qwen3-4B (Yang
Equation
87.50.6.
78. 12-12.1
70.31-7.9
60,94-3.66
2025). and gpt-oss-20b (OpenAl al. as verifiers, employed GRPO (Shao al. 2024) as the RL algorithm, and trained on DeepMath-103K. The accuracy changes on MATH500 (Hendrycks et al.] 2021). AMC23 (mathai 2023), and Miner- vaMath [Lewkowycz et al. (2022) are shown provide addi- tional correlation analysis based on Rejection Sampling Fine-tuning and all the experimental details in Appendix !
Interval
96.09
94.53
82.81
60.94-3.66
2025
98.00
78.00
60.00
Matrix
96.09
94.53
86.72
75.78
Multi-choice
99.00
96.80
88.40
$6.80
Single-choice
99.39
98.17
92.07
59.15
Multiple-choice
98. 21 0.79
94,05 2:75
77.98
10.42
49.40 7.40
State selection
99.40
98.21
95.24
61.90
String
92.60
90.80
78.00
$7.6
in Figure 3 Meanwhile,
Specific
93.60
93.31
81.69
59.01
Sermantic
90.38 2.22
85.26
54.49.3.11
From our experimental results, we observe that models performing well on VerifyBench, when used verifiers in RL training, lead to greater improvements related tasks. For example, employing

---

Published as a conference paper at ICLR 2026
gpt-oss-20b and Qwen3-4B yields significant gains, whereas using models with weaker VerifyBench performance as verifiers may even degrade the model's 's capability. This is because such models in- troduce substantial bias in reward assignment, resulting in inaccurate supervision. Our experiments demonstrate that VerifyBench provides effective guidance in RI. training scenarios, highlighting the practical utility of our benchmarks.
CONCLUSION
In this paper, we present two dedicated benchmarks, VerifyBench and VerifyBench-Harte to evaluate reference-based reward systems in the context reasoning-focusee reinforcement learning. These benchmarks were built with high-quality, carefully curated data and extensive human annotation. Our results reveal that current verifiers, especially those with smaller model sizes, still face con- siderable challenges in accurately assessing reasoning completions. Through detailed analysis, provide insights into the strengths and weaknesses of existing systems and highlight opportunities for improvement. The proposed benchmarks fill critical gap in the evaluation landscape, offering a principled foundation for understanding verifier accuracy and guiding the development of more effective reasoning models trained via reinforcement learning.
ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation of China (No. 62506332), "Pi- oneer" and "Leading Goose" R&D Program of Zhejiang (NO. 2026C02A1223) and CCF-Tencent Rhino-Bird Open Research Fund.
ETHICS STATEMENT
All human annotators involved in constructing the benchmarks were assigned reasonable workloads and fairly compensated for their contributions.
Our annotation process involves minimal subjective preference. Human annotators performed the verification tasks following our detailed instructions. The content of the annotations does not involve ethical issues and poses no ethical risks.
REPRODUCIBILITY STATEMENT
We provide the VerifyBench dataset and evaluation code in the supplementary materials to facilitate the research community's 's use of our benchmark. In Appendix J we detail the hyperparameter set- tings used in evaluations, enabling researchers to reproduce our results. For the RL experiments in Section 5.2, we conducted experiments using the 'remote rm* recipe provided by veRI. (Sheng et al. 2025) The detailed experimental configurations also included in Appendix[]te support producibility. Nevertheless, due to the inherent stochasticity in LLM training and inference, results may still exhibit variability.
REFERENCES
01 AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, et al. Yi: Open foundation models by Ol.ai, January 2025.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, al. Abductive commonsense reasoning. In International Conference on Learning Representations, September 2019.
Ding Chen, Qingchen Yu, Pengyuan Wang, Wentao Zhang, Bo Tang, et al. xverify: Efficient answer verifier for reasoning model evaluations, April 2025.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
10

---

Published as a conference paper at ICLR 2026
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, et al. Training veri- fiers to solve math word problems, November 2021.
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, al. Deepseek-rl: Incen- tivizing reasoning capability in llms via reinforcement learning, January 2025a.
DeepSeek-Al AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, et Deepseek-v3 technical report, February 2025b.
Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. Mathodyssey: Benchmarking mathe- matical problem-sol -olvinn skills in large language models using odyssey math data. Scientific Data, 12(1):1392, August 2025. ISSN 2052-4463. doi: 10. 10.10388<41197-02--52222
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, et al. How to evaluate reward models for rlhf, October 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, et al. Gemma technical report, March 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The Ilama herd of models, November 2024.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, et al. Folio: Natural language reasoning with first-order logic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22017-22031, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 106653//12022.cmm// main. 1229.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, al. Olympiadhench: challeng- ing benchmark for promoting agi with olympiad-lvvel bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3828-38883 38500, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 11..6633/1/02222accooon.g.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, al. Deepmath- challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning, April 2025.
103k: 10ppmtth-103k:
large-scale,
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, et al. Measuring mathe- matical problem solving with the math dataset. In Thirty- Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), August 2021.
Haitao Hong. Yuchen Yan, Xingyu Wu, Guiyang Hou, Wenqi Zhang, et al. Cooper: Co-optimizing policy and reward models in reinforcement learning for large language models, August 2025.
Mchran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. Boardgameqa: dataset for natural language reasoning with contradictory infor- mation. In Thirty-Sevntth Conference Neural Information Processing Systems Datasets and Benchmarks Track, November 2023.
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, et al. Big-bench extra hard, May 2025.
Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, et al. Process reward models that think, April 2025.
Sunghwan Kim, Dongjin Kang. Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo. Evaluating robustness of reward models for mathematical reasoning, October 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, et al. Kimi k1.5; Scaling rein- forcement learning with lmss, January 2025.
11

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems Principles, pp. 611-626, Koblenz Germany, October 2023.
ACM. doi: 10. 10.114555 1,.116660000663613333
Hynek Kydlíček. Math-verify: Math verification library, May 2025.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, et al. Re- wardbench: Evaluating reward models for language modeling. In Findings of the Associa- tion for Computational Linguisties: NAACL 2025, pp. 1755-1797, Albuquerque, New Mex- ico, April 2025. Association for Computational Linguistics. ISBN 1998--9117--195-7. doi: N..18653/v1/2022 hin.:/si//22......innnnmnnooo naacl.96.
Aitor Lewkowyzz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, et al. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems, October 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions, July 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, et al. Let's verify step by step. In The Twelfth International Conference on Learning Representations, October 2023.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pp. 3622-3628, Yoko- hama, Japan, July 2020. International Joint Conferences on Artificial Intelligence Organization. doi: 0149666/jcai.202005555
Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, et Compassverifier: A unified and robust verifier for llms evaluation and outcome reward, August 2025a.
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style. In The Thirteenth International Con- ference Learning Representations, October 2024.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, al. Inference-time scaling for generalist reward modeling, April 2025b.
mathai. Amc23 datasets at hugging face. https//wwgggiiiiicc...ddaeeeeeet 2023. 2023.
Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal innovation. May 2025.
Shen-yun Miao, Chao-Chun Liang, and Keh- Yih Su. diverse corpus for evaluating and developing english math word problem solvers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Lingnistics, pp. 975-984, Online, July 2020. Association for Computational Linguistics. doi: 10. Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math, February 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, et al. Webgpt: Browser- assisted question-answering with human feedback, June 2022.
Yixin Nic, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, et al. Adversarial nli: A new benchmark for natural language understanding, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, Online, July 2020. Association for Computational Linguistics. doi: 10..6631-112220..ammmmm..
OpenAI. Introducing openai ol. http::/wgeeaaiicooiieeeemmmmmmme 2024.
12

---

Published as a conference paper at ICLR 2026
OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, al. Gpt-oss-120b & gpt-oss- 20b model card, August 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 15:27730-27744 December 2022.
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, et al. Logichench: To- wards systematic evaluation of logical reasoning ability of large language models. LnPrro Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16679-13777, Bangkok, Thailand, August 2024. Association for Computational Lin- guisties. doi: 10.18655/v1/2024.acl-be
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 maneel-main.1eee
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, et al. Multi- logieval: Towards evaluating multi-step logical reasoning ability of large language models. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp- 10856-208777 Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.1 2..18666//112222.mmml 1atpmnlllmmmmm11111
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen2.5 technical report, January 2025.
Qwen Team. https:/qqenilm. httpuubbiooblogggggg-bb---eeeeee
Qwq: Reflect deeply on the boundaries of the unknown.
November 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, et al. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, August 2024.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-op of thothh. In The Eleventh International Conference on Learning Representattons, September 2022.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical rea- soning abilities of neural models. In International Conference on Learning Representations, September 2018
ByteDDacee Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, et al. Seed1 --think-t Advanc- ing superb reasoning models with reinforcement learning, April 2025.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, et al. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. In The Thirleenth International Conference on Learning Representations, October 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, February 2024.
Guangming Sheng. Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, al. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279 1297, March 2025. doi: 10. 14513689031133666 13666755.
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton, et al. Clutrr: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNL) pp. 4506-45 4515, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 1/155//11D111144
13

---

Published as a conference paper at ICLR 2026
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, et. Learning summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pp. 3008-3021. Curran Associates, Inc., 2020.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, et al. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains, April 2025.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, et al. Challenging big-bench tasks and whether chain-of-though can solve them. In Findings of the Association for Computational Linguistics: ACI. 2023, pp. 13003-13051, Toronto, Canada, July 2023. Associa- tion for Computational Linguistics. doi: 10. (0.18633/v1/2233 1f0533//1222...nnningg--eee findings-acl.882.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proofwriter: Generating implications, proofs, and abductive statements over natural language. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3621-3634. Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 findings-acl.317.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed H. Chi, al. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, September 2022.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-Eight Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track, November 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, al. Chain-of-thoughh prompting elicits reasoning in large language models. In Advances in Neural Information Pro- cessing Systems, October 2022.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks, December 2015.
Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, al. Tablebench: A comprehensive and complex benchmark for table question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 39(24):5597725506 April 2025. ISSN 2374-3468, 2159-5399, doi: 3.699aaaiiv33124..3444
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, al. A minimalist approach to Ilm reasoning: From rejection sampling to reinforce, April 2025.
Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? a comprehensive evaluation and beyond. IEEE Transac- tions on Knowledge and Data Engineering, 37(4):120011644 April 2025a. ISSN 1558-2191. doi: 10.
Yuhui Xu, Hanze Dong, Lei Wang, Caiming Xiong, and Junnan Li. Reward models identify consis- tency, not causality, February 2025b.
An Yang, Beichen Zhang. Binyuan Hui, Bofei Gao, Bowen Yu, al. Qwen2.5-matt technical report: Toward mathematical expert model via self-improvement, September 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, al. Qwen3 technical report, May 2025.
Nathan Young, Qiming Bao, Joshua Bensemann, and Michael Witbrock. Abductionrules: Training transformers to explain unexpected inputs. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings the Association for Computational Linguistics: ACL 2022, pp. 218-227. Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ v1/2022, findingsnigg
14

---

Published as a conference paper at ICLR 2026
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng, Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations, September 2019.
Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, et al. Naturalreasoning: Reasoning in the wild with 2.8m challenging questions, February 2025.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well do large language models perform in arithmetic tasks?, March 2023.
Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. Quiet-star: Language models can teach themselves to think before speaking. In First Conference on Language Modeling, August 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi, et al. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1472.
Zhang, Yifan, Stackmathqa: curated collection million mathematical questions and answers sourced from stack exchange. https://huggingficc..o http://ww.....teeee/eaa/a///aaaa
2024.
Chujie Zheng. Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, al. Processbench: Iden-
tifying process errors in mathematical reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics Vooluee Long Papers), pp. 1009-1024, Vienna, Austria, July 2025, Association for Computational Linguistics. ISBN 979-8-89176-251-00 doi: 10.18653/v 1/202/ 12205..ac--oon..50.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, et al. Sglang: Efficient execution of structured language model programs. In The Thirty-Fighth Annual Conference on Neural Information Processing Systems, November 2024.
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, et al. Ar-lsat: Investigating ana- lytical reasoning of text, April 2021.
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, et al. Rmb: Comprehensively benchmarking reward models in Ilm alignment. In The Thirteenth International Conference on Learning Representations, October 2024.
15

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Preliminaries
Benchmark Construction
3.1
Construction of VerityBench
3.2
Construction VerifyBench-
Evaluation Results
Overall Performance
Reference-answert Play an Important Role in Verification
Performance of Reference- free Reward Models
5
Analysis
5.1
Error Analysis
5.2
Correlation Analysis
Conclusion
10
LLM Usage Declaration
18
Limitations
18
Related Works
18
Reward Systems for Reinforcement Learning
18
Evaluation of Reward Systems
19
D Data Source
19
LLM Usage For Benchmark Construction
21
Detailed Benchmark Statisties
Source Model Distribution
21
F2
Token Length Distribution of Completions Verify Bench( -Hard)
22
Prompt Templates
23
G.1
Prompt Template for Answer Type Classification
23
G
Prompt Template for LLM-as-a-judge
24
G.3
Prompt Template for LLM-as- without Reference
24
Details of Human Annotation
25
Instruction for Annotators
25
16

---

Published as a conference paper at ICLR 2026
H.2 Annotators
25
H.3 Design for Human Annotation Process
25
H.3.1 Page for Primary Annotators
26
H.3.2 Page for Antottannss
26
Full Taxonomy of VerifyBench and Berchy
27
J
Correlation Experimental Details
27
J.1
Reinforcement Learning Experiments
27
J.I.1
Setup
27
J.1.2
Experimental Results
................................
28
J.2
Rejection Sampling Fine-tuning Experiments
J.2.1
Setup
................................
28
J.2.2
Experimental Results
K
Examples of VerifyBench
30
K.1
An example of VerityBench (Answer type: Numeric Values)
30
K.2
An example VerifyBench (Answer type: Expression)
................................
30
K.3
An example of VerilyBench (Answer type: Multi-choice)
31
K.4 ..4
An example of VerifyBench (Answer type: String)
32
Examples of -erihyBeench-Hard
33
example Heri (Answer type: Numeric Values)
33
An example of VeriiyBench-Hard (Answer type: Expression)
34
An example of HAnthBBeeech-HHaadAAnnee type: Multi-choice)
34
L.4
An example of VerityBencc-Harr (Answer type: String)
35
M Error Case Study
36
17

---

Published as a conference paper at ICLR 2026
A LLM USAGE DECLARATION
In writing this paper, we only used LLMs for polishing. The generation of ideas this work did not involve any assistance from LLMs. The experimental design and manuseript writing were not directly produced by LLMs either. The models were used solely as polishing tool: specifically, first drafted the manuscript, then refined it with the help of an LLM, and finally the authors conducted another round of verification after polishing.
B LIMITATIONS
Limited Data Domain In this paper, we utilize datasets exclusively from general reasoning. logi- cal reasoning, and mathematical reasoning, which do not the full spectrum of reasoning types, such as commonsense reasoning. Consequently, our test sets may not adequately evaluate the quality of reward systems in out-of-domain scenarios.
Bias from Human Annotation The construction of VerifyBench and VerifyBench- Hard involved extensive human annotation. Although all annotators were trained and a double-checking strategy was employed, it remains challenging to entirely eliminate annotation bias inherent in manual label- ing processes.
Reward Hacking Could Not Be Identified While experiments demonstrate that rule-based reward systems perform worse than prodel-based approaches on both VerifyBench and Hard, critical issue remains unaddressed: reward hacking. Future research should focus detect- ing and evaluating reward hacking phenomena.
Proof Problems Excluded During annotation, our guidelines explicitly excluded proof-based questions. We believe such problems require more specialized verification methods, such as for- mal languages like Lean4. Consequently, proof questions are not included in this study, and their verification remains an open research challenge.
Binary Scoring System The benchmark constructed this paper employs a binary scoring sys- tem, where each completion is labeled as either correct incorrect. However, real-world scenarios often involve more nuanced cases, such as partially correct reasoning processes or correct solu- tions to subproblems. Introducing more fine-grained evaluation scheme could better capture these complexities.
C RELATED WORKS
C.I REWARD SYSTEMS FOR REINFORCEMENT LEARNING
Early reward models (RMs) (Christiano et al. 2017 Stiennon al. 2020 Ouyang al. 2022
trained to predict human preference rankings, typically treat the entire response as the evaluation unit. However, such outcome-level RMs lack insight into intermediate reasoning steps, making step-level eтrоr correction infeasible (Xu et al. (025b). To address this limitation, process-level RMs (Lightman et al., 2023 Setlur al.) (2024) have been introduced to assign scores at each reasoning step, thereby providing stepwise feedback. Despite their effectiveness, process-level RMs require extensive manual step-level annotations, resulting in exponential increases data collection costs and training complexity (Khalifa et al. 2025).
Building on these advances, DeepSeek-RI (DeepSeek-A) Al et al., 2025a) employs rule-based reward functions that leverage predefined, maintainable rules for pattern matching and logical validation, offering simplicity and efficiency. However, as task diversity expands, the manual creation of such rules faces significant challenges related to scalability and coverage, ultimately limiting its applica- bility in open- open-ended generation scenarios.
More recently, DeepSeek-GRis GRM (Liu al. 2025b) and ThinkPRM (Khalifa et al. 2025) have plored integrating reasoning capabilities into RMs by developing generative reward models (GRMs). GRMs reformulate the scoring task as token-generation problem: before outputting numerical
18

---

Published as a conference paper at ICLR 2026
score, the model first generates thain-offtthougeee (CoT) (Wei et al. 2022) that explicates its evalua- tion criteria and rationale. This approach not only bridges the interpretability gap between black-box discriminative models and brittle rule-based systems but also substantially enhances test-time scal- ing capabilities.
With the advancement of Reinforcement Learning with Verifiable Reward (RLVR), using reference answers to verify correctness on reasoning tasks has become the mainstream approach for RL train- ing of reasoning models. Most existing work employs rule-based verifiers assess the consistency between rollouts and reference answers (DeepSeek-Al) et al. 2025a et al., 2025). thereby provid- ing rewards in RL. These rule-based functions typically rely on extensive human-crafte crafted heuristic rules and often lack verification robustness (Hong et al. (2225.. Consequently, recent work has incorporated reference answers into reward models, leveraging the generalization capabilities of
model-based verifiers to improve verification accuracy (Chen et al. 2025 Liu et al. 2025a Su
al. 2025).
C.2 EVALUATION OF REWARD SYSTEMS
There are two primary approaches to evaluating reward systems. The first approach employs stan- dardized benchmarks that objectively assess reward system effectiveness by designing diverse tasks and datasets (Frick al. 2024). The second approach examines the performance of reward systems when integrated directly into downstream optimization loops, such as Best-of-N selection (Nakano
et al. 2022) rejection sampling fine-tuning (Zelikman al. 2024 Xiong et al. 2025).
to measure
their impact on generation quality and alignment.
Reward system benchmarks can be further categorized into outcome-level (Liu et al. 2024 Lam- bert et al. 2025) and process-level (Lightman et al.] 2023 Zheng et al.) (2025) suites. In constructing these benchmarks, researchers generate multiple responses to the same prompt by varying model architectures or hyperparameters. During the manual annotation phase, outcome-level benchmarks require annotators compare assign multi-point scores complete responses, emphasizing over- all preference. In contrast, process-leve benchmarks provide fine-gr ine-grained gold verdicts by requiring step-by-uue -t-epye- correctness labels for each reasoning step.
Beyond benchmark based evaluation, practical applications of reward systems serve as another com- mon assessment method. In the Best-of-N (BoN) paradigm, WebGPT (Nakano al. 2022) troduced using reward model to score N candidate answers and select the top-ranked response. Subsequent work has framed reward models as downstream rankers, for example, Celf-Connistency in chain-of-thought models (Wang et al.] 2022). where the reward model identifies the most coher- ent solution among candidates. Unlike Best-of-N, rejection sampling fine-tuning (RFT) (Zelikman et al. 2024 Xiong et al. 2025) samples multiple trajectories from the current policy, scores them using a reward model, and retains only the highest-scoring examples as silver supervision for further fine-tuning. This approach has proven particularly effective bootstrapping reasoning capabilities without requiring full preference-leariing pipelines.
During the course of this work, we also discovered several concurrent works that constructed eval- uation benchmarks for reward systems based on reference answers. Specifically, Liu et al. (2025a) proposed VerifierBench, which employs similar construction approach to our VerifyBench (easy version). However, compared to our work, they utilized fewer datasets to generate evaluation sam- ples, using 14 data sources while we employed 41 data sources. Additionally, we adopted a more fine-grained classification system for annotation. Furthermore, we propose a disagreement-basee which includes samples that are difficult
sample selection strategy to construct BerifyBBechhHHee for model- verifiers to evaluate, thereby enhancing the benchmark's difficulty. Our benchmarks also incorporate outputs from reasoning models, aligning with the mainstream forms contempo- rary model outputs and improving the benchmarks" coverage.
D DATA SOURCE
Table T provides comprehensive overview of all datasets used in constructing VerifyBench, detail- ing their respective licenses and the number of samples drawn from each. All data usage strictly complies with the terms and conditions stipulated by the original sources.
19

---

Published as a conference paper at ICLR 2026
Table 6: The datasets used and the number of samples drawn from each, including the license information of these datasets.
Domain
Source
License
# of Questions
BBH (Suzgun et al. 2023)
MIT
4520
BBEH Kazemi et al. 2025
Apache 2.0
6511
general
MMLU-pro Wang al. 2024
Apache 2.0
2000
retural_ Yuan el al.
2025
CC-BY-NC 4.0
10000
AbductionRules (Young et al.
2022
MIT
1000
anig (Bhagavatula
2019)
1000
anli (Nie et al. (020))
CC-BY-NC 4.0
1000
ARLSAT (Zhong
2021
MIT
230
bAbl15 (Weston et al.
2015
1000
bAbl16 Weston
2015)
1000
Boardgame (Kazemi et al.
2023
CC-BY-4.0
1000
clutrr Sinha et al.)
2019
CC-BY-NC 4.0
1000
FOLIO Han al. 2024
CC-BY-SA-4.0
134
hellaswag (Zellers et al.
2019
MIT
1000
logic reasonin
logicbenchBQA (Parmar et al.
2024
MIT
1000
ICGI (Parmar et al. 2024
MIT
1000
LogiQA (Liu et al.
2020)
1000
Multil-ogiEsve (Patel et al.
2024)
MIT
1000
NeuLRabduetive (Xu al.
2025a
1000
NeuLRdeduetive (Xu et al.
2025a
1000
Rinduuunee (Xu al.
2025a)
1000
ProntoQA (Saparov & He
2022)
Apache 2.0
500
ProofWriter (Taforrd et al.
2021)
1000
ReClor Yuu et al. 20199)
500
tablebench [Wu et al. 2025
Apache 2.0
886
AIME24
MIT
30
AIME25
MIT
30
asdiv-a (Miao et al. 2020)
CC-BY-NC 4.0
1218
Math Odyssey (Fang al.
2025
MIT
389
diamond (Rein et al.
2024
MIT
198
gsm8k Cobbe al. 2021
MIT
1319
math401 Yuan et al.
2023
392
mathematics Saxton et al.
2018)
Apache 2.0
3360
math math_reasoning
MATH((endryckks et al. 202T
MIT
5000
OlympiadBench-LLN He et al. 2021)
2024
MIT
675
SVAMP (Patel et al.
MIT
1000
NuminaMath-Col (Li et al.
2024)
Apache 2.0
20000
orca-math www.hh-wooooodd-ddddeee (Mitra al.
2024
MIT
10000
ArrOfProblemSolving
self-curated
7997
stackmathqa (Zhang, Yifan
2024
CC-BY-4.0
10000
DeepMath-1065 103K RL (He et al. 2025
MIT
20000
The datasets we used mainly fall into three categories: general_reasonngg logic_reasoning, and reasoingg. Among them, sense Sonserreasoning In our selection, general.raasoning also includes subsets such as common- consists of datasets, Dogicrreasoning includes 21 datasets, and math reasoning comprises datasets, among which one is collected by ourselves. All of the above datasets provide original reference answers. In dataset selection, do not differentiate between the training and test splits of the original datasets; instead, we only use their questions and reference answers.
20

---

Published as a conference paper at ICLR 2026
E LLM USAGE FOR BENCHMARK CONSTRUCTION
We carefully selected 22 LLMs from 10 families of both open-souree and closed-source models to generate com- pletions. Each model produced one completion for each question, which was collected as candidate completions for subsequent processing.
We list all the LLMs we used to generate completions for curated question in Table 7
For all models, we strictly adhered to their licenses and ensured that the generated data would not be used for any model training, but solely for evaluation purposes. For open-source models, we employed the vLLM (Kwon et al. 2023) framework for generation. The generation hyper-parameten were set to temperature=0.7 and top. p=0.95, with all other parameters kept at their default values.
Table 7: All LLMs used to generate completions in this paper. We employed multiple open- open-source and closed-source models of varying scales to generate completions, thereby enhancing the robust- ness of our constructed benchmark.
Series
Model
OpenAl
1pX-40-2024-11-2-
go--4o-mini
anthropic
Blaude337-soonee
deepseek-math
1repxeek-matt-7--instmee (Shao et al.
2024
Seepseek-math-i (Shao et al.
2024
DeepSeek
DeepSeck-Ve V3 DeepSeck-AL
2025b)
DeepSeck-R1 DeepScck-Al ct al.
2025a)
DeeSSeek-S
2025a
DeepSeek-Al
2025a
gemma-3
gemma-3-1b-it
(Gemma Team et al.
2025
gemma-3-4b-it (Gemma Team el al.
2025
gemma-3-12b-it (Gemma Team
2025
Llama-3
Llama-3.3-708-Instmucl (Grattafioni al.
2024
Llama-3-8B-Inniiee GGattattori al.
2024
Qwen2.5
Qwen2.5-7B Instruct Owen et al. 2025
Owwe2.5572B-Inntrree (Qwen 2025
Qwen2.5
Qwen2.5-Math- 1.ttl-----IInnsteee Yang
2024
Qwen2.5-Math-778 -BIInstruct (Yang et al.
2024)
Qwenn2.5-Mat-77BB-nnsss Inssruuttt Yang ct
2024
QwQ
QwQ-32B (Qwen Team 2024)
Yi-1.5
Yi-1.5-9B-Chat (Al 2025
Yi-1.5-34B-Chal (Al el al. 2025
F DETAILED BENCHMARK STATISTICS
In this section, we present additional statistics for VerifyBench and VerifyBench- Hard, including the distribution of completion source models for both benchmarks (Appendix F.1) and the distribution of token lengths (Appendix F2)
F.1 SOURCE MODEL DISTRIBUTION
VerifyBench and VerifyBench Har----ahh-ddd better understand their distribution, report the distribution of source models in both benchmarks, as shown in Table g
Hard include completions generated by different source models. To
On VerifyBench and HirffyBenc--Hrde the distribution of completions produced by different mod- els does not exhibit extreme imbalance (each model contributes roughly 2-7% on VerifyBench, and 1-12% VerifyBench-Hardd
21

---

Published as a conference paper at ICLR 2026
Table 8: Distribution of source models which generated the completions in VerifyBench and VerifyBench-Hard
VerifyBench
Verity -aad
Series
Model
# samples ratio samples ratio
OpenAI
gpt-4o-2024-1--20
91
4.55
0.00
gpt-4o-mini
79
3.95
0.00
anthropic
claude-3.7-somme
50
2.50
0.00
deepseck math
-eath 7b-instruct
107
5.35
101
10.10
deepseek-math-700
92
4.60
72
7.20
DeepSeek
DeepSeek- V3
80
4.00
0.00
DeepSeek-
101
5.05
0.00
DeepSeck-RR RI Distill-Qwen-
85
4.25
0.00
DeepSeck-R ---beckk-R--isstte Diai---iiil----e Saaaill-Qwen-32BB
92
4.60
112
11.20
gemma-3
gemma-3-
148
7.40
123
12.30
gemma-3-4b-jj
87
4.35
80
8.00
1-mma-3-12b-it
87
4.35
0.70
Llama-3
(lama-3.3-70B-Inttuue
84
4.20
0.00
Llama-3-8B-Instren
117
5.85
56
5.60
Qwen2.5
Qwen2.5-7B-Instmu
82
4.10
106
10.60
Qwennt.572B-Instruuct
86
4.30
18
1.80
Qwen2.5-Math
Qwen2.5-Math-155BBB1
86
4.30
49
4.90
Qwven25-MMath-7B-nsstnuc
83
4.15
91
9.10
Qween2.5-Mah-772-IIssttcc
77
3.85
2.10
QwQ
QwQ-32B
120
6.00
50
5.00
Yi-1.5
Yi-
92
4.60
21
2.10
Yi-1.5-34B-CHI
74
3.70
9.30
F.2 TOKEN LENGTH DISTRIBUTION OF COMPLETIONS VERIFYBENCHH-HARD)
We provide the token-length distributions of completions in both VerifyBench and VerifyBench- Hard, as shown in Figure
VerifyBench
VerlfyBench-Hard
500
3.8
Cumufattve Frequancy
0.8
400
Samples
600
0c
0.6
100
400
0k
0.5k
1k
155k
2k
2.5k
36
3.5k
4k
4k-
0.0
0k
0.5k
1.5k
2x
2.5k
3k
3.5k
4k
4k4
0.0
of Tokens (Completion)
# of Tokens (Completion)
VerifyBench
(b) VerifyBench--Hard
Figure& 4: Token length distribution of completions in VerifyBench and Verify BerifyaenchHard.
It can be that the token-length range of the completions in constructed VerifyBench is broad: includes shorter chat-style completions as well as longer RI-like completions. This diversity improves the coverage and realism of VerifyBench, making it more reliable benchmark.
22

---

Published as a conference paper at ICLR 2026
G PROMPT TEMPLATES
G.1 PROMPT TEMPLATE FOR ANSWER TYPE CLASSIFICATION
We present the prompt template used to generate types in Figure 5 (question) and {answer) are placeholders that will be replaced with actual question and answer content.
Prompt for Answer Type Classification
You are a professional LLM evaluator and now you are tasked with identifying the type answer that corresponds to problem, here is what is required of you:
1. will give you a problem and the corresponding solution, you need analyze the problem carefully and understand what objective you need to get for that problem;
2. Please choose the most appropriate type for the target (i.e. final answer) to the problem, based the objective of the give problem, the types available you include:
A Numeric values
A1 Integers(whole numbers), e-g. 351, etc.
Numerical values containing constants, e.g. 2\pi, etc.
A3 Fractions/Float numbers/Proportions, e.g. \frac(1H2), 3.123, 1:2, etc.
Numerical values containing radicals, e.g. \sqrt(3)
A5 Complex numbers, e.g. 3+4i
Angles, e.g. c20*\circ
A7 Non-decimal numbers (not base 10 numeric values), e.g. 1234_8
Multiple values with no order between them, e.g. 1,-2, 1 and
B Mathematical expressions
B1 Algebraic formulas, e.g. axtb, etc.
B2 Equations or systems of equations, e.g. ax+b=0, etc.
B3 intervals, including concatenation and intersection of intervals, e.g. (1,2) (1,2) \cup (3,4),
etc.
B4 A set, collection of elements, e.g. (1,2,3), (1,2,3), etc.B5 matrix vector, e.g.
\begin(pmatrix- \\ 5 \end(pmatrix), (1, 2) etc.
C Multiple-choice questions
Single-choice questions, choose correct option from multiple choices
C2 Multiple choice, select multiple correct choices from multiple options
Finite state selection, choose one correct state from finite number of states, e.g. True/False, yes/no, valid/invalid, etc.
D String
D1 Specific natural language representation, more concerned with whether particular word expression is mentioned correctly not
D2 Semantically consistent natural language representations, where the answer difficult to be expressed in single unified word, and there preference for comparing the semantic consistency answers E Other cases not categorized by the above categories
3. Please analyze carefully and output the serial numbers of the corresponding answer categories, you need to output your reasons for making such classification. Your output should be in the format 'explanation' is your reasoning, 'type' is the serial number of the final corresponding answer category, and you can output more than one serial number, separated by "-
legal json in ("explanation", "xxx", "type": "xo<"), where
4. You only need to care about the type of the final answer. Question:(question) Answer:(answer)
Figure 5: Prompt for answer type classification. {question) and (answer) are placeholders that will be replaced with actual question and answer content.
23

---

Published as a conference paper at ICLR 2026
G.2 PROMPT TEMPLATE FOR LLM-AS-A-JUDGE
We present the prompt template used in LLM-as-a-judge evaluation with reference answer in Figure (question ion), { [answer), and (completion) are placeholders that will be replaced with actual question, answer, and completion content.
Prompt for llm-as-a-judge
Given the following math problem and the reference answer. Judge the correctness of the answers given later, with some ability to generalize and match the form and format of the answer results. The following specific requirements are followed when judging:
1. Judge only whether the final result of the reference agree; do not consider whether there are any errors in the process. Don't t verify the correctness of the answer by yourself, please only refer to the reference correctness of the answer.
and the
be judged
for the
2. The reference answer and the answer to be judged only need to be essentially the same, ignoring irrelevant details such as units, symbols, whether or not to approximate, the of expression in the answer. The two answers considered to be consistent they equivalently transformable.
3. All your analysis answer must be in English.
4. Please analyze the judged answer and try to compare with the reference answer. At the end of all analysis, give the result of the judgment on an extra line at the end of the answer in the form "Final Judgment: Yes/No".
Problem: (question)
Reference Answer: (answer)
Solution to be evaluated: (completion)
Figure 6: Prompt for LLM-as-a-judge evaluation. {question] [answer], and (completion) are placeholders that will be replaced with actual question, answer, and completion content.
G.3 PROMPT TEMPLATE FOR
WITHOUT REFERENCE
We present the prompt template we used LLM as-ae evaluation with reference answer in Figure 7 (quest onn and (comples ) are placeholders that will be replaced with actual question completion content.
Prompt for lim-as-a-judge without Reference Answer
Given the following math problem, please judge the correctness the answers given later. The following specific requirements are followed when judging:
1. All your analysis answer must be in English.
2. Please analyze the math problem and the answer and try to tell whether the given completion is correct answer. the end of all analysis, give the result of the judgment on extra line at the end of the answer in the form Final Judgment: Yes/No ".
Problem: (question)
Solution to evaluated: (completion)
Figure 7: Prompt for LLM-as-a-judge evaluation without reference answers. fquest ion) and (completion ion] are placeholders that will be replaced with actual question and completion content.
24

---

Published as a conference paper at ICLR 2026
H DETAILS OF HUMAN ANNOTATION
We present comprehensive details regarding our human annotation process: (1) the composition of our annotation team, including the knowledge backgrounds of annotators, and (2) the annotation interface design (for annotators and for meta-annotators), encompassing annotation instructions and user interface specifications.
H.1 INSTRUCTION FOR ANNOTATORS
Before initiating the manual annotation process. conducted a 60-minute video training session for all annotators. After the training, we also provided written instructions detailing the annotation procedure, which were distributed to all annotators. In addition, these instructions were made cessible on the annotation interface itself, enabling annotators to consult them real time. The following is the instruction text provided.
Instruction for Annotators
### What should do
**Step 1 1 Check the answer type. The page has been ane-tnnotteed, please check whether this annotation matches the actual situation. Please select the most appropriate option from the dropdown list, where option E is not suitable for VerifyBench. If you select option E, you can skip step and directly click the Next Button.
**Step 2:** Check answer correctness. A total of 4 corresponding answers displayed, each answer has been pre-annotated for correctness. Please check whether this annotation matches the actual situation, and if it doesn't ' match, please make the appropriate corrections. **Step 3: ** Click the Next Button! You have successfully completed the annotation for this case. Thanks for your contribution!
H.2 ANNOTATORS
Our annotation consists of 6 members, including 4 annotators and 2 meta-annotators. The meta-annotators are responsible for resolving disagreements that arise during the annotation process.
Among the four annotators, all of whom have engaged in research on LLM reasoning for over three months. The two meta- annotators are current PhD students who have conducted research LLM reasoning for more than one year. Among the two meta-annoke annottoooss,, one was responsible for resolving disagreements between general reasoning and math reasoning categories, while the other handled conflicts between general reasoning and logic reasoning categories.
undergraduate student and the other three master's students,
Our primary annotator team consists of the following members:
Annotator febd: Undergraduate student with 5 months of experience in LLM reasoning
search.
Annotator cfaa: Master's student with months of experience in LLM reasoning research.
Annotator ebed: Master's student with 6 months of experience in LLM reasoning research, Annotator ffee: Master's degree holder with I year of experience in LLM reasoning research.
Our meta-annotator team consists of the following members:
Meta-annotator ebed: Doctoral student with year of experience in LLM reasoning research, specializing reasoning and logical reasoning domains.
Meta-annotator feea: Doctoral student with year of experience in LLM reasoning research, specializing reasoning and mathematical reasoning domains.
H.3 UI DESIGN FOR HUMAN ANNOTATION PROCESS
We developed web-based annotation interface using Streamlit to facilitate human labeling. All annotations were conducted through the web interface, with different annotators distinguished by unique user parameters. We constructed two distinct interfaces: for primary annotators and
25

---

Published as a conference paper at ICLR 2026
another for meta annotators. Primary annotators directly labeled the data, while meta annotators solved disagreements between primary annotators through our double-check verification mechanism and made final labeling decisions based on the annotations from both primary annotators.
H.3.1 PAGE FOR PRIMARY ANNOTATORS
We designed an annotation interface for annotators that displays the current annotation progress. By clicking on interface elements, annotators can view the complete data in JSON format to make informed judgments. The interface presents one question, one reference answer, and four corre- sponding completions
time. Annotators label the answer category using dropdown components and mark the correctness of the four completions using checkboxes. Our annotation interface sup- ports LaTeX formula rendering. enabling annotators to perform mathematical problem annotation in more readable format. For VerifyBench annotation, the interface displays pre-annotation results, while for CerifyBench-Hare annotation, no pre-annotation results are shown. The interface incor- porates an invalid button enable annotators to quickly mark samples unsuitable for VerifyBench Any sample marked as invalid by any annotator is excluded from subsequent annotation processes. Additionally, skip button is implemented to allow annotators bypass samples they cannot eval- uate with confidence. Figure & shows the actual annotation interface for primary annotators.
Question
Unadl on san Ressonn Clearooto sedis pois resen 1000 ald" ter abwo E hawe enertte woe duver acc aboul
Answer
hargu-gr repooneta
reproaati io
cosec prourn tee eelovneti tradin
LEU kelts Aebs seveesinng
thehe mecho bedirrd Ae- BRrveee Cintoen
fow protowe
oour poo now the
co-dimenden
#I FELiBaant Eefteu caimau Thenrhetio
P.p - oe ow the pertel
wethotee moseeen weking
Eae enc tnwilomie Kigeran dhvile Hell)
Yoda hyday
Final Answer
&
(N Nu/ Pleeeeneee ia siviige
6.50 airaeeate thedtanee ailarrs uit shre
devesion muablc connet
fan 2f
endinoormet ti tieernt heeloomte alperii cil-
saii jest wall yourr ressan on (
Conclusion
pofey
somemmn fige kydle
Fely Nudto
Figure Annotation interface for primary annotators, one sample per page for annotation.
H.3.2 PAGE FOR META-ANNOTATORS
The interface designed for meta-annotators differs slightly from that of primary annotators. The key distinction lies the meta-annotators' ability to view detailed annotations from primary an- notators, with disagreements between annotators clearly displayed on the page to facilitate final judgment based on the primary annotation information. Furthermore, we designed domain-based filtering element for meta-annotators, enabling them to select domains aligned with their knowledge background and expertise. Other page logic remains identical to the primary annotators' interface. Figure illustrates the actual annotation interface for meta-annotators.
26

---

Published as a conference paper at ICLR 2026
Question
wenboo herorte
Answer
Ceren
Greet
worosar's Yue
Nondd 2
semen dbyct
Chaple du mvvyy coead mnsantss
Nabes Thiiee eupetacing un
Mnvermmnd dhoal charg
arpess unforc
chepecchent Aecsus fashiee
daspero
banc prstess budfil
Conclusion
the seceee" pur Cialaciioo pullanc
heuneige vici selere
leae kuggch rapae cusinnstctoon rigel
desic sanrdlyya bonth
riear ca abo pett poii our acc charge
bothureecccoeeeee of
coorties.
proautee fefore eneernn
pruent comestaa dus sbabt
melemunn alber
Figure 9: Annotation interface for meta-annotators, one sample per page for annotation.
I FULL TAXONOMY OF VERIFYBENCE AND VERIFY
During human annotation, adopted a finer-grained labeling scheme. Beyond the four main cat- egories mentioned the text (Numeric Values, Expressions, Multi-choice, and String), we further annotated more detailed subcategories within each type for analysis. Specifically, Numeric Values includes 8 subcategories, Expressions includes 5 subcategories, Multi-choice includes subcate- gories, and String includes 2 subcategories. The descriptions of all subcategories presented in Table 9
J CORRELATION EXPERIMENTAL DETAILS
Our correlation experiments consist of two parts: we conducted both Reinforcement Learning (RL) (Section J.1) and Rejection Sampling Fine-tuning (RFT) (Section J.2) to validate the usability of VerifyBench.
J.1 REINFORCEMENT LEARNING EXPERIMENTS
J.1.1 SETUP
Training. For RL experiments, we adopted Qwen2.5-333 --3tstttttt (Owen et al. 2025) as the base model and conducted training with the SeppMtth-103k (He et al. 2025 dataset. For the RL algo- rithm, we employed GRPO (Shao et al. (2024) optimization, setting the global batch size to 512 and the mini-batch size to 128. For each query, sampled 16 responses compute the group relative
27

---

Published as a conference paper at ICLR 2026
Table Full taxonomy and their description of VerifyBench and VerifyBench- Hard
Label
Type
Sub-Label
Sub-Type Description
A
Numeric Values
A1
Integers(whole numbers). e.g. 2, 351, etc.
A2
Numerical values containing constants, e.g. 2T,, etc.
A3
Fractions/Float numbers/Propottion, c.g. 3.13, etc. Numerical values containing radicals, e.g. v Complex numbers, e.g.
A4
AS
A6
Angles, c.g. 120"
A7
Non-decimal numbers (not base 10 numeric values), c.g. 1234g Multiple valucs with order between them, e.g. 1,- and 2, etc.
A8
B
Expressions
B1
Algebraie formulas, e.g. + b, etc. Equations or systems equations, e.g. + 0, etc. Intervals, including concatenation and intersection intervals, e.g. (1, 2] (1,2) (3, 4), etc.
B2
B3
B4
A set, collection of elements, e.g. (1,2,3), (1, etc.
B5
A matrix or vector,
16
(1,2) etc.
5
C
Multi-choice
C1
Single-choice questions, choose correct option from multi-
ple choices.
C2
Multiple choice, select multiple correct choices from multiple
options.
C3
Finite state selection, choose correct state from finite num- ber states, e.g. Truc/False, yes/no, valid/invalid, etc.
D
String
D1
Specific natural language representation, whether particular word or expression is mentioned correctly or not
concerned with
D2
Semantically consistent natural language representations, where the answer is difficult to be expressed in single uni- fied word, and there is preference for comparing the semantic consistency answers
Others
Other not categorized by the above categories
advantage. The maximum prompt length was set to 1024, and the maximum response length was to 7168. For rollouts, we utilized the vLLM (Kwon al. 2023) engine with temperature fixed at 1.0 and top-p 1.0. Training was performed with the veRL (Sheng et al. 2025) framework using FSDP the backend. The learning rate during training was set to le-6, with 20 warmup steps. For the LLM verifier, we deployed services through SGLang (Zheng al. 2024) and performed verification via remote calls. All RL experiments were conducted on H200 GPUs.
Evaluation. For evaluation, leveraged the functionality of the veRL framework conduct assessments on checkpoints every 10 training steps during RL training. To reduce evaluation vari- ance, we the temperature 0.7 and top-p to 0.95, sampling each test example 8 times and then computing the average accuracy.
J.1.2 EXPERIMENTAL RESULTS
We applied a sliding-window window smoothing method to the curves presented in the main text. In Fig- ure 10, we show the original curves before smoothing (light-colored curves).
J.2 REJECTION SAMPLING FINE-TUNING EXPERIMENTS
J.2.1
SETUP
Training.
For the rejection sampling fine-tuning experiments, we used Llama-3.1-8B (Grattafion) the base model for SFT. The learning was set constant value of le-5. Training
al. 2024)
28

---

Published as a conference paper at ICLR 2026
MATH500
AMC23
MinervaMath
28
66
Accuracy
40
26
64
38
36
24
34
32
50
100
150
200
50
100
150
200
50
100
150
200
Training Step
Training Step
Training Step
1lama-3.:33--nnae. (60.95)
Qwen3-1.7B (81.10)
Qwen3-4B (92.00)
opt-oss-20b (94.8)
math-verify
Figure 10: The performance(%) (6) of RL across different LLM judges which have various performance on VerifyBench.
was conducted using the Megatron-LMM framework, with global batch size of 256 and context length of 4096. To accelerate training, packed the training samples and trained for one epoch in total. All training experiments were conducted on 32 Ascend H910B-64G GPUs.
Evaluation. For evaluation, we used the (KL.M(wwon al. 2023 framework for inference. To reduce evaluation variance, set the temperature to 0.7 and sampled each test example 16 times, then computed the average accuracy. All inference were conducted on NVIDIA A100-80G.
J.2.2 EXPERIMENTAL. RESULTS
In our experiments, we applied rejection sampling to implement reference-based reward systems. For each question in the GSM8K and MATH training sets, generated 64 candidate completions using Wwen2.5-Math-7B-Imsse responses were subsequently filtered by verifier models with varying performance levels, on Ver- ifyBench: Llama-3. 1--B-- 1-8B-Inst Qwen3-4B, Qwen3-1.78 and gpt-oss-20b, 20b, and aule-based ver- ifier, math-verify. Only completions consistently verified as correct were retained form the SFT training data.
(Yang et al. 2024) with a sampling temperature of 0.7. These
gsm8k
MATH500
SVAMP
44
Accuracy
85.5
85.0
83
40
84.5
38
50
100
150
200
250
300
50
100
150
200
250
300
50
100
150
200
250
300
Training Step
Training Step
Training Step
Llama-3 1-88-Instructi73.55
Qwen3-4B(92.00g
got-os-20b/94.c 80)
math-verity
Figure 11:
The performance($6) (6) of RFT across different LLM judges and rule-based math-verify which have various performance on VerifyBench.
The resulting models were evaluated on multiple mathematical reasoning benchmarks. including GSM8K (Cobbe et al.) 2021), MATH500 (Hendrycks et al.) 2021 Lightman et al.) (023). and SVAMP (Patel et al. (021). As shown Figure [1] on GSMSK, MATH500, and SVAMP. us- ing gpt-oss-20b, verifier that achieves higher accuracy VerifyBench consistently outperforms Llama-3. 3lama-3.1-8B-- SB-Inutruct a lower-scoring verifier, under the same training steps. This highlights the strong alignment between VerifyBench and practical usage. Our benchmarks serve reliable tools to guide the development of reward systems, leading to more effective training and improved model performance.
29

---

Published as a conference paper at ICLR 2026
K EXAMPLES OF VERIFYBENCH
In this section, we provide representative samples from the four answer-type categories in Verity- Bench to facilitate clearer understanding of the benchmark we constructed. Due space limita- tions, we omitted part of the text and used <omitted> as placeholder.
Figure K. presents sample from the Numeric Values category, Figure K.2 shows sample from the Expression category. Figure K.3 illluststas a sample from the Multi-choice category, and Figure K.4 displays a sample from the String category.
K.1
AN EXAMPLE OF VERIFYBENCH (ANSWER TYPE: NUMBRIC VALUES)
VerifyBench (Answer type: Numeric Values)
Question:
How many positive and negative integers is $12$ multiple of?
Reference answer:
The number $12$ is multiple of $-12, -6, -4, -3, -2, -1, 1, 2, 3, 4, 6, $ and $12, $ for a total of (bboxed(12)5 integers.
Completion:
We want to find the number of positive and negative integers that divide $12$.
The positive integers that divide $12$ are the positive divisors of $12$. The divisors of $12$ are the integers that divide $12$ without leaving a remainder.
We can find the divisors of $12$ by listing pairs of integers that multiply to $12s.
$1 Atimes 12 = 125
$2 Atimes 12$
<omitted>
The number of positive integers that divide $12$ 6. The number of negative integers that divide $12$ is 6.
Final Answer: The final answer is $\boxed (6)$
Golden Label: x False
K.2 AN EXAMPLE OF VERIFYBENCH (ANSWER TYPE: EXPRESSION)
VerifyBench (Answer type: Expression)
Question:
Consider two lines parameterized by: \[
\begin (pmatrix) V (pmatrix); \begin(pmatrix) 1 \ 2 (pmatrii:) + t Abegin (pmatrix) \i -3 (endpixiixi/// x A\ y (ema(pmatrix) = \begin(
MandAn\[An
pmatrix) 4 \\ 5 (matriix) + u (pegin(pmatrix] \l -1 Nend
(pmatrix) \]
30

---

Published as a conference paper at ICLR 2026
Determine the coordinates of their intersection point.
Reference answer:
For the first line, A[ (pmair(p x \\ y (end(pmatrix) 1 W 2 (mnatrixx + t \begin (pmatrix) 2 \
-3 \end(pnatrix) \begin(pmatrix) 2t \\ \end pmatrix) VI For the second line,
<omitted>
2 3 (-6) (enatriiii (pegtnippaatrix 1 12 \\ + 18 \
end(pmatrix) = (oagin(pmarixx -11 Vi 20 \end ( (mmatrx)) \]
Sc, the intersecti point is ( \boxed \begin (pmatrix) -11 W 20 Nend (pmatrix)) \).
Completion:
Okay, have to find the intersection point of two lines given in parametric form. The first line is parameterized as:
<omitted>
Thus, the coordinates of the intersection point are A(boxed ((-11, 20) HV)
Golden Label: True
K.3
AN EXAMPLE OF VERIFYBENCH (ANSWER TYPE: MULTI-CHOICE)
VerifyBench (Answer type: Multi-choice)
Question:
Given that point SP$ lies on the circle $C: x (2]]-x+y"" (2)-6) +14=0$, and the 1ine $AB: x+y-2=0$, then which of the following statements is true?
Line $AB$ intersects circle SCS.
B: Line $AB$ is disjoint from circle $CS.
The distance from point $P$ to line SABS is greater than frac(1)
The distance from point SP$ to line SABS is less than $5$.
Reference answer:
To analyze the relationship between circle $C$ and line SAB$, we first need to rewrite the equation
<0mitted>
And since the maximum distance is $2\sqrt (2) + 2$, which is less than $5$, statement D is also true. Therefore, the correct choices are $\boxed(BCD)5.
Completion:
To analyze the problem, l break it down step by step.
31

---

Published as a conference paper at ICLR 2026
### step 1: Rewrite the equation of the circle V( C \) The given equation of the circle is: AI x 6x y 6y + 14
0. \I Complete the square for both A x W
<omitted>
which is: A[ 2\sqrt 2. \J Numerically: VI (2srrt(2) \ approx 2. 2.828, \quaad 2 (ssqrt(2) (2) 2 Naporox 0.828, \quad S (2) \аpprox 4.828. \J Thus, the distance from ( V to the line ( AB V) is always greater than \ \frac(1H2| A) and less than ( V. Therefore, both **statement C** and statement D** are true.
### Final Answer: The true statements are: **Bax: Line (< AB \) is disjoint from circle V C V)
**C**: The distance from point \ P \) to line A AB \) is
greater than \( (1rac(1) V).
**D*x: The distance from point \O \) to line A AB \ is
less than ( 5 V
Golden Label: True
K.4 AN EXAMPLE OF VERIFYBENCH (ANSWER TYPE: STRING)
VerifyBench (Answer type: String)
Question:
Context:
Brian is swan.
Bernhard is a frog.
Lily is rhino.
Bernhard is white. Greg is swan. Brian is gray.
Greg is gray. Lily is yellow.
Julius is frog.
What coler is Julius?
Reference answer:
white
Completion:
The color of Julius is not explicitly mentioned in the provided context. Therefore, we cannot determine what color Julius is based the information given.
Golden Label: False
32

---

Published as a conference paper at ICLR 2026
L EXAMPLES OF VERIFYBENCH-HARD
In this section, we present hard-to-verify samples from the four different answer-type categories in VerifyBench-Haite enabling a clearer understanding of the construction and characteristics of -erifyBench-Hard. Due to space limitations, we omitted part of the text and used <omi tedd> as placeholder.
Figure L.I presents sample from the Numeric Values category, Figure L.2 shows sample from the Expression category, Figure L.3 illustrates sample from the Multi-choice category, and Figure displays sample from the String category.
L.1
AN EXAMPLE OF VERIFYBENCH-HARD (ANSWER TYPE: NUMERIC VALUES)
VerifyBench-Hardd (Answer type: Numeric Values)
Question:
Given the function $f (x) Acos Asin xs, where $E" (x) $ is the second derivative of the function (x) find the value of $5 (\frac()p1) $
Reference answer:
First, let's s find the first and second derivatives the function $x)
<ommited>
Rearrange the fraction match one of the answer choices:
$$- \boxed[-\fraci!
(3)
Completion:
To find the value of \ (x the function \( (x) \cos x Asin A), need to follow these steps:
\) for
1. **Find the first derivative \ El (x) *)***
The function \ fx)) \) is given by:
A[
f(x) \cos x - Asin
A]
<ommited>
Thus, the value of \ E
\) is:
\[
\boxed(\frac(
Asqrt (3)) (2))
Golden Label: False
33

---

Published as a conference paper at ICLR 2026
L.2 AN EXAMPLE OF VERIFYBENCH- (ANSWER TYPE: EXPRESSION)
VerifyBench-Hard (Answer type: Expression)
Question:
$A$ and $B$ are constants such that
(x+2) (x $$Find (xt1] \xrac(A)(x22).55
(xx
Reference answer:
We have that
\frac(4x+5) (x+1) (x-2) ]=) (x+2) (Mfrac(B) $sMultiplying both sides by $ (x+1) (x-2) gives
954x+5-A(x*1) (x- SSwiihh we can expand to get (BB)) comparing coefficients, we see that $A
+B=4$ and $A-2B=5$. can solve these equations get $A=1$ and $B=3$. Then
$$ 3#Nbegin(alligmed] (B+a1
(x+11 (x+1)
& - (X
& ( (x+1) (x-2) H).
\end(aligned)55
Completion:
To solve the given problem, we start by expressing the right- hand side of the equation with common denominator:
<ommited>
This expression is already in its simplest form, the final answer is:
M
(x-1)
(x+1) (1)(221)
AJ
Golden Label:
True
L.3
AN EXAMPLE OF VERIFYBENCH-HARRD (ANSWER TYPE: MULTI-CHOICE)
VerityBench-HHard (Answer type: Multi-choice)
Question:
Context:
In the criminal justice system, investigators aim to solve crimes. They knew solid detective work led to clues, and forensic analysis provided evidence. However, some crimes went unsolved, lacking clues evidence. They knew if there were sufficient resources, detection would be thorough. Question:
The forensics tean found evidence. From the context, did the investigation tean have sufficient resources? yes or no?
34

---

Published as a conference paper at ICLR 2026
Reference answer:
no
Completion:
The context does not explicitly state whether the investigation team had sufficient resources or not. It only mentions that solid detective work and forensic analysis are involved in solving crimes and that some crimes unsolved due to lack of clues evidence. So, the answer is:
No, the context does not provide enough information determine if the investigation team had sufficient resources.
Golden Label: True
L.4 AN EXAMPLE OF VERIFYBENCH-HARE (ANSWER TYPE: STRING)
35

---

Published as a conference paper at ICLR 2026
Verify Hard (Answer type: String)
Question:
Sort the following words alphabetically: List: wintertime mobster ugh brown fleabane longish ipsilateral cortex hair diminutive trapezoidal guy richfield highfalutin borough foot delphinium cosec abstract.
Reference answer:
abstract borough brown cortex cosec delphinium diminutive fleabane foot guy hair highfalutin ipsilateral longish mobster richfield trapezoidal ugh wintertime
Completion:
Here's s the list of words sorted alphabetically:
1. Abstract
borough
3. Brown
Cosec
5. Delphinium
6. Diminutive
7. Fleabane
8 Guy
9. Hair
10. Highfalutin
11. ipsilateral
12. Longish
Mobster
14. Richfield
Trapezoidal
16. Ugh
17. Wintertime
Golden Label: False
M ERROR CASE STUDY
To facilitate a deeper understanding of the error patterns exhibited by current models on Verify- Bench, we present four representative failure cases of gpt-oss-20b. These examples span multiple answer types, illustrating the breadth of scenarios in which the model's verification judgments break down.
Case 1 (Answer type: Numeric Values)
Question:
Zoe picked five apples from her tree. Now the tree has six apples still on it. What's s the difference between the number of apples Zoe picked and now the tree has?
Reference answer:
Completion:
36

---

Published as a conference paper at ICLR 2026
To solve the problem, need to determine the difference between the number of apples Zoe picked and the number of apples still on the tree.
First, let's s identify the number of apples 2oe picked. According to the problem, Zoe picked apples.
Next, we need to determine the number of apples still on the tree. The problem states that after Zoe picked the apples, the tree still has 6 apples.
New, calculate the difference between the number of apples Zoe picked and the number of apples still on the tree. This can be done by subtracting the number of apples still on the tree from the number of apples Zoe picked:
M 5 -1 WJ
Therefore, the difference between the number of apples Zoe picked and the number of apples still on the tree is \(bboxed (-1}))
Judgment:
We need to judge if the evaluated answer's final result matches reference answer. Reference answer: 1. Evaluated answer gives -1. Are they considered essentially same? No, vs -1 not equivalent. So mismatch. Therefore judgment No. The evaluated solution computes the difference as (5-6 -1V giving a final result of \--V) The reference answer states the correct result ((1)) Since (-11) and (1V) are not equivalent (they differ in sign), the evaluated answer does not match the reference answer.
**Final Judgment: No**
Golden Label: True
Case 2 (Answer type: Expression)
Question:
A 1ine passing through the left focus $F 1$ of hyperbola and perpendicular to the real axis of the hyperbola intersects the hyperbola at points $A$ and $B$. If there exists point $Cs on the line of the imaginary axis of the hyperbola such that (Avec(AC) Acdot \vec(BC) 0$, then the range of the eccentricity $e$ of the hyperbola is
Reference answer:
Let the equation of the hyperbola be SXfrac(x"2) (b²)) 18, with SAVleft \frac(b 2) (a) \right) s, $B\ left (-c, (a) and (C, $)
Afrac(
From $\vec Nedot \Bec 0$,
37

---

Published as a conference paper at ICLR 2026
we get $ = \brac(b"4) (a"2) c²2 \gegslant 08, and $e \ geqslant \fraac)sqrt(55) (){2).
Therefore, the range of the eccentricity $e$ of the hyperbola is \boxeede \geqslant Afrac (1ra((ss 1):2)}..
Completion:
To solve the problem, we need to analyze the given conditions and use the properties of the hyperbola. Let's start by defining the hyperbola and its properties.
<ommited>
Therefore, the range of the eccentricity \ V) is: \I [beft (sqra() (5) Ainfty Aright) }. \
Judgment:
We need to compare final result with reference: reference says e \ge (2qrt5+1)/2 /2 \approx (2.236+1) 2211618.. The evaluates answer says e Age sqrt (grt(((3+rrrt))) /2) Compute: sqrt.5aaprro22 .236, ++2.236=5.236 ((3+sqrt5)/2) equals (sqrt5+1)/27 Let's check algebra: (sqrt5 /2) squared = (5+2sqrt5+1)/4 = (6+2sqrt5)/4
/2=2.618, sqrt sgrt'approx1. .188. Indeed sqrt
(3+sqrt5)/2. So sqrt ((++sqrt5) /2) (sqrt5+1) /2 (positive). So the evaluated answer's s range is e \ge sqrt (3+tggrt5)/2 /2) which equals (sqrt5+1) /2. So they match. The answer also gave interval [that, \infty). Reference gave e \ge that. So consistent. Therefore final judgment Yes.
The evaluated answer ultimately states that the eccentricity must satisfy
M
e Age (ffrt )2)), AJ
and notes that this is the lower bound for \ee) Using the identity
M[
the lower bound is exactly the same as the one given in the reference answer, which is \ ferkge\\rac///grr55511/1 answers therefore give the same range for the eccentricity ( the evaluated answer additionally writes the range (IV, fract (sgrt5+1) \infty) which is equivalent to the reference's \eegge Tnggelfrac fracf (s
Both
**Final
Judgment: Yes**
Golden Label: False
38

---

Published as a conference paper at ICLR 2026
Case 3 (Answer type: Multi-choice)
Question:
Given the function $f (x) = (2x) [x+1] + 26, with its domain SM$ and its range as $[1, 21$, the correct statement
(s) among the following is/are
(-linfty, 119; (3) ΩM \subseteq (-) infty, 119; (4) $0 Vin MS; (5) $1 Vin MS.
(1) $M [0, 21 (2)
Reference answer:
Since (x) 2 (2x) 2 (x+1) 2]5,
(2 1) Vin [1,
$\therefore 2'x Vin [-1, 119, which means $2"x Vin [0, 21$ $Atherefore x \in (-\infty, 118, which means the domain of the function (x (2x)) (x+1) 2$ is $ (-\infty, 11$. When the function's minimum value is 1, it only happens at $x =0$, thus, (4) $0 Vin MS is correct, When the function value is 2, it only happens at $x=1s, thus, (5) $1 Vin M$ is correct Also, it must be that $M \subseteq (-Vinfty, 11;; hence, (3) is correct
when $M = [0, 1Js, at this time, the range of the function is $11, 21$, $itherefore correct, hence, (2) is incorrect;
(-linfty, 1]$ is not necessarily
When $x=2s, the function value is 10, hence, (1) SM [0, 2 is incorrect.
In summary, the correct statement numbers are \boxed (3), (4) {
Completion:
We tasked with analyzing the function \ f (x) (2x))
2 (x+x) 2 V,, its domain ( M M,, and its range \ [1, 2] V Let's carefully evaluate the given statements.
<ommited>
Now we evaluate the given statements:
1. \ [0, 2] Wr **aase since \l M [c, 1] M,, not V
[0, 2] A)
2. \ M = (-\infty, 11 V:: **alse**, since \ M [0, 1] M
not ( (-\infty, 1] \).
3. \ M \subseteq (-\infty, 1] A): Txuuee,
Judgment:
We need to judge the solution to be evaluated (the "Solution to be evaluated") against the reference answer. The reference answer says the correct statements
(3), (4), (5). The solution to be evaluated says: It goes through steps, and at the end it says: "Thus, the domain is [0, 11." Then it says: "Now we evaluate the given statements: 1. = [0, 2]: False, (-\infty, 1]: False, 3. \subseteq (-\infty, 1]: True, 4. 0 Vin M: True, 5. 1 Vin M: True." So the solution to be eval lated says the correct statements are (3), (4), (5). That
39

---

Published as a conference paper at ICLR 2026
matches the reference answer. So the final judgment is Yes. We need to output the final judgment line. The solution to be evaluated concludes that the correct statements are (3), (4), and (5), which matches the reference answer's conclusion that the correct statements (3), (4), and (5). Therefore, the answers agree.
Final Judgment: Yes
Golden Label: False
Case 4 (Answer type: String)
Question:
Context:
Greg is a lion.
Julius is swan.
Lily is a lion. Greg is yellow. Bernhard is a lion. Brian is lion. Brian is yellow. Julius white. Bernhard is yellow.
What color is Lily?
Reference answer:
yellow
Completion:
According to the context, Lily is a lion, and there is no information about her color. Since Greg, Bernhard, and Brian are all lions and are yellow, it is possible that Lily is also yellow, but it is not explicitly stated.
Judgment:
We need to judge if the solution to be evaluated agrees with reference answer. Reference answer: "yellow". Evaluated sclution says: "it is possible that Lily is also yellow, but it is not explicitly stated." This does not assert yellow definitively. The final result is not "yellow". So does it agree? The requirement: judge only whether final result of reference answer and answer to be judged agree. The evaluated answer's final result is ambiquous, not a definitive color. So does that match "yellow"? No. Therefore judgment: No. The evaluated answer does not definitively state that Lily's color yellow; it only suggests that yellow is possible and notes the lack of explicit information. The reference answer definitively states "yellow." Because the evaluated answer's final result is not the same definitive color, it does not agree with the reference answer.
Final Judgment: No
40

---

Published as a conference paper at ICLR 2026
Golden Label: True
41


Reviews:
[{"review_id": "TJTkbOk29M", "reviewer": "Reviewer_R9TK", "summary": "This paper presents VerifyBench and a more challenging version VerfiyBench-Hard to assess reference-based reward systems used in training large reasoning models (LRMs). The authors develop a step-by-step pipeline to create these benchmarks. The authors evaluate a large pool of large language models (LLMs) and reward models (RMs) on these benchmarks.", "questions": "1. The authors demonstrate that better verifier models lead to better LLMs when used in RL training. What is the relationship between the verifier model's performance and the resulting LLM's performance? I am wondering whether weaker/smaller LLMs typically require stronger verifiers while stronger/larger LLMs can work well with weaker verifiers, such as direct matching with ground truth answers. Is this true?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The proposed benchmarks, VerifyBench and VerifyBench-Hard, are well-designed to evaluate the capabilities of reward models in assessing the correctness of reasoning steps in LLM-generated outputs. These benchmarks could be valuable resources for the research community.", "weaknesses": "1. The motivation of the paper is somewhat questionable. Typically, the correctness of responses can be directly verified against ground truth answers without the need for reward models. Using a reward model to verify correctness can achieve higher accuracy compared to directly comparing with ground truth answers, as demonstrated in Figure 12, 13, and 14. However, it is unclear whether the added complexity of using reward models is justified given the performance gain.\n2. The benchmark appears to be relatively easy for current state-of-the-art LLMs and RMs. As shown in Table 2, the best-performing LLMs and RMs achieve over 90% accuracy on VerifyBench, leaving limited room for improvement. In the evaluation, The verifier models are esstentially tasked to compare the consistency between the reference answer and the model-generated answer, which should be relatively straightforward for current LLMs.\n3. In Section 5.2 and Appendix H, the authors demonstrate that better models on VerifyBench is a better verifier in RL training. However, it is unclear whether these models are better than directly matching with ground truth answers under the same computation budget.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "isKPOYBNLp", "reviewer": "Reviewer_8czc", "summary": "This paper presents VerifyBench, a benchmark designed to evaluate the accuracy of reference-based reward systems. Specifically, given a question, reference answer and a generated completion, the benchmark tests if models are able to correctly classify the generated completion as correct/incorrect. A well-designed pipeline consisting of human annotation and large-scale model generations is used to curate VerifyBench and VerifyBench-Hard, a harder variant of the same task. Evaluations on the benchmark show that current models (especially smaller ones), have significant room for improvement on the task.", "questions": "1. What is the final source (model) distribution of completions? Models have a bias to prefer their own completions, so all completions coming from a small number of models will be undesirable. \n\n2. Do completions include long reasoning traces (chains of thought) or only the final summarized solutions? Reasoning models generate long thinking traces involving complex behavior such as self-reflection/verification and backtracking. I personally do not think these traces should be part of the completion, instead the completion should be a final summary of the solution (Eg. Reasoning traces of GPT5 are not visible, only the final solution is presented to the user). I am curious to hear how authors approached this.\n\n3. What does the length distribution of the completions in the benchmark look like?\n\n4.  I liked the experiment of training with the best verifiers on the benchmark. Another experiment, which is much easier to run, is to use verifiers for rejection sampling (best-of-N). This can be compared to pass@N (using a perfect verifier). The difference in performance between the two methods would be another indicator of the strength of a verifier (an ideal verifier should closely match the pass@N curve). I believe this is also a cleaner experiment, as training might have second-order effects.   \n\n5. My main doubt is on the performance of reference answers. If reference answers are provided to the model, then isn’t the task simply just to extract the final answer from the completion and check equivalence to the reference answer. In domains like Math, this is exactly what a symbolic parser (e.g., Math-verify) does. This might be a harder task in non-math domains, so I am curious to understand the benefits of this in tasks like Math. In general, if one of the applications of these verifiers is RL training in domains where ground truth answers are unavailable, then a more practical evaluation is one without reference answers. Similarly, rolling out long traces from verifiers is also not practical for RL training, especially if slightly worse but significantly cheaper symbolic parsers are available. \n\n6. Adding on to the previous point, if verifiers are to be used for inference scaling (best-of-N), then reference answers will not be available. Perhaps, the best thing to do is to expand the benchmark to have a reference and non-reference setting? This would be able to get the best of both worlds. \n\n7. How exactly is controlled downsampling performed, and does it distort the difficulty/accuracy distribution of the dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, smooth, and easy to follow, with clear structure and presentation.\n\n- The data curation pipelines are comprehensive and well-designed. \n\n- The benchmark tackles a challenge that is not addressed in existing benchmarks.", "weaknesses": "- The final source distribution of completions may be biased toward specific models, which could undermine benchmark diversity. More analysis on this would address this. \n\n- Some more clarifications on the data curation pipeline will further increase clarity of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C1LJE0oLOz", "reviewer": "Reviewer_ApHj", "summary": "This paper introduces VerifyBench and its more challenging variant, VerifyBench-Hard, as two new benchmarks designed for the systematic evaluation of Reference-Based Verifiers. The two benchmarks cover four distinct answer formats and three different reasoning task types. Through extensive experimentation involving Rule-Based Verifiers, LLM-as-a-Judge methods, and Model-Based Verifiers, this study reveals that while current state-of-the-art model-based approaches perform well in general testing, VerifyBench-Hard still poses a significant challenge to them. The paper further explores the performance of traditional ORMs and experimentally confirms that a higher score on VerifyBench positively correlates with the greater performance improvement the verifier provides during RL training.", "questions": "1.RL training with Qwen-4B as the verifier clearly outperforms training with weaker models, while OSS does not yield additional gains. Does this suggest that there is in fact some “threshold” that determines whether a verifier is basically reasonable, and that once this threshold is reached, further gains become minor?\n\n2.In the RFT experiments, were models stronger than Qwen-4B (e.g., OSS used in the RL experiments) also evaluated? If so, how did they perform?\n\n3.Can you provide some challenging examples from VerifyBench and illustrate how the models judged them correctly or incorrectly? In particular, for the OSS model, examining its reasoning traces may clarify why it makes incorrect decisions. This could in turn inform future verifier improvements.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark provides a validated evaluation framework that enables quantitative assessment of verifiers.\n\n2. The test setting is diverse and challenging, and it is able to distinguish even among SOTA models.\n\n3. The experiments empirically demonstrate a correlation between higher benchmark scores and better training performance.", "weaknesses": "1. The paper does not appear to include examples of difficult samples. Showing more complex and challenging instances from the benchmark would be convicing.\n\n2. The benchmark is mainly focused on mathematics, so it may not reflect performance on verifiable tasks across the broader domain, such as evaluating physics expressions or chemical equations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xV8pX1OFOY", "reviewer": "Reviewer_uUti", "summary": "This paper introduces VerifyBench and VerifyBench-Hard, two benchmarks for evaluating reference-based reward systems in reinforcement learning for large language model reasoning. Unlike previous reward benchmarks that rely on pairwise preference judgments, VerifyBench assesses whether the reward models are able to identify if a model-generated response is correct or not, given reference answers. The datasets are built through large-scale curation and careful human annotation.", "questions": "- Could you clarify the difference between LLM-as-a-judge and Model-based Verifier? Since both involve LLMs producing binary verification signals, what distinctions make the latter uniquely \"model-based\"?\n\n- In the RL correlation results, the VerifyBench-Hard scores suggest that stronger verifiers (e.g., gpt-oss-20b) outperform weaker ones (e.g., Qwen3-4B), yet this does not translate linearly into downstream RL gains (Figure 3). What do you think are the sources of this mismatch?\n\n- Do you plan to maintain a public and continuously updated leaderboard (similar to RewardBench) to track verifier progress over time? Such a platform could help make VerifyBench the de facto standard for reference-based reward evaluation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The work is timely, filling a clear gap in the evaluation landscape as the community shifts from pairwise preference to reference-based reward evaluation in RL for reasoning models.\n\n- The pipeline ensures high diversity (spanning general, logical, and mathematical reasoning), and the use of human annotators for verification improves reliability and validity.\n\n- The RL correlation experiment convincingly demonstrates that verifier quality (as measured by VerifyBench) directly impacts downstream RL performance, emphasizing the benchmark’s practical utility.", "weaknesses": "- Despite the diversity of question types, coding-related tasks (e.g., code generation, debugging) are notably absent. \n\n- The benchmark does not include regex-based verifiers, which are widely used in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)