Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning
Authors: Aleksander Ficek
Somshubra Majumdar
Vahid Noroozi
Boris Ginsburg
Venue: COLM 2025 (2025)
Keywords: code generation and understanding
benchmarking
NLP datasets
evaluation methodologies
automatic evaluation of datasets
evaluation
metrics
reproducibility
statistical testing for evaluation
TL;DR: This paper benchmarks synthetic verification methods for code correctness, showing reasoning models improve test case generation and verification accuracy.

Abstract: Synthetic verification techniques such as generating test cases and reward modelling are common ways to enhance the coding capabilities of large language models (LLM) beyond predefined tests. Additionally, code verification has recently found great success as a critical component in improving reasoning capability of LLMs via reinforcement learning. In this paper, we propose an approach which can transform existing coding benchmarks into scoring and ranking datasets to evaluate the effectiveness of synthetic verifiers. We also propose multiple metrics to measure different aspects of the synthetic verifiers with the proposed benchmarks. By employing the proposed approach, we release four new benchmarks (HE-R, HE-R+, MBPP-R, and MBPP-R+), and analyzed synthetic verification methods with standard, reasoning-based, and reward-based LLMs. Our experiments show that reasoning can significantly improve test case generation and that scaling the number of test cases enhances the verification accuracy.

Published as a conference paper at COLM 2025
Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning
Aleksander Ficek, Somshubra Majumdar, Vahid Noroozi, Boris Ginsburg NVIDIA
Santa Clara, CA 15213, USA {aficek, smajurdar, vnoroozi, bginnbburg@nnvidia..
Abstract
Synthetic verification techniques such as generating test cases and reward modelling are common ways to enhance the coding capabilities of large language models (LLM) beyond predefined tests. Additionally, code verifi- cation has recently found great success critical component in improving reasoning capability of LLMs via reinforcement learning. In this paper, we propose approach which can transform existing coding benchmarks into scoring and ranking datasets to evaluate the effectiveness of synthetic verifiers. We also propose multiple metrics to measure different aspects of the synthetic verifiers with the proposed benchmarks. By employing the proposed approach, we release four new benchmarks (HE-R, HE-R+, MBPP-R, and MBPP-R+), and analyzed synthetic verification methods with standard, reasoning-basee
and reward-based LLMs. Our experiments show that reasoning can significantly improve test case generation and that scaling the number of test cases enhances the verification accuracy.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, particularly in code generation. Their advancements extend to solving algorithmic challenges in competitive programming, real-world software engineering tasks, and enhanc- ing automated code testing. Recently, reasoning models such as DeepSeek-R1 (DeepSeek-Al 2025) have found substantial improvements in math problem-solving and code generation by leveraging large-scale reinforcement learning (RL) and rule-based reward systems. In the context of coding capabilities, they utilized code execution on predefined test cases to generate the signal for RI, enabling the reasoning capability in an LLM. This highlights the importance of code verification with respect to the latest advances in reasoning models.
Although effective, execution-based scoring approach faces clear bottleneck due to the scarcity of the coding problems with predefined test cases. To address this constraint, many prior works have explored synthetically generated test cases and unit tests to automatically verify code quality and coverage (Schäfer et al., 2024; Chen et al., 2022). Additionally, some other works employ coding reward models to improve results on coding benchmarks (Zeng et al., 2025; Ma et al., 2025). In this paper, we collectively refer to these approaches as synthetic verifiers.
There are numerous benchmarks that assess various aspects of software engineering capa- bilities. Datasets such as HumanEval (HE) (Chen et al., 2021), Mostly Basic Programming Problems (MBPP) (Austin et al., 2021), and CodeContests (Li et al., 2022) are commonly used to evaluate the algorithmic and competitive programming skills of LLMs. Other benchmarks, including TESTEVAL (Wang et al., 2025), TestGenEval (Jain et al., 2024a), and SWT-Bench (Mündler et al., 2025), focus on assessing an LLM's ability to generate test cases for given solution or feature. While these benchmarks are highly useful, they do not
'The benchmarks and code used in this paper are publicly available at https: https:://hugginggaace.. batasets/nviidia/ccrriii Veri Verifiers and
1

---

Published as a conference paper at COLM 2025
Ranking with Predefined Test Cases
Solution
Solution #2
Solution
Question "Wiite a function to reverse the words a given string."
Ranking with LLM Generated Test Cases
Solution #2
Solution It1
Solution #3
Figure 1: The figure illustrates how predefined test cases rank different solutions and how synthetic verifier rankings are compared during evaluation.
evaluate whether synthetic verification methods effectively select better code solutions, a task we exemplify in Figure 1. Reward model benchmarks, such as RewardBench (Lambert et al., 2024), provide rankings of solutions, but these rankings are limited to preference pairs and their focus is not on coding problems. More details about past works can be found in section In particular, rating test case generation suites by how well they rank solutions remains largely unexplored.
Our work is an important contribution for advancing other works that rely high quality synthetic verifiers as a component of their training and inference pipelines. Verifiers can be used filter synthetic code generation data (Wei et al., 2024) parallel generations at inference-tine learning approaches may be enhanced by fine-grained score or assessment coding solutions, more than just pass and fail, to be able to learn effectively (Liu et al., 2023a).
select from from several Zhang et al. (2025). Additionally, reinforcement
In this paper, we propose an approach to transform any existing coding benchmark with predefined test cases into benchmarks that asses synthetic verifiers like test case generation or reward modelling. We also propose multiple evaluation metrics to measure different aspects of the synthetic verifiers with the new benchmarks. By employing the proposed approach, we created four ranking and scoring benchmarks; HE-R, HE-R+, MBPP-R and MBPP-R+ based on the HE-R, HE-R+, MBPP-R and MBPP-R+ respectively. The new bench- marks assess how well synthetic verification methods approximate solution correctness and their ability to identify the best solution for given problem. Then we demonstrate the benefit of these benchmarks by making quantitative observations about the ability of LLM's to generate test cases, the advantage of reasoning models in this domain, and the comparison of different synthetic verification methods like test case generation and reward models. To our knowledge, we are the first to study test case generation with reasoning models in depth. We plan to release our new benchmarks along with the code publicly.
In summary, we make the following contributions in our work:
1. We provide recipe to transform any coding benchmark with predefined test cases into a code scoring and ranking benchmark.
2. We certify our recipe by creating code scoring and ranking versions of HumanEval and MBPP datasets: HE-R, HE-R+, MBPP-R, MBPP-R+.
3. We use our benchmark evaluate the case generation capability in standard and reasoning LLMs alongside coding reward models. We show that reasoning model are more effective in generating test cases compared to non-reasoning ones.
2 Proposed Approach
We outline our proposed process to transform coding benchmark into a scoring and ranking benchmark in Figure 2. We assume the base benchmark contains a collections of coding questions
instructions along with their predefined test cases. We start by generating a set of solutions by employing multiple prompting techniques and filtering, with the goal of producing set of solutions which cover wide spectrum of code
2

---

Published as a conference paper at COLM 2025
accuracy. This property enables to have better and more fine-grained evaluation of synthetic verifiers. After deduplicating the solutions, we calculate the pass rate each solution using the predefined tests. Then use these scores to rank the final set of solutions per question. Finally, we propose collection of metrics to measure the quality of the synthetic verifiers the new benchmarks. We provide more details about each stage in this section.
Generation Cycle
1.
Raak: I Solution
LLM
Test Case Score
Deduplicate
0.6
Benchmark
Candidate Solution
Question
Kanking
Raak: 3 Solation
Figure 2: Diagram of the process for turning a coding cataset into a code scoring and ranking benchmark.
2.1 Generating Solutions
Initially, we generate some potential solutions by iterating over each question in the dataset and use multiple prompts with an LLM to produce a response/ solution for the given question. This generation cycle of solutions is repeated across multiple prompts, sampling hyperparameters, and seeds to increase the diversity of solutions. We used mainly two prompts provided in Figure 8, one for generating correct and one for generating fully incorrect or partially incorrect solutions. We produce naturally and artificially incorrect responses provide realistic failure modes while limiting self-evaluation bias. After responses are generated, the code solutions are extracted from the responses and their scores are calculated with the predefined test cases. We then aggregate all the generated solutions for each problem, building diverse population of solutions per problem.
craponse/solution
2.2 Filtering and Ranking
For each problem, we deduplicate solutions that achieve the same score (fraction of test cases passed) and tie-break using the lower average execution time to select the more optimal solutions. We always select the ground truth solution as the solutions which passes all predefined test cases. We also filter out solutions that fail completely due to non-assertion errors. This ensures exclude solutions that may be almost correct but achieve score of zero due to syntax errors or other trivial issues. Finally, we apply selection algorithm to select the solutions that most evenly distributed in terms of the fraction of test cases passed. Formally, let
S (s1, s...,,
denote the set of deduplicated solutions, sorted in descending order such that $2 2 ... 2 Sn, where s; represents the fraction of test cases passed by solution We assume that 51 M = 1.0 and define the minimum score
min(s E S: 0.1) if such s exists
Sn
otherwise
To account for cases when & exceeds we define the effective selection count as:
= min
For 1,2,.... .,..,,,'' 1, we compute target scores:
i T =
k (1 m).
3

---

Published as a conference paper at COLM 2025
For each T,, we select an unchosen solution that minimizes the absolute difference to Tj
sp argmin
Is 2--1 Tíl-
SES (-)
Finally, we include the solution corresponding to sp,, yielding the selected set:
S* $2 -2.., 5p
As an example, when m = 0.0, our selection algorithm chooses solutions that best approxi- mate the quantiles (0.0, 0.25, 0.5, 0.75, 1.0)- We continue generating solutions described in subsection 2.1 and apply filtering stages until we achieve the desired number of uniquely scored solutions per problem. Any problem that does not reach the target k after multiple rounds may either be discarded or supplemented with manually created solutions.
3 Proposed Benchmarks: HE-R, HE-R+, MBPP-R, MBPP-R+
3.1 Creation of the Benchmarks
We created four new benchmarks (HE-R, HE-R+, MBPP-R, and MBPP-R+) using the pro- posed approach introduced in section 2 based the commonly used coding benchmarks of HE, HE+, MBPP, and MBPP+. We used GPT-4o (2024-11-20) OpenAl (2024a) as the generator LLM to produce the candidate solutions. The extended versions of HumanEval and MBPP include significantly more test cases, making the passing scores a more reliable proxy for the overall solution correctness. Table 1 shows statistics of the created benchmarks including dataset size, average number of test cases per problem, total number of synthetic solutions, and the average score of selected solutions. HE-R and MBPP-R have significantly lower number of test cases which can affect the quality of the code verification. In cases where the benchmark has a limited number of predefined test cases but a ground-truth solution, we recommend to follow previous methods to generate additional ground truth test cases, similar to how HE+ and MBPP+ benchmarks are created Liu et al. (2023b). In future works recommend using a family of models at the generation stage to dilute the self-evaluation bias from an individual model.
For transformed benchmarks, set k = to ensure that HE-R+ and MBPP-R+ contain at least five uniquely scored solutions per problem. For some problems, the automated pro- cess could not find enough solutions with the desired requirements, therefore we manually annotated 10 solutions for HE-R+ and 15 solutions for MBPP-R+. We use a k of 2 to 5 for samples in the base versions of HumanEval (HE-R) and MBPP (MBPP-R) because of the limited number of predefined test cases. In Table 3, we perform saturation analysis of HE+ and MBPP+ better understand how many test cases are necessary determine proposed benchmark is suitable for transformation. The lower bound of the confidence for HE+ and k k= 5 for MBPP+. These results show that modest number of tests already yields stable rankings, consistent with our success in using the limited number of test cases available when transforming the base versions of HumanEval and MBPP.
interval exceeds the conventional high correlation threshold (p 2 0.90)
HE-R
HF-R+
MBPP-R
MBPP-R+
Number of problems
164
164
974
378
Average number of test cases
9.6
764.1
3.0
108.5
Number of synthetic solutions
742
820
3249
820
Average score of solutions
0.52
0.50
0.49
Table 1: Original and transformed benchmark metrics.
3.2 Analysis of the Test Scores
We showed the distribution of the test score differences per problem for HE-R+ and MBPP- R+ in Figure 3. Test score differences are calculated the difference between the highest
4

---

Published as a conference paper at COLM 2025
and lowest scoring solutions for each problem. This difference can be indicator of the correctness coverage of the generated solutions. As it can seen, all solutions exhibit minimum score difference of 0.5, with the majority having difference between 0.9 and 1.0. higher difference between the solutions shows that each solution varies in quality such that it is distinguishable by generated test cases and coding reward models.
Maximum Test Score Difference
Maximum Test. Score Difference
100
80
Frequency
60
100
50
20
Test Score Difference
Test Score Difference
(a) HE-R+.
(b) MBPP-R+
Figure 3: Histograms of maximum difference between test case scores for each problem.
Figure 4 presents the distribution of the fraction of the tests passed for all solutions in HE-R+ and MBPP-R+. The histograms reveal bimodal distribution, which aligns with expectations, the ground truth solution is always included, and there is commonly solution that fails most tests. The remaining scores conform to the typical target quantiles of (0.0, 0.25, 0.5, 0.75, 1.0).
Distribution of Test Scores
Distribution of Test Scores
175
400
125
250
75
100
25
50
(a) HE-R+
(b) MBPP-R+
Figure 4: Histograms of distribution of test case scores in each dataset.
Histograms of the number of solutions per problem, fraction of tests passed and test score differences for all four benchmarks can be found in Appendix C. As can be seen, HE-R and MBPP-R show similar pattern to HE-R+ and MBPP-R+ except that there are less unique candidate solution scores due to the limited number of test cases provided to differentiate their quality shown in Table 1.

---

Published as a conference paper at COLM 2025
4 Experimental Settings
After creating the benchmarks, we used them to explore and evaluate two categories of synthetic verification methods: 1) synthetic test case generation, and 2) code reward modelling. We evaluated several models from different providers on our benchmarks (AI, 2024; Qwen et al., 2025; Hui et al., 2024; OpenAI, 2024a;b; DeepSeck-AI, 2025).
4.1 Test Case Generation
For evaluating models on the task of test case generation, we select a well-suited prompt (Figure 14) to generate an appropriate number of test cases for each problem in the bench- mark. In our prompt, we provide two examples HumanEval format and ensure the model wraps each test case appropriately. We used nucleus sampling with tem perature of 1.0 for all the generations. Considering the large number of experiments and the limited context size of the LLMs, generated 10 test cases per problem for our primary results as feel this provides a reasonable coverage of edge cases. In order to compute the test scores, we executed each test case and the provided solution with timeout of 3 seconds after noticing negligible differences in non-assertion timeout errors at higher amounts.
4.2 Code Reward Modelling
For evaluating reward models, we used the reward model to estimate the correctness and quality of all the solutions in the benchmarks with the prompt shown in Figure 15, We found applying brief preamble in the prompt improves the accuracy of the reward models. We computed the reward score for each solution and normalized using the highest and lowest scores for each problem. Then, the ranking achieved by these scores evaluated moootron4-340B-Revee RNvirdd 2024) we use
with the different metrics. For the model only correctness attribute for the reward score this fits best the goals of our evaluations.
4.3 Metrics
For all the experiments, we propose to use the following metrics compare and evaluate different LLMs. These evaluation metrics quantify synthetic verifier's s ability to correctly score and rank solution correctness.
Top-1 Accuracy: Determines if the reward or the unit tests generated by an LLM can correctly rank the best solution first.
Spearman's p Coefficient: Evaluates the strength and correlation between the ranking achieved by the LLM's assessment versus the expected ranking. Expected ranking is determined by the ranking achieved by executing the predefined test cases.
Bottom-1 Accuracy: Determines if the reward model or the test cases generated by the LLM can correctly rank the worst solution last.
Mean Absolute Error (MAE): Quantifies the absolute difference between the expected and estimated fraction of test cases passed.
We underscore Top-1 Accuracy and Spearman's 's coefficient as the primary metrics that encapsulate the ability for synthetic verifiers to select the best solution and rank all solutions appropriately. Bottom-1 and Mean Absolute Error are suitable secondary metrics that provide additional signals on scoring incorrect solutions alongside the delta from expected score. All ranking-based metrics are averaged across all questions within each benchmark. If test case scores result in ties, compute the fraction of correctly ranked top solutions relative to the number of tied entries for Top-1 and Bottom-1 evaluations. For Spearman's Coefficient we assign the tied ranks their average position.

---

Published as a conference paper at COLM 2025
Results
In Table 2, we present our main results on HE-R+ and MBPP-R+ with 17 standard, reward, and reasoning-base LLMs while Table 4 presents the results on HE-R and MBPP-R.
5.1 Results Test Case Generation
We find that the performance of self-generated test cases on our benchmarks generally correlates with the generating model's performance on the original HumanEval and MBPP. As can be seen, top performing regular models HumanEval and MBPP such Qwen2.5- Coder-32B-Instruct, Qwenn25-728-Instrrue and GPT-4o perform the best among regular models. Also, larger and stronger models in each family of models outperform their smaller variants on almost all benchmarks and metrics.
HE-R+
MBPP-R+
Top-1
Spearman Bottom-1
MAE
Top-1
Spearman Bottom-1
MAE
Standard Models
Meta-Llama-31-88--nttte
55.9
0.58
0.28
48.5
0.45
51.1
0.31
peta-llama-3.-770--nnnstt
66.8
0.67
71.2
0.24
61.0
0.63
67.3
0.25
Meta-Llama-33-700BBB
73.8
0.77
79.7
0.22
67.7
0.67
68.7
0.24
Qwenn.5-7B-Instruut
0.76
0.23
0.64
0.25
Dwen2.5-32B-Instruud
74.9
0.79
77.5
0.22
68.8
0.72
75.0
0.23
Qwen2.5-72B-Instruct
78.3
0.80
76.6
0.21
71.4
0.73
75.0
0.22
Oween25-CCoder-BB-nnstruc
71.2
0.75
73.8
0.23
0.63
68.3
0.26
Swven25-Coder-322--nnmmme
79.1
0.83
80.7
0.21
68.5
0.72
73.9
0.23
GPT-4o (2024-11-20)
76.8
0.81
76.4
0.21
70.8
0.71
71.9
0.22
Reward Models
AceCodeRM-7B
68.3
0.65
62.8
0.23
70.9
0.52
40.5
0.27
AceCodeRM-32B
77.4
0.68
53.5
0.23
74.9
0.57
39.2
0.26
Nemotron-7oB-Rewwe
60.4
0.61
53.7
0.24
69.6
0.53
39.4
0.27
Nemotron443400-Rewward
76.2
0.67
59.2
0.23
75.1
0.59
46.0
0.25
Reasoning Models
DeepSeek-RI-Diie RLL-DistillQWeen-32 Qwen-32B
0.78
0.22
70.1
0.65
0.24
DeepSeek-R1
83.8
0.85
81.4
0.20
77.5
0.74
75.7
0.21
ol-mini (2024-09-12)
82.5
0.83
79.7
0.20
74.5
0.72
73.5
0.21
o3-mini (2025-01-31)
0.85
84.0
0.18
79.9
0.78
80.1
0.20
Table 2: All model results on HE-R+ and MBPP-R+.
Our findings highlight the effectiveness of test case generation once a model surpasses certain capability threshold. Achieving 79.1% and 71.4% accuracy in differentiating the best solution with only 10 test cases is a significant challenge, requiring deep problem understanding and the ability to construct minimal yet highly effective test suite that exposes subtle errors. Spearman's Coefficient and Bottom-1 values demonstrate the generated test cases also label imperfect solutions accurately. Models with at least 32B parameters demonstrate these capabilities, accurately selecting the best Top-1 and Bottom-1 solutions. However, this is not their upper limit, we show in the following subsection 5.3 that increasing the number of test cases improves performance further. Notably, these models have not been explicitly trained for test case generation yet still perform well, demonstrating significant potential for refining LLMs for this task. Figure 5a shows the breakdown of the total number of generated test cases that are passed or failed due to assertion errors and non-assertion errors MBPP-R+, generated by similarly shows the distribution of test case scores produced by the model which exhibits similar bimodal distribution as the ground truth test scores in Figure 4b.
Figure 5b
7

---

Published as a conference paper at COLM 2025
Error Distribution (MBPP-R+)
Distribution Test Scores (MBPP-R+)
Ansertstn Error
10.0%
250
Passed
Frequeacy
Assertion
100
0.0
0.2
0.4
0.8
1.0
Score
(a) Error distribution of generated test cases.
(b) Distribution of generated test case scores.
Figure 5: Analysis of test cases generated by Dit-SSekk-RR--isill------ae on MBPP-R+
5.2 Reasoning Model Results
We observe that the enhanced coding capabilities reasoning models translates im- proved test case generation. In a head-to-head comparison, DeepSee--R--Dssill-QWeeeee outperforms Swwe2.5--2B-IInttree
Top-1 accuracy but falls short in Spearman's coef- ficient evaluations. However, incorporating DeepSeek-R1, ol-mini, and 03-mini leads to significant improvements across all metrics, positioning them as the most effective synthetic verifiers currently available, especially when scaling the number of test cases. Figure 18 presents sample CoT from CipSSee--R--Distil--O- convincingly explores many pathways cover potential solutions with its test cases.
which illustrates how the model
5.3 Number of Test Cases Study
Figure 6 shows that while 10 test cases demonstrate reasonable effectiveness but scaling the test cases allows the model better cover of the possible cases in a problem. We see that the reasoning capabilities of DeepSeek-R1 allow the model to scale the number of test cases effectively achieving a HE-R+ Top-1 of 91.6% while plateuing on MBPP-R+. Qwen2.5. Coder-32B-Instrutee and DiapSeek-RR--isiill---e-------e
alternatively start to plateau at the 20 test case mark. This exemplifies similar findings from Ma et (2025) where they scale test cases to improve reward signals in Llama3.1-70B Further exploration is encouraged to assess the limits reasoning and scaling of test cases to improve accuracy. Additional metrics while scaling test cases be seen in 5 5.
5.4 Code Reward Model Results
Converting the original benchmarks into scoring and ranking format enables unique comparison of different synthetic verification methods like test case generation and reward models. From our results in Table 2, the best reward models are AceCoderRM-322 for HE-R+ and Nemotron4-340B- Reward for MBPP-R+. We find that the best performing reasoning and standard models outperform the reward models in most metrics. In similarly sized models like Qwen2.5-Coder-S22 32B-Instructt,, the Top-1 scores are competitive in HE-R+ and better in the of MBPP-R+ while struggling in ranking the varying qualities of incorrect solutions. This could be due to subjectivity ranking incorrect solutions, they may be functionally incorrect but qualitatively exhibiting meaningful quality. We encourage self-generated test
8

---

Published as a conference paper at COLM 2025
Scaling Tast Cases on HE-R+
Scaling Test Cases on MBPP-R+
Qweezz Seannt---eeer-22---- Jestrcct
DrepSeok-KK B--eekk-----i-ll----- Quee
310
175
773
250
22.
20.0
775
675
750
65.0
Gurei. E-dddr Sedee--ii--sstrreet
(a) HE-R+
(b) MBPP-R+
Figure 6: Top-1 scores of D. DeepSeek ReepSeee-RR--isiil---eee... and
DeepSeek-RI when scaling number of test cases.
cases as suitable synthetic verifier for determining the correctness of solution but see promising opportunities to further enhance reward models for coding.
Solution Inclusion
Finally, we examine the impact of prompting with and without provided solution, as shown in Figure 7. All models exhibit significant performance degradation when given potentially incorrect solution and tasked with writing test cases to evaluate it. We find that the models have a bias towards adhering to any solutions provided even when specifically prompting against this. This is supported by previous works that find LLM's to be worse at providing test cases when provided incorrect compared to correct code (Huang et al., 2024). We therefore refrain from including the solution for the rest of our experiments means to improve test case quality and limit self-evaluation bias.
Prompting With vs Without Solution on HE+
With Solution
80
Solation
78.3
78.2
73.8
73.6
70.3
70
60
50
40
30
20
10
Llama-3 3.1 1-8B
Llama-a 3.3-708----10 113---0--mmmmt 1-770t-mmttrutt Instruct
Owen2 4--28---nmae Instruct t-728-tnstruct
RI HI-Dietil--oe D1-Dietit-Owan-332 Owan-328
DeepSeek-R1
Figure 7: Generating test cases with and without solutions for various models.

---

Published as a conference paper at COLM 2025
6
Related Works
Prior work primarily validates self-generated test cases within isolated systems or limited studies. Wei et al. (2024) conducts an ablation study showing that filtering LLM-generated solutions with self-generated test cases improves synthetic data quality, evidenced by downstream supervised fine-tuning results. Light et al. (2024) compares self-generated validation tests to ground truth tests on HumanEval to highlight the impact of accurate test cases on their Scattered Forest Search method. Zhang et al. (2023) justifies its oracle verifier strategy by comparing its test cases on correct solutions, Additional techniques use test case generation to improve results on coding tasks (Liu et al., 2024; Ridnik et al., 2024; Xu al., 2024; Ye et al., 2025). This paper unifies these approaches by introducing a benchmark for systematically assessing synthetic verifier's 's abilities at determining correct solutions.
As mentioned in the introduction, creating evaluations for test case generation is well explored area. This includes many benchmarks and systems that compete over quantifying coverage, mutation testing, validity and efficiency (Wang et al., 2025; Jain et al., 2024a; Mündler et al., 2025; Jain et al., 2024b; Taherkhani & Hemmati, 2024; Ahmed et al., 2024; Peng et al., 2025; Ryan et al., 2024; Li & Yuan, 2024; Zhang et al., 2024). Crucially, we do not assess an LLM's ability to generate test cases but rather the effectivness of LLM generated test cases to determine solution quality and rank. This aligns with CodeJudge-Evaä Zhao et al. (2024), which employs similar methodology to benchmark LLM-as-a-Judge.
Our work aligns closely with reward model evaluation such in the case of RewardBench (Lambert et al., 2024). Similarly, Zeng et al. (2025) leverages synthetic test cases to train coding reward models, evaluating them via best-of-N sampling. Ma al. (2025) explores using generated test cases as binary signals to train a test-generating reward model, assessed through best-of-N performance. Despite these advances, a standardized benchmark for comparative evaluation remains lacking. Our work addresses this gap while also advancing test case generation tion across state-of-the-art standard, reasoning, and reward models.
7 Conclusion
We introduce a systematic approach to transform any coding benchmark with predefined test cases into a ranking and scoring benchmark for evaluating synthetic verification methods. Our method involves generating a diverse set of LLM-produced solutions, scoring them based on the fraction of test cases passed, and applying structured filtering process to create reliably ranked datasets. We validate this approach by developing HE-R, HE-R+, MBPP-R, and MBPP-R+, which provide standardized framework for assessing the effectiveness of synthetic verification strategies. We then use our transformed datasets to explore and uncover the effectiveness of standard, reward and reasoning based LLM's. Using our transformed datasets, we investigate the effectiveness of test case-based verification, the impact of reasoning models, and the relative strengths of reward models. Our findings reveal key insights into the performance of various LLM paradigms, highlighting the potential of Dasoning-enhaneee models and scaling test case generation for improved accuracy.
Limitations
We highlight the following limitations in our work. First, the verification benchmarks produced using our approach rely directly on the ground truth of the original benchmark for benchmark transformation. Therefore, the produced verifier benchmark quality is directly tied to the reliability and accuracy of the original benchmark. Secondly, we were practically constrained by the output context length pricing when generating test cases using the API-based reasoning models. This led to strategically selecting 10 test cases based on the plateau of performance from non-reasoning models at that value while remaining within this the pricing constraint.
10

---

Published as a conference paper at COLM 2025
References
Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, and Saurabh Sinha. Tdd- bench verified: Can Ilms generate tests for issues before they get resolved?, 2024. URL https://arxiv wwtgg/////////// 02883.
Meta AI. The llama 3 herd of models, 2024. URL / 217833.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program synthesis with large language models, 2021. URL httptss///rxx....gg//////2111 07732.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests, 2022. URL https://arxiv, org/abs/2207 10397.
10397.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https: / http://rww....rr/////////... 03374.
DeepSeek-AL. AL. Deepseek-rl: Incenttiiinn reasoning capability in llms via reinforcement learning, 2025. URL https:://ariii..rrgaaasssss
12948.
Dong Huang, Jie M. Zhang, Mingzhe Du, Mark Harman, and Heming Cui. Rethinking the influence of source code on test case generation, 2024. URL https://arxiss http:/aaxxx....aaabbb/// org/abs/ 2409. 2409.09464.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. wwbps///49.///////0... 12186.
URL
Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. Testgeneval: A real world unit test generation and test completion benchmark, 2024a. URL http:p::///axii.oo//aa////////// 00752.
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2E: Turning any github repository into programming agent environment. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 21196-21224 PMLR, 21-27 Jul 2024b. URL https: pttp//iocceeens.ccc/e////. mlr wwt.ppereee///aiin//aaaees
html.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Han- naneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URL https:///axxii...../// http:/wwww.rg//////4...... 13787.
Kefan Li and Yuan Yuan. Large language models as test case generators: Performance wttp///g///b/////////////////// 13340.
evaluation and enhancement, 2024. URL
11

---

Published as a conference paper at COLM 2025
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Aute,e, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378(6624): 1092-1097, December 2022. ISSN 1095-9203, doi: 10.1126 science.abql158. URL http: //dx. //dx.doi.org/10. 10.
Jonathan Light, Yue Wu, Yiyou Sun, Wenchao Yu, Yanchi liu, Xujiang Zhao, Ziniu Hu, Haifeng Chen, and Wei Cheng. Scattered forest search: Smarter code space exploration with llms, 2024. URL htttt: http:/ivw..rrg//////////// 2411105011 05010.
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. Rltf: Reinforcement learning from unit test feedback, 2023a. URL https: //arxiv wwt.//sr//////////// 2307. 04349.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code gener- ated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023b. URL httts::/aaxxv..orrbbb///0
01210.
Zhihan Liu, Shenao Zhang, Yongfei Liu, Boyi Liu, Yingxiang Yang, and Zhaoran Wang. Dstc: Direct preference learning with only self-generated tests and code to improve code Ims, 2024. URL /tttsi://arxvv..oo org/o/g////o org/abs/// /2s1/ 13611.
Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Dynamic scaling of unit tests for code reward modeling, 2025. URL htttp::://arxi..o//bb///////// .10e1.010000 01054.
Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents, 2025. URL http::////arx abs/2406. 12952.
Nvidia. - 340b technical report, 2024. UBLLttoss//arivv.orr////////// 11704.
OpenAl. Gpt-4o system card, 2024a. URL SKKLLntpps//axxiv.or///////22222
OpenAL. Openai system card, 2024b. URL httpss//aaxxivorrg/b////1211
Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. Coffe: A code efficiency benchmark for code generation, 2025. URL http://:////axxioo//////////////
02827.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: //ap:v//aiiivvv/// wtt.rxi.oor//bb//111111 15115.
Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering, 2024. URL httpss//arxiivoor//aa/////////
Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. Code-aware prompting: A study of coverage guided test generation in regression setting using 11m, 2024. http.ttts::/arxxv.oorrr 00097.
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. I.F.F. Transactions on Software Engineering, 10(1) 2024. doi: 10.1109/TSE.202..33
Hamed Taherkhani and Hadi Hemmati. Valtest: Automated validation of language model generated test cases, 2024. URL https://arxix. wttpwvvv.../////411111/// 08254.
12

---

Published as a conference paper at COLM 2025
Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. Testeval: Benchmarking large language models for test case generation, 2025. URL https://axxiv.orr abrpg//bbb/26666/ 04531.
Yuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding, Naman Jain, Zachary Mueller, Harm de Vries, Leandro von Werra, Arjun Guha, and Lingming Zhang. Selfcodealign: Self-aligmment for code generation, 2024. URL http:://aaravv..rrg//////2100 24198.
Qingyao Xu, Dingkang Yang, and Lihua Zhang. Code optimization saain-of-thought Structured understanding and self-checking, In Proceedings of the 2024 4th International Conference on Artificial Intelligence, Big Data and Algorithmss, CAIBDA '24, pp. 425-430, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400710247, doi: 02.1445/3604407366000
URL attpps://oli org/10. 1///10.115/3399988 1145/3699407.36994773
Yufan Ye, Ting Zhang, Wenbin Jiang, and Hua Huang. Process-supervised reinforcement learning for code generation, 2025. URL wttttt:s///aaiieee www.wi.wrog/bb///22.... 01715.
Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis, 2025. URL https://arxiv. org//bbs/2502.0177 01718.
Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. Algo: Synthesizing algorithmic programs with llm-generated oracle verifiers, 2023. URL https://arxiw.orr/bb/////
14591.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling next-token prediction, 2025. URL
https://arxiv.orr wttp://w..wg///////.....//
Quanjun Zhang, Ye Shang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. Testbench: Evaluating class-level test case generation capability of large language models, 2024. URL
http://xxii..or///////4/ 17561.
Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, and Jing Ma. Codejudge-eval: Can large language models be good judges in code understanding?, 2024. URL https: httpxivorrg/bb//44666 10718.
13

---

Published as a conference paper at COLM 2025
A Producing Solution Prompt
Producing Correct Solutions Prompt
Using only Python code, write solution to the given coding problem. Here are other guidelines for completing this task:
1. Enclose the code in python code block "pyython.
Do not include any unit tests in your answer, only generate the function.
3. The code must still compile, the only errors in the code should be logical. Include any necessary imports with your code, only import libraries included the standard library.
Question: {question)
Answer:
Producing Partially Correct Solutions Prompt
Using only Python code, write somewhat incorrect solution to the given coding prob- lem.
Do not provide any hints what is the mistake. Here are other guidelines for completing this task:
1. Enclose the code in python code block "python.
Do not include any unit tests in your answer, only generate the function.
3. The code must still compile, the only errors in the code should be logical.
4. Include any necessary imports with your code, only import libraries included in the
standard library.
5. Do not add any hints as to the error you made.
Here some suggestions:
Do not handle negative numbers
Do not handle duplicate values
Introduce rounding errors
Ignore the last element list
Only handle specific values
Only works for certain ranges of values or lengths
Question:
{question}
Answer:
Figure 8: Prompt templates for producing solutions
14

---

Published as a conference paper at COLM 2025
B Saturation Analysis
Pmean
995% low
P95% high
Up
k
Pmean
995% low
P95% high
0g
0.725
0.722
0.728
0.016
1
0.739
0.737
0.740
0.008
2
0.815
0.813
0.817
0.011
2
0.834
0.833
0.836
0.007
0.857
0.855
0.859
0.009
3
0.875
0.874
0.876
0.005
4
0.879
0.878
0.881
0.009
4
0.900
0.899
0.901
0.006
0.895
0.894
0.897
0.008
5
0.915
0.914
0.916
0.004
0.905
0.903
0.906
0.008
6
0.925
0.924
0.926
0.004
0,913
0.912
0,914
0.006
7
0.933
0,932
0.933
0.004
8
0.919
0.917
0.920
0.007
8
0,938
0.938
0.939
0.004
0.925
0.924
0.926
0.007
9
0.943
0.942
0.944
0.004
10
0.929
0.927
0.930
0.006
10
0.948
0.948
0.949
0.003
(a) HR+
(b) MBPP+
Table 3: Saturation analysis of number of test cases with HE+ and MBPP+ benchmarks.
15

---

Published as a conference paper at COLM 2025
C
Benchmark Analysis Visualizations
Solutions per Sample
Distribution Test Scores
Maximum Test Score Difference
100
110
Frequeaayy
20
320
00
Number Saluttens
That Score Diffeennce
Figure 9: HumanEval (HE-R) benchmark analysis.
per Sample
100
175
120
350
370
120
125
320
00
75
60
30
25
20
Number Sulutioas
Text
Test Scare Difference
Figure 10: HumanEval Plus (HE-R+) benchmark analysis.
Solutions per Sample
Distribution Scores
Maximum Difference
800
400
100
260
Number Solutleas
Test Score
Test Scare Diffeeence
Figure 11: MBPP (MBPP-R) benchmark analysis.
Solutions Sample
Distribution of Test Scores
Maximum Test Score Difference
420
200
300
220
310
50
0.5
1.00
Number of Solnttons
Thxt Scom
Trst Scare Dittrennce
Figure 12: MBPP Plus (MBPP-R+) benchmark analysis.
16

---

Published as a conference paper at COLM 2025
D
Test Case Generation Prompts
Test Case Generation Without Solution Prompt
You are expert at writing assertion test cases and below is question with function signature and test cases, You must generate 10 assert test cases that will be used to evaluate the code solution's correctness. You must adhere the provided function signature and test case format. Here are some examples that you should use as reference:
Question:
from typing import Optional def First\_repeated__chrrrrs (s: str) Optional[str]: =up
Find the first repeated character in given string.
>>> first\_repeatedl_chhl ("abbbbaac) 'a'
Test Cases:
<assertion>assert firstt_rppeteed_ ("!@#$% ")!") "!"</assertion>
<assertion>assert first_reppated__hhrr"aaaaa ("abcdedcba") == "d"</assertion>
<assertion>assert firsst_repeated_charr ("") 2/assrrtstrt
<assertion>assert firsstrepeatedd_char a"aaa == "a"</assertion>
<assertion>assert firsstrrepeated_char ("a"" = "None "None" 2/assetiint
Here are guidelines for writing the assertion test cases:
You must wrap each assertion test case with tags jassertion, and j/ /aasserttionn 2. Do not start the assert with any indents or spaces. You must not import any unit testing libraries for the assertions such as "unittest" or "pytest".
Each assertion must be complete and immediately executable. Assume the code solution is provided, do not repeat it. Avoid unnecessary string literals, incorrect escaping, wrapping ""'python" or other redundancies.
Remember, your responsibility to carefully read the question and generate test cases that will evaluate the correctness of the solution.
Here the question you must provide assertion test cases for:
Question: {question}
Test Cases:
Figure 13: Prompt template for test case generation without solution
17

---

Published as a conference paper at COLM 2025
Test Case Generation With Solution Prompt
You expert at writing assertion test cases and below is question with function signature and completed code solution. You must generate 10 assert statements that will be used to evaluate the code solution's correctness which may may not be correct. Here are some examples that you should use as reference:
Question:
from typing import Optional def firstt_rrpeateed_chhrrrs str) Optional[str]:
Find the first repeated character given string.
>>> (" 'a'
Solution:
from typing import Optional def firsttrepeated__hhrrs str) Optional[str]:
Find the first repeated character in a given string.
>>> sirst_repeated_charrrabbaa 'a'
for index, in enumerate(s):
if iD:ineex co.nount(C (c) 1:
return c
return None
Test Cases:
<assertion>assert first__epeetedd_chan caaiidd_haaaar""""""aaa ("!@#$% "*!!) == "!"</assertion>
<assertion>assert first_repeated_chhe a"t___rpppaatddchrrrraaadddaaaa ("abcdedcba") == "d"</assertion>
<assertion>assert first_rrppeteed_dee cared_chhaahh ("") == "None" "/aassertrio
<assertion>assert first_repeated chaecchaar"" caaieedaaaaaaaaaaa""" ("aaaa") == "a"</assertinn>
<assertion>assert first_reppatedd_hhan ("a") "None" </assertion
Here guidelines for writing the assertion test cases:
You must wrap each assertion test case with tags jassertion; and i/assertion..
2. Do not start the assert with any indents spaces.
3. You must not import any unit testing libraries for the assertions such as "unittest" or "pytest".
4. Each assertion must be complete and immediately executable. Assume the code solution is provided, do not repeat it.
5. Avoid unnecessary string literals, incorrect escaping, wrapping in ""'python" or other redundancies.
6. Remember, it is your responsibility to carefully read the question and generate test cases that will evaluate the correctness the solution.
Here is the question and code solution you must provide assertion test cases for:
Question: {question)
Solution: (solution}
Test Cases:
Figure 14: Prompt template for test case generation without solution
18

---

Published as a conference paper at COLM 2025
E
Reward Model Prompts
Reward Model User Prompt
You are given coding problem for which you need to cenerate/compleet solution that as accurate as possible.
Please complete the function with the Python programming language.
This the problem you must solve: {question}
Reward Model Assistant Prompt
Here is the solution to the given problem:
{solution)
Figure 15: Prompt templates for reward model user and assistant turns
19

---

Published as a conference paper at COLM 2025
F
Supplementary results
HE-R
MBPP-R
Top-1
Spearman Bottom-1 MAE
Top-1
Spearman
Bottom-1 MAE
Standard Models
Meta-Llama-31--8--hhttuut
63.6
0.69
70.6
0.24
55.4
0.42
56.4
0.35
Mcta-Llama-331-00--nnsttttt
74.2
0.80
78.8
0.18
72.0
0.68
72.7
0.25
Meta-Llama-333--00--nmsstt
84.8
0.14
76.2
0.23
Qwen225-77-Instruc
81.2
0.86
78.4
0.18
72.5
0.66
72.5
0.26
Qwen2.5-32B-Imstruct
85.7
0.90
86.9
0.13
78.3
0.75
78.8
0.22
Qwenn:5-72B-Instrucct
85.3
0.90
84.2
0.14
76.9
0.74
78.3
0.22
Qweem255Coode--7H--nstmau
81.6
0.86
82.4
0.17
67.2
01.61
69.4
0.27
Owen225-Coodrr:32BBmsttte
0.13
77.6
0.23
GPT-4o (2024-11-20)
81.8
0.88
84.6
0.14
74.9
0.71
76.3
0.22
Reward Modell
AceCodeRM-7B
71.3
0.68
18.9
0.22
71.2
0.53
35.7
0.26
AceCodeRM-32B
80.5
0.75
29.3
0.21
75.8
0.60
38.6
0.24
Nemolron-70B-RRward
65.2
0.65
16.5
0.22
45.7
24.9
0.35
Nemotrond- Rewwde
82.9
0.72
20.7
0.21
66.0
0.55
37.5
0.26
Reasoning Models
81.8
0.86
81.0
0.13
74.1
0.68
73.0
0.23
DeepSeck-Ro R1
90.9
0.92
84.0
0.11
81.2
0.77
79.2
0.20
Table 4: Model results on HE-R and MBPP-R.
HE-R
MBPP-R
Top-1 Spearman Bottom-1 MAE
Top-1 Spearman Bottom-1 MAE
-wenn.5-Cooer--22--nnie
5 Test Cases
77.5
0.81
75.7
0.22
61.9
0.67
67.1
0.24
Test Cases
79.1
0.83
80.7
0.21
68.5
0.72
73.9
0.23
15 Test Cases
81.4
0.82
82.9
0.21
70.1
0.72
76.7
0.22
20 Test Cases
80.9
0.80
82.2
0.21
70.2
0.71
76.7
0.23
Test Cases
0.81
80.4
0.22
80.1
DeepSeek-R1 -i-ieek-------llll---- Qwen-32B
5 Test Cases
72.4
0.76
72.2
0.23
65.4
0.65
64.1
0.24
10 Test Cases
78.2
0.78
74.1
0.22
70.1
0.65
68.5
0.24
15 Test Cases
83.7
0.77
744
0.21
68.3
0.62
63.1
0.24
20 Test Cases
78.0
0.81
79.9
0.20
69.2
0.64
69.0
0.23
25 Test Cases
77.6
0.76
74.1
0.21
71.2
0.63
65.5
0.23
DeepSeek-RI
5 Test Cases
78.4
0.79
74.4
0.21
69.1
0.71
69.4
0.23
Test Cases
83.8
0.85
81.4
0.20
77.5
0.74
75.7
0.21
15 Test Cases
86.2
0.84
84.7
0.19
79.9
0.76
77.8
0.20
20 Test Cases
88.2
0.86
85.4
0.18
81.2
0.76
76.7
0.19
25 Test Cases
0.86
85.4
0.19
0.75
0.20
Table 5: Test case scaling results.
20

---

Published as a conference paper at COLM 2025
G Model Test Case Scoring Results
Error Distribution (HE-R+)
Non-Assertion Brvor
Non-Assection Erre
1,8%
2.4%
Tost Case Fassed 56.77
Assertion Brvor
Assertion Brcor
42.9%
46.7%
Error Distribution (MBPP-R)
Distribution (MBPP-R+)
Non-Assertios Error 11.5%
Non-Assertion Error 10.0%
Case Passed
45. 1%
48.2%
Assertion
Assertion Brror
43.3%
Figure 16: CeepSeek-RI-Disttil-Oeee test error distributions.
21

---

Published as a conference paper at COLM 2025
Distribution of Test Scores (HE-R)
Distribution of Test Scores (HE-R+)
100
60
Prequency
40
30
20
10
0.0
0.6
1.0
0.0
0.2
0.4
0.6
0.8
Tast. Score
Test Score
Distribution of Test Scores (MBPP-R)
Distribution of Test Scores (MBPP-R+)
5.00
200
4.00
Preguenty
Proquency
150
50
0.0
0.2
0.4
0..6
11.0
1.0
0.0
0.2
0.4
0.6
0.88
1.0
Test Score
Test
Score
Figure 17: DeepSeek-R--Distil---e test case score distributions.
22

---

Published as a conference paper at COLM 2025
H
Reasoning Model Chain-of-Thoughts
Sample MBPP Question (MBPP Question 809)
Write function to check if each element of second tuple is smaller than its corresponding element in the first tuple.
assert check_smaller(Ch, 2, 3), (2, 3, 4)) = False
DeepSSek-RI-Distill-----r-- Reasoning Trace
Alright, need to write 10 assertion test cases for the function check smaller, which takes two tuples and returns True if each element in the second tuple is smaller than the corresponding element the first tuple.
Let me think about different scenarios that the function might encounter.
First, the sample test case provided is when all elements of the second tuple are larger. That's good start.
Then, I should consider cases where the second tuple has smaller elements for all posi- tions, which should return True.
Next, it's important to test cases where the second tuple is empty. According to the function's 's intended behavior, since there are no elements to compare, it should return True.
Also, leading or trailing zeros might be tricky, so should include test case where the first tuple has zero and the second has a positive number, ensuring returns False.
What about negative numbers? If both tuples have negative numbers, the function should correctly compare them. For example, (-5,-6) compared to (4,-7) should return False because the first element is smaller, but the second is larger. Wait, no the function should return True only if all second elements are smaller. In this case, the first elements of the second tuple (-4, -7) is larger, so the result is False. Wait, that might not be correct. Wait, the function should return True only if every element in the second tuple is smaller. So if the second tuple has any element not smaller, return False. So, if the first tuple is ( -5, -6 ), and the second is (-,--7 -7), then: First element: -4 is larger than -5, not smaller, so function should return False. I'll include that case well.
I should also consider tuples of different lengths. Oh, but according to the function signature, it's taking two tuples. What if they re of different lengths? For example, tuplel is (1, 2) and tuple2 is (1, 2, 3). Do we need to handle that? The problem description doesn't 't mention it, so perhaps the function assumes that the tuples are of the same length. So can test with same-length tuples, but also think about varying lengths. But since the function's docstring doesn't t specify, maybe I can assume the tuples are of the same length. However, I'll include a test case where the second tuple is longer, but the function only checks up to the shorter length.
Figure 18: Baa-Seek-RR--iisill------- reasoning trace example
23


Reviews:
[{"review_id": "4tEsp9pxaX", "reviewer": "Reviewer_FFwf", "summary": "This paper proposes a way to transform code generation benchmarks to verification quality benchmarks.\n\nGiven some coding problems and tests, they obtain a set of candidate programs for each of the problems and a gold ranking of these candidate programs based on tests. The quality of a verifier is determined by how good the predicted ranking matches the gold ranking.\n\nThey used this recipe to transform HumanEval and MBPP to verification benchmarks and evaluate several models' ability to generate tests.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Test generation and their evaluation is an important topic to be discussed.\n\n2. The metrics proposed are multi-facet and cover different aspects of the tests.\n\n3. The techniques and findings are intuitive.", "reasons_to_reject": "1. Lack of evaluation on more modern coding benchmarks. HumanEval and MBPP are two benchmarks that are largely saturated and too simplistic for modern LLMs. It would be great if you could evaluate the recipe on more modern benchmarks such as NaturalCodeBench, LiveCodeBench, etc.\n\n2. A bit overclaiming about reinforcement learning. On lines 5 and 21, the authors claim that test quality is important for reinforcement learning. While this is an understandable claim, it is not supported by downstream experiments in code-related RL. On line 42, the authors use \"reinforcement learning needs a fine-grained score or assessment of a coding solution\" to justify their method. Is there any definitive evidence that supports this claim? Because people seem to be using mostly binary rewards these days.", "questions_to_authors": "Since the authors claim that fine-grained score is important for reinforcement learning, I wonder if there's any concrete evidence that supports this claim. Also, there are several metrics", "extra_scores": {}}, {"review_id": "xcs2AbgK7v", "reviewer": "Reviewer_3nRV", "summary": "This paper proposes a pipeline to repurpose datasets (HE-R, HE-R+, MBPP-R, MBPP-R+) for benchmarking the performance of synthetic verifiers based on the existing code benchmarks. The experiments on the synthetic verifiers with different LLMs show that the reasoning models can significantly improve the quality and scale of test case generation", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It provides a pipeline to transfer the existing code generation benchmark to the code verifier benchmark.\n2. Based on the repurpose benchmark, it shows several interesting findings on the performance of the LLM-based verifier in test case generation and reward modeling. It also provides a detailed analysis of the error patterns and the effect of test case number.", "reasons_to_reject": "1. There is no guarantee of the reliability of the new benchmark. The ground-truth score and rank of the repurpose benchmark are based on the limited number of predefined test cases. As shown in Table 1, the number of predefined test cases in HE-R and MBPP-R is small, and these cases may not be specific enough to identify the correct and incorrect solutions. There is no specific way mentioned in the proposed transform pipeline on how to measure the reliability of the transformed benchmark. For example, how many test cases in the original dataset can make the new benchmark reliable enough for evaluating the verifier?\n2. There may exist self evaluation bias: the verifiers that share the same LLMs with the solution generation model may have a higher score. For example, it is unclear whether the better performance in GPT-4o, o1-mini, and o3-mini is affected by the fact that these incorrect solutions are generated by GPT-4o.", "questions_to_authors": "What if the synthesis of incorrect or partially correct solutions diverges from the real incorrect solutions that LLMs will generate? Will this lead to a misalignment between the evaluation result of verifiers and their actual performance when used as a reward model for ranking or RL learning?", "extra_scores": {}}, {"review_id": "W7NAhJla7z", "reviewer": "Reviewer_WWNV", "summary": "This paper presents an approach to scoring code generation in a nuanced way that supports comparison/ranking by leveraging RL preparation for reasoning models.  This is important work since correct vs incorrect designations are too blunt to adequately address the need for finetuning.  The prior work section is nicely written, with each paragraph that summarizes an area of work ending with a statement about how this work contributes over-and-above what came before.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There is a clear need for this work.  The technical approach is clearly described, and the results look promising.  The graphs in the results section especially make a compelling case for the strength of the results.  New benchmark datasets for sharing are created and validated.", "reasons_to_reject": "No real reason to reject -- this is a well done piece of work.  If I was pushed to find a weakness, I would say it's that the paper takes advantage of resources that are on the frontier currently in the field, but though the approach is resourceful, it's not particularly creative.\n\nSmaller comment: the writing is sloppy.  I see many grammatical disfluencies throughout.", "questions_to_authors": "I don't have any major questions.  As mentioned above, the related work section does talk about why the approach described is an advance over past work, but I wonder what the authors would argue is their main spark of insight/innovation? How should this work push thinking forward in the field?", "extra_scores": {}}, {"review_id": "Tjb6uMV2Ru", "reviewer": "Reviewer_49g2", "summary": "This paper is in the domain of test case generation for programs and precisely on the evaluation of test case generation. It generates a large number of solutions to benchmark questions, with a voluntary varying quality. Next, it uses these solutions to evaluate test case generation models, as the generated test cases should give high scores to high quality solutions and low ones to low quality solutions.\n\nThis work also allows to compare test case generation models with another kind of synthetic verification model, namely reward models. The results show that the latter are less efficients to judge solutions quality.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written. The experiments are convincing at confirming the utility of their solutions generation as a way to evaluate test case generation models.", "reasons_to_reject": "I may have missed important flaws, but I do not see obvious reasons to reject this paper.", "questions_to_authors": "The justification of generating 10 test cases by a \"feeling\" of adequacy is not acceptable. And why a 3 seconds timout, in the same paragraph and not 2, 2.5 or 4 ?\n\n\nA few typos or presentation problems:\n- abstract: \"a an\"\n- page 2: \"a benchmarks\"\n- page 4: \"Creations of the…\"\n- page 4: \"used use\"\n- page 5: correct citations like \"Qwen Qwen…\". The repetition is not necessary\n- pahe 5, line 158: \"For evaluating\" ?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a pipeline and recipe for transforming code benchmarks into new datasets to evaluate synthetic code verifiers. Reviewers generally acknowledge the need for this work, appreciate the clear presentation, and find the results promising. However, some reviewers raise questions and concerns about the recipe's generalizability and the choice of HumanEval and MBPP as source benchmark datasets. Although the authors provide a comprehensive rebuttal, it does not entirely alleviate the reviewers' concerns. After carefully considering the reviewers' comments and the authors' responses, I believe these concerns are valid but not critical. To address them, the authors should incorporate their responses into the next version of the paper and, preferably, test their framework on additional benchmarks to further demonstrate its effectiveness.
Decision: Accept