The Limits of Inference Scaling Through Resampling
Authors: Benedikt Stroebl
Sayash Kapoor
Arvind Narayanan
Venue: ICLR 2026 Poster (2026)
Keywords: Inference scaling
Resampling
Reasoning
LLMs
NLP
Primary area: generative models
TL;DR: Resampling with imperfect verifiers has fundamental limits, making it impossible for weaker models to match stronger ones, regardless of compute budget.

Abstract: Recent research has generated hope that inference scaling, such as resampling solutions until they pass verifiers like unit tests, could allow weaker models to match stronger ones. Beyond inference, this approach also enables training reasoning models, where data is curated using rejection sampling against a verifier. However, we show that this approach is fundamentally limited when verifiers are imperfect and have a non-zero probability of producing false positives. Resampling cannot decrease this probability, so it imposes an upper bound to the accuracy of resampling-based inference scaling, regardless of compute budget. Our analysis shows that there is a strong correlation between the model’s single-sample accuracy and its false positive rate on HumanEval and MBPP, whose unit tests have limited coverage. Therefore, no amount of inference scaling of weaker models can enable them to match the single-sample accuracy of a sufficiently strong model. Empirical results show that optimal sampling attempts are often fewer than 10, as the negative utility of false positives outweighs benefits, bending inference scaling curves downward. Finally, false positives may have other undesirable qualities, like poor adherence to coding style conventions.

Published as a conference paper at ICLR 2026
THE LIMITS OF INFERENCE SCALING THROUGH RESAMPLING
Benedikt Stroebl Sayash Kapoor Arvind Narayanan Princeton University
(stroebl, sayashk, arvindn) @princeton....omo edu
ABSTRACT
Recent research has generated hope that inference scaling, such as resampling solutions until they pass verifiers like unit tests, could allow weaker models match stronger ones. Beyond inference, this approach also enables training rea- soning models, where data is curated using rejection sampling against verifier. However, we show that this approach is fundamentally limited when verifiers are imperfect and have non-zero probability of producing false positives. Resampling cannot decrease this probability, so it imposes an upper bound to the accuracy of resampling-baa based inference scaling, regardless of compute budget. Our analysis shows that there is strong correlation between the model's 's single-sample accu- racy and its false positive rate on HumanEval and MBPP, whose unit tests have limited coverage. Therefore, enable them to match the sangle-sampl accuracy of a sufficiently strong model. Empirical results show that optimal sampling attempts often fewer than 10, the negative utility of false positives outweighs benefits, bending inference scaling curves downward. Finally, false positives may have other undesirable qualities, like poor adherence to coding style conventions.
amount of inference scaling of weaker models
INTRODUCTION
Scaling the amount of compute used during inference promising way to improve LLM per- formance. Techniques include reasoning (Wei al., 2023 Wu et al. 2024 Setlur et al. 2024).
reflecting on model outputs to revise candidate solutions (Shinn et al. 2023 Zhong et al. 2024), and
compositions of these and other atomic techniques (Saad-Falcon al. 2024 Welleck al. 2024).
Inference scaling through resampling stands out for its simplicity and broad applicability. It works by generating many candidate outputs until one is satisfactory, based on feedback from verifier (Song) et al., 2024; Qin et al.) 2024] Brown et al.] [024]] Hassid et al.] [2024) Li et al.) (202a)). Unlike techniques such as majority voting where gains from inference scaling quickly plateau (Table I resampling has given rise to the hope of usefully scaling inference compute by many orders of
magnitude.
We provide evidence that tempers this assumption. Our key concern is that the generalization gap- where model performs well on benehmarks but fails to generalize the real-world amplified when using repeated sampling to lift the performance of weaker models.
Specifically, we study the use of unit tests as verifiers for coding benchmarks, to see inference scaling for less capable models allows us to match the accuracy of more capable models. We make the following contributions.
Review of inference scaling techniques and their limitations (Section 2). We review papers inference scaling, categorizing the primary techniques and listing their domain domain-specific applications and known limitations.
Demonstration of generalization gap (Section 3), We provide empirical evidence on two bench- marks, HumanEval+ and MBPP+ (Liu al. (023b), showing that the apparent gains from resampling with imperfect verifiers are unlikely to translate into real-world performance. Despite achieving comparable results stronger models on standard unit tests, less capable models suffer from larger

---

Published as a conference paper at ICLR 2026
Llama 3.1 70B
1.0
Accuracy Conditional on Passing HumanEval Unit Tests
0.8
GPT-4o
0.9
GPT-40's Pars@]
Phi-3 Vini
Hema 1.1 605
0.6
Command
Piic Muduum Lama 3.1 88
Uiemw 70n
0.8
Code6en2 CodaCamn
SurCadee- dd
*
Command Code
Code 76
Reward
0.4
Meura TaMaer 7n
CodeTSp
Retio
0.7
Lummand-iigtt
0.2
C/B Ratio CaBio
C/B-Ratio
C/B-Ratio
0.6
0.0
C/B Ratio
Canditional Pass@1
0.22x 0.72, 0.891, p 0.000
0.5
0.2
0.4
0.6
1.0
-0.2
100
10 K (Attempts)
10a
Single-Sample Accuracy (Pass@1)
Sample-sample accuracy resampling limits HumanEval+. The x-axis shows single-sample ac- curacy on HumanEval+ (which contains comprehen- sive unit tests), while the y-axis shows the highest achievable accuracy when resampling with an infinite compute budget, using HvmanEval's more limited unit tests as verifiers. Weaker models (models with lower single-sample accuracy) produce false positive solutions at higher rates. Models below the cutoff line are unable match GPT-4o through resampling, as GPT-4o's Pass@ exceeds the accuracy of such a model even when conditioned its solutions passing the unit tests. Results on MBPP+ follow a similar pattern (Figure 3)
The cost of false positives limits the reward of resampling. False positives w.r.t. an imperfect verifier (HumanEval unit tests) incur a "cost" (e.g., subtle bugs in code), while correct answers provide benefit. The reward (y-axis) depends on this cost-benefit ratio (C/B-Ratio), Problem instances that require more attempts tend to be harder, hence positives. Thus, even with zero computational cost, for realistic cost-benefit ratios, the optimal number of samples K is finite and very low.
susceptible to false
Figure 1: Overview of our main findings.
generalization gap gap-produiing incorrect solutions that fail the extended test suite (false positives) at higher rates than stronger models.
In particular, we observe that even if given an infinite inference budget, in many cases weaker model cannot match the performance of a single invocation of a sufficiently strong model.
Empirical analysis to understand the limitations of inference scaling with imperfect verifiers (Section[4). We examine how introducing a cost (negative utility) for returning false positives impacts the optimal number of resampling attempts inference budget, the optimal number of samples is often finite and very low (e.g., K < 5 in Figure Hence, resampling quickly reaches a point of diminishing returns without bridging the performance gap for smaller models. If the cost of an incorrect solution is higher than the benefit of correct solution, the optimal K can be zero the risk of false positive for a weak model is high enough that it is effectively useless (Figure 4p. In Section C we present theoretical model that complements the findings this section.
HumanEval+. We find that even with infinite
We show that the reliance on imperfect verifiers not only affects the functional correctness but also overall quality of the generated code. We evaluate candidate solutions on HumanEval+ based on various readability metrics such as adherence to naming conventions (that we specify in the prompt) like snake_ca se and came 1 Case, line-level commenting, and guidelines regarding the maximum line length and number lines in function implementations. We find that false positive solutions are lower quality across all models and metrics when compared to true positive solutions. While other aspects of code quality such as simplicity and modularity are harder to test automatically, we speculate that the same pattern holds for those properties as well.
Evidence that this affects code quality beyond correctness (Section 5).
We also conduct a qualitative analysis to identify recurring error types causing a larger generalization gap for weaker models.
2

---

Published as a conference paper at ICLR 2026
Generator
Generator
Generator
Solution Candidates
Reject
Propose
Majority
voting
Verifier
Accept
Output
Ensembling
Ranking
c) Verification
Figure 2: Schematic illustration of various resampling techniques for inference scaling.
Our findings have three additional implications. First, they show the importance of building highly accurate verifiers. This goal might benefit from treating verification technology as a specialized subfield with its own metries and benchmarks. This especially true for training-timm time uses: models trained with feedback from imperfect verifiers may learn to exploit weaknesses in the verifier rather than solve tasks robustly, potentially leading safety concerns (Krakovna et al. 2020 Amodei et 2016).
Second, the use of imperfect verifiers as the ground truth for evaluation is flawed. We used Hu- manEval+ and MBPP+ for evaluation; the gaps we identify would have been invisible if we had used HumanEval and MBPP both as verifiers and as benchmarks. While the limitations of these benchmarks for measuring absolute performance are well known (Zhang et al. 2024b] Liu et al.) 2023b), our results show that they might result misleading comparisons between models as well.
Third, our findings highlight limitations in resampling-basee data curation for reasoning models. Reasoning models rely on datasets curated through rejection sampling against verifiers. When these verifiers imperfect, the curated datasets risk including mislabeled examples, which incurs cost on model performance. This introduces bottleneck: without stronger base models or highly accurate verifiers, the gains from resampling-basee -baee data curation to train reasoning models are likely limited.
While we do not claim that resampling is the predominant scaling technique, our findings suggest that the persisting gap between oracle and imperfect verifiers should be taken seriously and could pose limitations across inference scaling strategies. We invite research on ways to mitigate the issues identified this work.
SCALING INFERENCE COMPUTE WITH VERIFIERS
Table provides overview of the main techniques for scaling inference compute with LLMs.. Some methods such as majority voting [Wang et al. 2023b Chen et al.; 2024a) or resampling using verifiers (Brown et al. 2024 Xin al. (0024) generate many candidate solutions and then select one. Other methods such as reasoning (Wei et al. 2023) and critique (Shinn et al. 2023 Madaan al. 2023) refine single solution. In practice, these methods can be combined in flexible ways and the distinction between them is not always clear (Section A.I Note that our notion of inference scaling excludes methods such as those used train OpenAl's of series of models, since we are only looking improvements during inference time to available language models, rather than training improvements.
All these methods except verifier-based resampling are known to have important limitations that cast doubt on how much scaling is truly possible, as summarized in Table Resampling using verifiers has different control flow than other methods, which gives it an intuitive appeal (Figure 2 we can potentially regenerate solutions indefinitely until one is correct. This enthusiasm around resampling partly driven by the empirically observed inference scaling laws, which suggest that the fraction of tasks for which we find at least one correct solution scales predictably with the number of samples multiple orders of magnitude (Brown et al. 2024
3

---

Published as a conference paper at ICLR 2026
Table 1: Overview of inference scaling techniques. This table shows the main categories of techniques for inference scaling along with their descriptions and known limitations. Note that rankers, often implemented with reward models, are sometimes referred as verifiers and the boundary can be unclear.
Technique Ranking
Description
Limitations
Scores and ranks the best samples from multiple candidates (Cobbe al. 2021
Doesn't 't scale with sample budget (Brown et al.
2024]
Underperform compared to other methods Zhang al. 2024a
Hassid al. 2024
2023a Lightman el al. 2023
Liu el al. 2023c Hosseini
ct al.
2024b Kirchner et al.
2024:
Setlur et al. 2024; Snell
2024] Vacareanu al.
2024
Chen et al. 2002cc
Majority Voting
Using consensus among multiple samples to determine final output Wang el al. 2023b
Hurts performance hard tasks and
monotonous scaling under task heterogeneity (Chen
ct al. 2024a)
Lt 2024b Wang et al.
Sample inefficient for queries with many answer pos-
2024b
sibilities (Wang al. 20246
Limited applicability for tasks with non-discrete
swers
Oracle Verification
Leverages ground-truth evaluator for free until correct
Not available for most domains
solution found (Xin et al.]
2024 First et al.
2023
Imperfect Verification (This paper)
Scores and accepts rejects candidate solutions (Zhang
Bigger generalization gap for weaker models (Sec- tion
et al. 2024a Davis et al., 2024
Optimal number of samples finite and low (Sec- tion
Yao et al. 2023 2003 Gundawar
et al. 2024 Kambhampati
Low code quality of false positives (Section
2024
However, the usefulness of this depends the availability of a capable verifier (Davis et al.) 2024). In some settings, we may have an oracle verifier, such as a proof checker, that does not suffer from false positives- that is, if the proof checker verifies the proof, it is guaranteed to be correct. False negatives of the verifier (including nontermination under a fixed compute budget) are less of a problem, as one can simply generate more samples until true positive is found. It is possible that every correct solution is a false negative of the verifier, but it is unclear if this is problem that arises practice.
But in other settings such as coding and reasoning, we only have imperfect verifiers such as unit tests or LM judges, which suffer from false positives: incorrect solutions that nonetheless pass the verifier. In these settings, we don't 't have easy methods to guarantee the correctness of generated solutions at inference time. As a result, we cannot distinguish between false positives and true positives simply by increasing the compute budget. We survey papers that use verifiers in Table 2
In this paper, we investigate effect scaling inference compute with access to imperfect verifiers. We distinguish ranking from verification: ranking selects the best among multiple candidates, while verification accepts or rejects each candidate independently against correctness criterion.
3 REPEATED SAMPLING WITH WEAKER MODELS LEADS TO WORSE GENERALIZABIIJITY
In computer programming tasks, unit tests are commonly employed as verifiers to assess the correct- ness of candidate solutions generated by language models. While unit tests are practical and efficient, they often suffer from imperfect test coverage, leading to false positives where incorrect solutions pass the tests (Gulwani et al. 2017). This affects many benchmarks such HvaanEval (Chen et al.
4

---

Published as a conference paper at ICLR 2026
Paper
Verifier
Verifier Type
Domain
Verifier Implementation
Category
Chen et al.
2022
Imperfect
Unit tests
Coding
Checks agreement of tests and sam-
ples
Shinn ct al.
2023
Imperfect
LM-as-judge.
Coding, QA
LLM evaluator generates decision re- wards
Unit tests
Yao et
2023
Imperfect
LM-as-judge
Planning
L.M evaluates reasoning steps
First ct al.
2023
Oracle
Proof checker
Math
Proof checker
Thakur a1 2024
Oracle
checker
Math
Proof checker
Yang al.
Oracle
Proof checker
Math
Proof checker
Wang
2025a
Oracle
Proof checker
Math
Proof checker
Azerbayev et
2024
Oracle
Proof checker
Math
Proof checker, Majority voting
Huang
2024
Oracle
Proof checker
Math
Proof checker
Xin et al. K2024
Oracle
Proof checker
Math
Proof checker
Brown el al.
21124
Oracle
Proof checker
Math
Proof checker
Davis ct al.
2024
Imperfect
LM-as-judge
QA, Math
LLM judges correctness of genera-
tions
Hassid et
2024
Imperfect
Unit tests
Coding
Unit tests
Zhang al
2024a
Imperfect
Generative RM
Math
Verfication as part of the model out-
put
Zhuge al.
2024
Imperfect
Agent-as-judge
Agents, Coding
Agents evaluate outputs other agents
Kapoor al.
2024
Imperfect
Unit tests LM-as-judge
Coding
Unit tests
Saad-Falcon ct al.
2024
Imperfect
Coding, Reasoning
LLM judges correctness of genera- tions
Liang et al.
2024
Imperfect
Program-of- thought
Coding, Math
Checks CoT against generated PoT
Cook et al.
2024
Imperfect
LM-as-judge
Instruction- following
LLM checks answer against gener- ated checklists
Gundawar 2024
Imperfect Agent-as-judge
Travel planning
Pre-defined constraints verified by critic agents
Table 2: Survey of papers on LLM verification methods, their approaches, and specific verifier implementations.
2021 APPS (Hendrycks al. 2021), or MBPP (Austin et al. 2021). This imperfection raises the question: Do less capable models produce false positives implementations that pass the standard unit tests but fail the comprehensive ones-at higher rate than stronger models?
Experimental setup. To investigate this, we conducted experiments on two widely used coding benchmarks: HumanEval+ and MBPP+. MBPP consists of simple programming tasks designed to evaluate the basic coding abilities of models (Austin al. [202I). HumanEval+ and MBPP+ are extensions the original HumanEval and MBPP benchmarks (Liu et al. 2023b) and contain additional hidden test cases to correctness beyond the unit tests included in the original benchmarks.
We evaluated models of varying capabilities, including weak and stronger models, generating at least 200 samples per task and model on HumanEval+ (1,000 for Command Light, to minimize tasks without any passing solutions) and 50 samples per task MBPP+. To assess the generalization gap, we then evaluated solutions that passed the original benchmark test sets on the more comprehensive hidden test cases (see Section D for details). These are extensive, and assume that solutions that pass the full set of tests are correct. (If this assumption is not true, the generalization gaps that we reveal only grow bigger.)
Findings. Weaker models exhibit higher probability of producing false positives compared to stronger models (Figure B). This probability scales inversely with the true capability. This linear relationship holds with remarkable consistency across models of various families, including Cohere's Command models, GPT-40, and the Llama-3.1 family. This suggests that while weaker models appear to perform well standard benchmarks through increased sampling, they fail to generalize effectively and, importantly, they generalize worse than more capable models. They tend to generate fragile solutions that exploit the limitations of the unit tests. We speculate that this is because weaker models' "true understanding" of the programming tasks is worse.
The empirical results reinforce
insight. Suppose IP Patrong (Correct) F Bweak ( (Correct Pass Verifier). That is, the single-sample accuracy of a strong model exceeds that of weaker model, even conditioned the weaker model passing the hase unit tests. Then the weaker
5

---

Published as a conference paper at ICLR 2026
model cannot match the performance of single invocation of the stronger model, no matter how big the compute budget for the weaker model. In Figure [] this is shown by a horizontal line. No model below the line match the performance of GPT-4o through resampling.
1.0
Passing HumanEval Unit Tests
Accuracy Conditional on
0.9
0.9
GPT-4e's Paseg]
4050
CPT-3 C11140
Cammenie
P1133 3 Medium
Uamo
Crrmand Midral Small
3.1
0.8
Command code Usme 130
Liara 3.1
0.8
Nitrol Large
Liwme 3.1 4052
SteCeder
Cdds uama 78
GFT-40's Pacog]
Weuni 75
70
0.7
Commundd gh:
CodeT5p 108
0.7
Command
0.6
0.6
0.22X 0.72. R² 0.891. 9.000
0.23% 0.87. 0,892. 0.000
0.5
0.5
0.2
0.4
0.6
0.3
1.0
0.2
0.4
0.6
0.8
1.0
Single-Sample Accuracy (Pass@1)
Single-Sample Accuracy (Pass@1)
Hvaa
MBPP+
Figure 3: Generalization gap with infinite compute budget. We show the relationship between the
accuracy of individual samples (X-axis) and the achievable accuracy given an infinite compute budget and limited unit tests (y-axis; note that it starts 0.5). We evaluate performance on the extended test suites of HumanEval+ and MBPP+, using the unit tests from the original benchmarks as verifiers. For both benchmarks, the trend is that less capable models are more likely to generate false positives than stronger models. In Section B we show our results with the full y-axis as well as upper (lower) bounds on the conditional accuracy accounting for tasks for which we did not observe any solutions passing the unit tests.
The effect is largely driven by subset of the tasks where the unit tests are poor. When we limit the analysis to these tasks, the relationship is even more pronounced Section B
Note that results rely human- aunan-generated unit tests verifiers. In practice, we might expect to use magguag--odel--gneeee how the findings change when the unit-test verifiers are LLM-generated.
unit tests for inference-tin citrenccettmeeeeee verification. It is an open question as to
HOW MANY SAMPLES ARE OPTIMAL?
In the previous section we looked at the behavior of resampling in the limit as the number of samples grows large. Now we look at inference scaling curves, which allow us to study how accuracy varies as a function of the number of samples.
We add one important detail:
we model the cost of false positives, such code that passes unit tests but has subtle bugs. The cost of bugs (which might result in buggy software being deployed) is not easily comparable the labor-saving benefit of correct solutions, and this cost-benefit ratio can vary greatly depending on the application. So we consider many possible values for the cost-benefit ratio, including zero, which is the setting considered in previous work on inference scaling. The ratio can potentially be much higher than in some applications, such as security sensitive ones, since bugs might translate exploitable vulnerabilities.
Experimental setup. For each model of interest, we generated 200 samples for each task in the HumanEval benchmark. For each 200, If a passing solution was found within K samples. we assigned rewards based on the outcome: a true positive yielded a benefit of 1, while a false positive incurred cost, with values set according to different cost-benefit ratios: 0, 2, 4, or 8 (Figure $. no passing solution was found within K samples, we assigned reward of 0 (both, cost and benefit are O). We repeated this whole process 1,000 times and computed the mean reward for each K. The set of samples was the same in all 1,000 runs, but the order of samples was randomly permuted. This setup allows us to empirically observe the relationship between the number of sampling attempts K and the reward for various cost-benefit ratios. The results illustrated in Figure 4 for the Llama-3.1 (Dubey et al.) 2024) and Code Llama (Rozière et al., 2024) model families. The scaling curves for GPT-4o are included in Figure 15

---

Published as a conference paper at ICLR 2026
Conditional Pass@1
C/B-Ratio
C/B-Ratio
Q/-RRatio
C/B-Ratio
C/B-Ratio
Llama 3.1 8B
Llama 3.1 70B
Code Llama 7B
Code Llama 13B
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
Reward
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
-0.210º
10 K (Attempts)
102
-0.210
10"
10
-0.210
10!
10'
-0.210º
10
10! K. (Attempts)
10'
K (Attempts)
K. (Attempts)
Figure 4: Inference scaling curves in the presence of cost for false positives. We show the reward as a function of the number attempts K across various cost-benefit ratios for the Llama-3. and Code Llama model families. Crosses mark the optimal number of samples for each setting. Standard inference scaling curves with no cost (i.c., cost-benefit ratio is 0) are provided for reference. We find that, even zero computational cost, there is finite optimal number of samples K that is often very low.
Llama 3.1 70B
Code Llama 7B
0.8
False Positive
0.2
0.0
8-15
16:31
32-63
64-127
0.0
8-15
16-31
32.83
64-127
RRnge (Powers of 2)
K Range (Powers of 2)
Figure 5: False positive rate function of the number of attempts K for Llama 3.1 70B and Code
Llama 7B on HumanEval+. We include plots for additional models in Section C
Findings. The results show that the effectiveness repeated sampling quickly reaches point of diminishing and even negative returns. Each additional attempt brings trade-off: although it might yield a correct solution, it might instead yield a false positive, and the false positive rate increases with K. (Figure 5. At first this is surprising, since sampling why it happens, we need to look at the distribution of task difficulty (Figure 6). which turns out to be strongly bimodal. The easy tasks get solved within few attempts, and for the remaining harder tasks, false positives are more likely. This aligns with findings by Chen et al. (2024a), who observed similar inverse U-shaped accuracy curves explained by the heterogeneity in task difficulties.
memoryless process. To understand
Thus, even with zero computational cost, the optimal number of samples is finite and low (Figure 4). For example, at cost-benefit ratio of 4, the optimal number of samples is K < 5 for all four models. If the ratio is high enough, the optimal number of samples is zero - the expected cost of false positive outweighs the expected benefit of a correct solution, the reward is always negative and is best not to attempt a solution at all.
We note one important caveat:
for some models such as -lamm-3.11-0B. the false positive rate increases dramatically with K, whereas for others such as the Code Llama and Command families, the increase much more gradual, resulting much higher values of the optimal K, especially for low cost-benefit ratios. We have not been able to identify any intuitive reason for this difference.
To summarize, weaker models cannot "sample their way" to top-level performance if the verifier cannot reliably filter out false positives because the risks quickly outweigh the benefits. Our findings in this section align with our theoretical model in Section which generalizes these findings to other benchmarks.
7

---

Published as a conference paper at ICLR 2026
5 FALSE POSITIVE SOLUTIONS ARE LOW-QUALITY EVEN BEYOND CORRECTNESS
While correctness is fundamental criterion for evaluating code generated by LLMs, it is not the only determinant of code quality. High-quality code possesses attributes beyond mere functionality, such as readability, maintainability, and efficiency. Readability simplifies error-checking and is considered one of the most useful properties of high-quality code (Börstler et al. 2023). It can be measured using various metrics, including code length guidelines (e.g.. PEP8), adherence naming conventions like snake _aaae case or camel Case, and consistent commenting (Zheng et al. (022). Intuitively, shorter code with clear variable names and informative comments is generally easier read and maintain.
To understand the relationship between imperfect verifiers and code quality, we evaluated the readability of code generated by various models in our setup.
Experimental setup. On HumanEval, we evaluate the readability scores of candidate solutions that pass standard unit tests and the more comprehensive test suite. For each measure of code readability, we use different prompt instructing the model to adhere to the desired guidelines (see Section D for detailed prompts). We rely on the prompts and implementation from Zheng al, (2024).
The results show notable differences in code quality between false positives and robust implementa- tions. False positives, passing only the standard but not the extended unit tests, tend to have worse code quality across all metrics (see Figure 7 This trend is consistent across models of varying capabilities. This suggests that the limitations of imperfect verifiers for coding tasks extend beyond correctness issues but also affect other code characteristies important for software development. This affects weaker models more, given that they are more prone to generate false positives.
An open question arising from our findings is whether fine-tuning LLMs on code quality metrics could improve not only the quality of generated code but also robustness (Jain et al. 2023 potentially mitigating the prevalence of false positives.
Llama 3.1 8B
GPT-4o
Liama 3.1 8B
GPT-4o
Fraction of Tasks
0.0.
0.2
0.6 Pass@1
0.4
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pass@1
Pass@1
Pass@1
HumanEval+
MBPP+
Figure 6: Distribution of task difficulties for Llama 3.1 8B and GPT-4o on HumanEval+ and MBPP+. We include barplots for all models on both benchmarks in Section B
Qualitative analysis of false positives. To better understand the characteristics of false positive implementations, we randomly sample 10 implementations across all models that pass the standard tests but fail the additional unit tests, 5 for each benchmark. Through manual analysis, we identified several recurring error types. All examples mentioned in the following are included in Section D.3
1. Logical errors: Such errors were common. For instance, in the HumanEval/30 task, the model is tasked with returning only positive numbers from a list. The solution shown Section D.3 incorrectly converts floats to integers, passing the basic tests that included only integers but failing extended tests that introduce float values.
2. Edge case handling: Sometimes solutions failed to account for atypical inputs, which happened to not be covered by the standard unit tests. For example, in HumanEval/6, the solution failed to handle an empty list input. It is important to note that tasks in HumanHyal and MBPP often are ambiguous solutions fail because they return an assertion error instead of an empty list for the edge case of getting an empty list as input. We expect that ambiguity should affect weaker and stronger models similarly, but have not tested this.
how edge cases should be handled. For example, for HumanEval/ 149 some
3. Ineffieient implementations: While most false positives result from logic errors or edge case mis- handling, some were also caused by inefficient implementations, For instance, in 5ym11Eyal/15.
8

---

Published as a conference paper at ICLR 2026
0.90 0.91
0.92 0.91
0.88
0.88
0.88
0.86
0.88
0.80
0.80
0.8
0.75
Probability
0.43 0.43
0.4
P(Adheres to camelCase
Pass Unit Tests)
P(Adheres camelCase Correct)
P(Adheres to camelCase False Positive)
0.0
GPT-40
GPT-3. 5 Turbo
Llama 3 708
Llama 3.1 88
Mistral Large
Mistral Small
Figure False positives tend to be lower-quality code than correct implementations. For example, false positive solutions fail to adhere to the came 1 Case naming convention more often than robust implementations. Figure I6 shows that this holds consistently across models and for all four code quality metrics we test. GPT-3.5 exhibits low relative performance in using came 1 Casse but performs comparably to other models in following snake_case (Section D). This has also been found in previous work (Zheng et al. 2024).
the solution involved for-loop that became inefficient when handling larger inputs, causing timeout on the extended tests. Following Liu et al. (2023b), we set the timeout such that each candidate solution must compute less than one second or four times the time takes run each test on the ground truth implementation, whichever is greater.
DISCUSSION
We study a setting where all generators are paired with the same verifier. The verifier has imperfect coverage, but false negatives. In real-world deployment settings, human-written unit tests rarely available and we would need to rely on the use of automated test generation techniques. These approaches include symbolic execution (Lukasczyk & Fraser 2022). specialized transformers (Tufano et al., 2021). and LLMs (Chen et al. 2024b 2022; Siddiq et al. 2024). Model-generated
tests introduce new challenges including a disparity between verifiers and risk of false negatives.
This could widen the generalization gap. We leave an investigation of the impact of model model-generated
unit tests as a next step.
Resampling is used not only to scale inference but also to train large reasoning models. Many of-te-of-the-arr reasoning models are trained on datasets curated through rejection sampling (NovaSky Team 2025 DeepSeek-AI et al.) 2025 [Bespoke Labs 2025). where verifiers filter out incorrect outputs. However, imperfect verifiers can introduce mislabeled examples, implicitly incurring a cost of false positives. We hypothesize that weaker models paired with imperfect verifiers fail to produce datasets of sufficient quality to train competitive reasoning models, creating a bottleneck: without stronger base models more accurate verifiers, gains from resampling-bast based data curation to train reasoning models are limited.
Although experiments focus coding tasks, want to emphasize that theoretical results are Nomain-agnosee resampling relying on imperfect verifiers with non-zero false positive rates will face the same fundamental ceiling.
Related work has also studied the risks of over-optimizing against imperfect rewards. For example, Gao et al. (2022) analyzes how optimizing against proxy reward models can lead degraded true

---

Published as a conference paper at ICLR 2026
performance. Our setting differs in that we focus on inference-ting resampling, where the ceiling on achievable accuracy arises directly from false positives rather than reward misalignment.
Finally, findings weaken support for previous papers' claims that resampling is effective strategy to increase accuracy by trading off inference time compute (Kapoor ct al. 2024 Chen et al. 2024a); here resampling with imperfect verifiers is inherently limited.
Limitations. Our experiments focus on repeated sampling in the context of coding tasks. Coding
offers clear example of the challenges posed by imperfect verifiers, other domains might exhibit different behavior. Future work could extend these findings to tasks such as reasoning (Hosseini
al.
2024a), web agents (Bai et al. 2024 He al., 2024), agent-user interaction (Yao al.
2024).
Another limitation is prompt sensitivity, which affects LLM evaluations (Biderman et al.
2024 Liang et al., 2023). While we followed the original authors' implementation provided with the
HumanEval+ and MBPP+ benchmarks (Liu et al. 2023b), prompt engineering could influence false positive generation. Additionally, we did not investigate how benchmark contamination affects our findings, models could be overly optimized for passing the standard test cases. We did not explore mitigation strategies such refining solutions after they passed the verifier (Saad-Falcon et al. 2024). Similarly, we did not test alternative strategies to inference scaling that, e.g., induce more diversity during sampling (Wang et al. 2024a). Finally, the optimal number of samples K that we identify depends on the cost-benefit ratio, which varies by application context. While show that K is finite and often very low across a range of ratios, we do not provide empirical evidence for what specific ratio applies in any given real-world deployment. In security-critical applications, the cost of false positive may far exceed the benefit of a correct solution, whereas in lower-stakes settings a higher K may remain practical.
We thank Boris Hanin, Peter Henderson, Alexander Wettig, Jon Saad-Falcon, and Zachary Siegel for providing valuable feedback on earlier drafts of this work. We thank Cohere for their support of this work with API credits. We also thank Schmidt Sciences for their funding and support of this work.
Acknowledgments.
Reproducibility. We release code to reproduce all experimental results of this paper in a GitHub repository]] This repository also contains all code samples for all models used in our experiments. Additionally, we provide an implementation of the theoretical model in Section C as a Python notebook.
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety, July 2016. arXiv:1606.06565.
16066 06565
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program Synthesis with Large Language Models, August 2021. URL htt.:ttg..//axii...rr///e/...... arXiv:2108.07733 [cs].
Zhangir Azerbayev. Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang. Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An Open Language Model For Math- ematies, March 2024. URL http:htttp///xxii.tt
abs /todb//23330 /2110. 1.0....000 10631 arXiv:2310.106
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. DigiRL: Training In-The-Win Device-Control Agents with Autonomous Reinforcement Learning, June 2024. URL
wwtp:/ii.rrg/aaa///aeeaaaaaaaa/ 2206. 1 11896 arXiv:2406.
Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distilla- tion. www.bespokelabs.oo hesspokeaas..iiblo//eepspp distillation, 2025. Accessed: 2025-01-22.
Stella Biderman. Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi,
10

---

Published as a conference paper at ICLR 2026
Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the Trenches on Reproducible Evaluation of Language Models, May 2024. hRR.httt::/
org abs 2405. 14782 1422
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, July 2024. URL http://arxiv. org abss 2407 21787 2Xiv:2407.21787 [cs].
Jürgen Börstler, Kwabena E. Bennin, Sara Hooshangi, Johan Jeuring, Hieke Keuning, Carsten Kleiner, Bonnie MacKellar, Rodrigo Duran, Harald Störrle, Daniel Toll, and Jelle van Assema. Developers talking about code quality. Empirical Softare Engineering, 28(6): 128, September 2023. ISSN 1573-7616. doi: 1111000/110644002---1 00007//00666-0--1188---0 /tL
1007 010.:/116444444 023 10381-0
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Code'T: Code Generation with Generated Tests, November 2022. URL httpttt:p//axiivorrrtt
abs/22077 10397 htt.:2207.113997
Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou, Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems, March 2024a. URL
org/ abs ab//aa/s///445 2200. 02419. arXiv:2403.02419 [cs, eess].
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained Code, July 2021. URL http: arxiv. orq/ wwt.ab/////aoooooooooco//oeaaeee awaaa//l1l 2200 03374 arXiv:2107.0333 [cs].
Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, pp. 572-576, New York, NY, USA. July 2024b. Association for Computing Machinery, ISBN 9798440006685. doi: 10. N0.1145/3633529336666 3663880 URL http://pppp.dd..eeeoooor/ooo//// /tt.://0.1111/33363333 3663801
Ziru Chen, Michael White, Raymond Mooney, Ali Payani, Yu Su, and Huan Sun. When is Tree Search Useful for LLM Planning? Depends on the Discriminator, June 2024c. URL http: 0890 arXiv:2402. 10890.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. URL http: //axxiv.org/abs/2/ 14168 arXiv:21 10. 14168 [cs].
Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation, October 2024. URL http://arxxv..rr//abb./
200.4000.00.... arXiv:2410.0360..
Jared Quincy Davis, Boris Hanin, Lingjiao Chen, Peter Bailis, Ion Stoica, and Matei Zaharia. Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design, July 2024. URL http////ttpp//////.../o/////////
16831 arXiv:2407.16833 [cs].

---

Published as a conference paper at ICLR 2026
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang. Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang. Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang, R. J. Chen, R. L.. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu. Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu. Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi. Yiliang Xiong. Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-R11 R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 1 2948 arXiv:2501. 122449 [cs].
January 2025. URL http:h/tttt:iii/e wwwpxiivvorrraaeoo .or.oog///aaaa hatp://b//55550
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of Verifictiin Reduces Hallucination in Large Language Models, September 2023. http::////ar org/ abs 22099 11495 arXiv:2309.1
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy. Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve. Gabrielle Lec, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi. Patrick Alrassy, Pengchuan Zhang, pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava. Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit
12

---

Published as a conference paper at ICLR 2026
Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh. Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisemman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock. Bram Wasti, Brandon Spence, Brami Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang, Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer. Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina- Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- -apttste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal I Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar. Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji. Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani. Pritish Yuvraj, Qian Liang, Rachad A Alao, Rachel Rodriguez, Rañ Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto. Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar. Shuqiang Zhang, Shugiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang.
Andrei
13

---

Published as a conference paper at ICLR 2026
Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He. Zach Rait, Zachary Vitoto, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Llama 3 Herd of Models, August 2024. UtL.ttt:://araa
org/ abs / 2407 2tt.//2477..2.... 21783 arXiv:2407.21788
Emily First, Markus N. Rabe, Talia Ringer, and Yuriy Brun. Baldur: Profe-Proof Generation and Repair with Large Language Models, March 2023. URL httt:://axxi...r///abbbbaa
04910 0 arXiv:2303.04910.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
http://rrrrra///........ 10760
Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. Program Synthesis. Foundations and Trends in Programming Languages, 4(1-2):1-19, July 2017. ISSN 2325-1107, 2325-1111. doi: 10.1561/25000000000 (RI.ttpp:: / //t:://wwwww www. nowpub hespouullssheee ishers com PGL- -GL-010.
Publisher: Now Publishers, Inc.
Atharva Gundawar, Karthik Valmeekam, Mudit Verma, and Subbarao Kambhampati. Robust Planning with Compound LLM Architectures: An LLM-Modulo Approach, November 2024. URL http:
/ www.ii.orgabbba..........o 14484 arXiv:2411.1444
Michael Hassid, Tal Remez, Jonas Gehring, Roy Schwartz, and Yossi Adi. The Larger the Better? Improved LLM Code-Generation via Budget Reallocation, July 2024. URL http:://axxxi obs/abs// 2404. 00725 arXiv:2404.0072. [cs].
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Web Building an End-to-End Web Agent with Large Multimodal Models, June 2024. URL
13919 arXiv:2401.13919.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding, January 2021. URL http: 03300 arXiv:2009.03300 [cs].
//a //arxiv. org/ http:/bbb/2000.0. wwxxiv.oxg/abb//209..o
Arian Hosseini, Alessandro Sordoni, Daniel Toyama, Aaron Courville, and Rishabh Agarwal. Not All LLM Reasoners Are Created Equal, October 2024a. URL http::/tttxtttitee org/ abs/ abs/2410
01748 arXiv:2410.01744 [cs].
Arian Hosscini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-STaR: Training Verifiers for Self-Taught Reasoners, August 2024b. URL http:
06457 urXiv:2402.06457 [cs].
Yinya Huang, Xiaohan Lin, Zhengying Liu, Qingxing Cao, Huajian Xin. Haiming Wang, Zhenguo Li, Linqi Song, and Xiaodan Liang. MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data, May 2024. URL http:http///xii/// org/ wwwp:o/gg//aoo./.oooooooo 2402. 08957 arXiv:2402.08957.
Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E. Gonzalez, Koushik Sen, and Ion Stoica. LLM-Assisted Code Cleaning For Training Accurate Code Generators, November 2023. URL arXiv:2311. 1.1994.
http: org/ abs///b//// 22111 14904
Subbarao Kambhampati, Karthik Valmeckam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bham- bri, Lucas Saldyt, and Anil Murthy, LLMSs Can't Plan, But Can Help Planning in Loduloodulo Frameworks, June 2024. URL http:: /ttt:://arxivv org/ abs 2402. 01817 arXiv:2402.01817
Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. Al Agents That Matter, July 2024.
abs /atp//abb////0.0 01502 arXiv:2407.01502.
Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover- Verifier Games improve legibility of LLM outputs, August 2024. URL http:://axix:
org/abs/ 2407. 1...113666 13692 arXiv:2407.1 13692 [cs].
Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg. Avoiding Side Effects By Considering Future Tasks, October 2020. htLptt::: : arriiv. orq/abs torg/abs/2111 2010
07877 arXiv:2010.07877.
14

---

Published as a conference paper at ICLR 2026
Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7B Language Models Already Possess Strong Math Capabilities, March 2024a.
org/ abs 2403 04 706. arXiv:2403.04706.
Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More Agents Is All You Need, 05120 arXiv:2402.0511 200102 version:
February 2024b.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren. Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr. Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha. Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic Evaluation of Language Models, October 2023.
URL http: /ttp://aaxvvvvv.i org/ aww./b/22222 09110 arXiv:2211.09110
Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, and Semih Yavuz. Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification, October 2024. URL http///aavvvv/e. org als 2210..55318 arXiv:2410.05318.
Hunter Lightman. Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step. May 2023. URL
http:// http://axivv... obtp:///rgbbbs/33333b 2005
arXiv:2305.20050.
Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. TinyGSM: achieving >80% on GSM8k with small language models, December 2023a. URL/ttt:://arxtt wtt.:o/gggabsa/..aa. 09241
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. November 2023b. Utt/ wtwp:/rrwwiieeeeeeorrreeee
-= qvx 10c7
Yixin Liu, Avi Singh, C. Daniel Freeman, John D. Co-Reyes, and Peter J. Liu. Improving Large Language Model Fine-tuning for Solving Math Problems, October 2023c. URL htt.::
org/ 2310. 10047 arXiv:2310.10047.
Stephan Lukasezyk and Gordon Fraser. Pynguin: automated unit test generation for Python. In Proceedings of the ACM/EEEE 44th International Conference on Software Engineering: Com- panion Proceedings, ICSE '22. pp. 168-172. New York, NY, USA, October 2022, Association for Computing Machinery. ISBN 978-1-4503-9228
doi: 10.1145/31044443366899 URL https://dl.. acm. org /otp:///oooooooooooooooooooooo 100 /ttp//0..111151114443333113
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Refin: Iterative Refinement with Self-Feedback, May 2023. URL UBBlitt:://arxiv.crr//aaas 2330/ 17651 arXiv:2303.1766
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingho Zhou, Silvio Savarese, and Caiming Xiong. CodeGen: An Open Large Language Model for Code with Multi- Turn Program Synthesis, February 2023. URL arXiv:2203.13474.
NovaSky Team. Sky-tl: Train your ol preview model within 450. https ai w..github.io//ostte sky t1, 2025. Accessed: 2025 01 09.
//novasky
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting, March 2024. URL http:: / /axxivvvvccomomo org/abs. 2306. 17563 arXiv:2306.1 17563.
15

---

Published as a conference paper at ICLR 2026
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Llama: Open Foundation Models for Code, January 2024. URL http://xxxv...gggbba/ss 3308 12950 arXiv:2308.122 12950.
Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E. Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, and Azalia Mirhoseini. Archon: An Architecture Search Framework for Inference- Time Techniques, October 2024. URL http:
//arxiv.org/abaa 2409. 15254 15X55
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning, October 2024. URL
arXiv:2410.08146.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634 8652, December 2023. URL
1b44b8 14444//8667788863533335555510002 Abstract Confoet-oooeeeeeccceooo html
Mohammed Latif Siddiq, Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. Using Large Language Models to Generate JUnit Tests: An Empirical Study, March 2024. URL http:ttt////ttt/
org/ abs//305 00418 arXiv:2305.00415
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling L.M Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, August 2024. URL htt.thttt////ii..oooo/
abs 2408 03314 #XXv:2408.03314 [es].
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non- non-Determmnism, July 2024. URL/attp://arxiv / abs/ 2407/2207..o2 2407210457 arXiv:2407.10457.
Amitayush Thakur, George Tsoukalas, Yeming Wen, Jimmy Xin, and Swarat Chaudhuri. An In-Context Learning Agent for Formal Theorem-Proving. August 2024. URL http::ttp//arr /bs/
2310. 04353
arXiv:2310.0435.
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit Test Case Generation with Transformers and Focal Context, May 2021. URL httt:://axxiv.oor
abs/2009.
arXiv:2009.05617.
Robert Vacareanu, Anurag Pratik, Evangelia Spiliopoulou, Zheng Qi, Giovanni Paolini, Neha Anna John, Jie Ma, Yassine Benajiba, and Miguel Ballesteros. General Purpose Verification for Chain of Thought Prompting, April 2024. URL htt.:htt:://// httt:::/aaxi...og////e///
2405. 00204 MxXiv:2455.0220 [cs].
Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue. and Hugh Zhang. Planning In Natural Language Improves LLM Search For Code Generation, September 2024a. arXiv:2409.03733 [es].
Haiming Wang, Huajian Xin, Chuanyang Zheng, Lin Li, Zhengying Liu, Qingxing Cao, Yinya Huang, Jing Xiong, Han Shi, Enze Xie, Jian Yin, Zhenguo Li, Heng Liao, and Xiaodan Liang. LEoO-rrover: Neural Theorem Proving with Growing Libraries, October 2023a. URL
abs/2310. 00656 arXiv:2310.00656.
Han Wang, Archik Prasad, Elias Stengel-Eskin, and Mohit Bansal. Soft Self-Consistency Im- proves Language Model Agents, June 2024b. HRL::tt:///aaa TRLLatpp//aaxxi..crr//////22221 13X11:2402.13212.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models, March 2023b. URI.httt:://ard org/ httgg/bb///20.// 11171 111.
16

---

Published as a conference paper at ICLR 2026
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le. and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 2023. URL http:://paax...oooo
abs 221//2211.111 arXiv:2201.11903.
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From Decoding to Mena-Genera:oon: Inference-time Algo- rithms for Large Language Models, June 2024. URL http:////aa.......r//////222---0 arXiv:2406. 16838.
Tianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Thinking LLMs: General Instruction Following with Thought Generation, October 2024. URL http://
arxiv. org/ abs /22/2241000 10630 arXiv:2410.10630.
Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. DeepSeck-Provets Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data, May 2024. URL http:://////xii.voo/
/bw.////..... 14333 arXiv:2405. 14333.
Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song. Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo: Theorem Proving with Searieval-Augmentee Language Models, October 2023. URL http::ttttt:iiiiee hwtp:/iwwwrgaab////aa///o 2306 15626 arXiv:2306.1566
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models, December 2023. /ttt: org/ ab// /205 10601 arXiv:2305.10601
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real- Real-World Domains, June 2024. URL http:://axiv.orr
12045 arXiv:2406.1 12044
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative Verifiers: Reward Modeling as Next-Token Prediction, August 2024a. URL http:
//arxiv. abs
http://..rrr/b////0//00/ 2408 15240 arXiv:2408.1522 15240 [cs].
Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Bate:a Examining Coding Performance Mismatch on HumanEval and Natural User Prompts, May 2024b. URL attp://aaxi..orr///a////222 04520 arXiv:2405.04522
Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. Beyond Correctness: Benchmarking Multi-dimenssonal Code Generation for Large Language Models, October 2024. / owt.a////ggaaaa/eaaoaoooo/s 2407.11470 arXiv:2407.11477
Li Zhong, Zilong Wang. and Jingbo Shang. Debug like Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step. June 2024. URL 2402. 1 16906 arXiv:2402.16906.
Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernic Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. agent-as-a-Judge: Evaluate Agents with Agents, October 2024.
URL. http:://p/...... / arxii org/ ab//a/bb/24111 10934. arXiv:2410. 10934.
17

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DETAILS ON SECTION
2
A. EDGE CASES IN OUR GENERATOR-VERITE SETTING
The setting described in Figure 2 considers verifiers and generators as distinct components, where verifiers score and accept or reject individual samples from the generator's s output to enable accuracy improvements through resampling.
This creates interesting edge cases with methods like Chain-of- Verification (CoVe) (Dhuliawala et al. 2023), Tree of Thoughts (ToT) (Yao al., 2023), and Reward Models (RMs) (Snell et al.) (022)) where verification and generation are more tightly coupled. While 'oT fits within our framework by producing aggregate scores and decision on whether the problem is solvable from given state or not (i.e potentially rejecting solutions), Co differs fundamentally in its verification approach. Instead of producing numeric scores and accepting or rejecting solution candidates, CoVe uses verification of intermediate facts used for answering question improve single response through iterative refinemett. This makes Co less suitable for inference scaling through resampling because there no way distinguish between the quality of multiple samples and using verifier's verdict for resampling.
18

---

Published as a conference paper at ICLR 2026
B ADDITIONAL DETAILS ON SECTION 3
GPT-3.5 Turbo
Llama 3.1 4058
Liama 3.1 8B
1.0
Fraction Tasks
Fraction of Tasks
Fraction Tasks
0.6
0.4
0.2
0.0
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.8
0.4
0.8
1.0
PasSnuy@1
Passeus
Passmus
Liama 3.1 70B
Mistral Large
Mistral Small
1.0
0.4
0.2
0.0
Passnuy@1
Passeru:01
Passpuu:01
Command
Command-Light
Command-A
1.0
0.4
0.2
0.0
Passn..@1
PasSmta@1
PasSytu @1
GPT-4o
1.0
0.8
0.4
0.2
0.0
Passk..@1
Figure 8: Distribution of task difficulties across models on MBPP+.
19

---

Published as a conference paper at ICLR 2026
CodeGen
Vicuna
CodeGen2
Fraction of Tasks
Fraction Tasks
Fraction of Tasks
0.5
0.6
0.4
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.5
Passmuu0
Passauu:1
PasSmuc
CodeT5+
StarCoder
0.8
0.6
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
P
PasssAu:911
Passeuc
Mistral
Command-R
1.0
0.8
0.6
0.0
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Passmu//1
PassAu@1
Pas5mu:
Command
Command-Light
1.0
0.8
0.6
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pa5s
Pass
Passmu:
3.1
Phi-3 Mini
Medium
1.0
0.8
0.6
0.4
0.0
0.4
1.0
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Passmuu@1
Passsuu@
Pa55ru
Liama
GPT-4o
1.0
0.8
0.6
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Passrc..01
PosSyt..11
Figure 9: Distribution of task difficulties across models on HumanEyal+.
20

---

Published as a conference paper at ICLR 2026
Liama 3.1 708
Llama 3.1 8B
Code Liama 7B
Code Liama 138
False Positive Rate
0.66
0.6
0.6
0.6
BA
RA
8.0
4
16-31
22.63
0.0
4
1-127
32.65
64-127
0.0
4
64-127
K Range (Powers of 2)
Range (Powers of 2)
K Range (Powers of 2)
Range (Powers of 2)
Command-Light
Command
Phi-3 Mini
Phi-3 Medium
0.8
0.5
0.4
0.4
0.4
0.2
0.2
16:32
6A-12
15
64:127
Range (Powers
Range (Powers
Range
Range (Powers
GPT-40
0.6
02
16-31
32.61
54-127
K Range (Powers 2)
Figure 10: False positive rates function of the number of attempts K in HumanEval+. K ranges are aggregated into bins of increasing size (i.e., powers of 2).
1.0
1.0
Passing HumanEval Unit Tests
Accuracy Conditional on
0.8
GPT-40's PPPT-40ssParsS3 Pars())
Lame 3,14558
0.8
Cannand-t
Mi-3 Mri.
"iiis Mali Doma 708
Tetn
0.6
Lodeten
Campand
Liama 3.1 m
0.6
Command
Uarra 3.1
GPT-40' Pass@]
Vicrss
Cade Unma 78
Misrra La
Ulmm
0.4
CodeT5g 148
0.4
Command
0.2
0.2
Ligt
0.52x 0.45, R² 0.893,
0.000
0.66x + 0,20, 0.891, 9 0.000
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Single-Sample Accuracy (Pass@1)
Samgle Accuracy (Pass@1)
HumanEval+
MBPP+
Figure 11: Relationship between the conditional accuracy after passing the standard unit tests and single-sample accuracy for tasks which the unit tests (i.e. the "verifier") have precision of less than 90%. For both benchmarks, we find a more pronounced relationship between capability and the probability of a false positive than when considering all tasks. Note that, the number of considered tasks with 70/150 and 128/321 is substantial. This plot shows the full y-axis.
B. DETAILS ON DATA AND IMPLEMENTATION
In the following, we provide more details on our analysis on HumanEval+ and MBPP+.
Sample Collection. To evaluate the generalization gap between weaker and stronger models, collected multiple samples per model and task. For both benchmarks and each model, we used samples generated with temperature setting of 0.8. For sample generation,
the implementation
21

---

Published as a conference paper at ICLR 2026
1.0
1.0
Passing HumanEval Unit Tests
Accuracy Conditional on
0.9
0.9
GPT-4o's Passg]
4050
CPT-35
Gemmenie
P113 Medium Liara 3.1 86
Lamn
Crrmandaa
Nista Small
3.1
0.8
Wcuna SiMiial 70 Cods uama 78
Semsand
Code Usma 130
0.8
Pacsg] GFT-40's GFTgg
Nitral Lorg
Lieme 3.1 4052
0.7
Communet ght
code 108
0.7
Command
Commend-Ught
0.6
0.6
0.22x 0.72. R 0.891 9.000
0.23% 0.87. 0,892. 0.000
0.5
0.5
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Single-Sample Accuracy (Pass@1)
Single-Sample Accuracy (Pass@1)
Hvaa
MBPP+
Figure 12: Worst-case upper and lower bounds for our HumanEval+ and MBPP+ analysis. We show the relationship between the accuracy of individual samples (x-axis) and the achievable accuracy given an infinite compute budget and limited unit tests (y-axis; note that it starts at 0.5). Error bars indicate the bounds on the conditional accuracy when we account for tasks for which we did not observe any passing solutions. For the upper (lower) bound, we set the conditional passing rate to 1 (0) when computing the accuracy estimates. Note that for Command Light, even after collecting 1000 samples for each task on HumanEval+, we still observed a substantial fraction of the tasks without a single passing solution.
provided by Liu al. (2023b), and other than the temperature use their default settings We collected a minimum of 50 samples for each model and task. For most models in our experiments Vicuna 7B, Mistral 7B, CodeT5p 16B, CodeGen, CodeGen2, Code Llama 7B, and Code Llama 13B, we used samples made available by Liu et al. (023b)) These were collected using the same temperature setting (i.e., 0.8). We had access to 200 samples per model and task for these models. Additionally, on HvaanEval+, we collected 200 samples for Llama 3.1, Phi-3, GPT-4o, and the Command family of models. For Command Light, we even collected 1000 samples for each task to reduce number of tasks without any solutions passing the HumanEval unit tests (Figure 12p. On MBPP+, we collected 50 samples for each model and task.
B.2 ADDITIONAL. DETAILS ON EXCLUDED TASKS FROM MBPP+
Tasks excluded by original EvalPlus authors (21 tasks). These exclusions are based update to MBPP+, during which the authors removed several broken tasks, reducing the total to 378 tasks. These tasks were excluded because of issues with the oracle implementation leading to unreliable evaluations
In addition the task excluded by the MBPP+ creators, in our evaluations, excluded a total of 57 tasks from the benchmark for two main reasons:
Tasks excluded due to additional oracle issues (28 tasks). We identified and excluded additional 28 tasks where all generated solutions that passed the base tests failed the extended test suite and across all models. We used this strict criterion positives that are in fact robust but fail the extended test suite due to bugs in the MBPP+ hamness. The primary cause was an implementation issue in the MBPP+ oracle when handling large numerical inputs. where the np allclose () function used for checking output equivalence would raise exceptions. After these exclusions, our final evaluation set consisted of 350 tasks from the MBPP+ benchmark.
ensure we would not count solutions false
Tasks excluded due to solutions passing plus tests but failing standard unit tests (29 tasks). An additional 29 tasks were excluded for which the extended unit tests yielded passing solutions that failed the base tests provided with the original benchmark. We intend to report these tasks to the
"Sec: 'See:
22

---

Published as a conference paper at ICLR 2026
benchmark creators, aiming to include these tasks in future version of the paper once the issue is resolved. Notably, excluding these tasks did not significantly impact our final results.
B.3 ADDITIONAL DETAILS ON EXCLUDED TASKS FROM HUMANEVAL+
Tasks excluded due to solutions passing plus tests but failing standard unit tests (14 tasks), Similar to MBPP+, we excluded 14 tasks from HumanEval+ from our analysis because passing solutions on the plus tests failed the standard unit tests. Including those tasks in the analysis did, as for MBPP+, not impact our results in any significant way. We plan to report these tasks to the benchmark creators.
23

---

Published as a conference paper at ICLR 2026
C ADDITIONAL DETAILS ON SECTION 4
In addition to the empirical analysis presented in Section in this appendix, we provide theoretical model that formalizes the limitations of inference scaling with imperfect verifiers and generalizes our findings to other benchmarks. We build on the verifier-based judge setup introduced by Davis et al. (2024). We provide a Python notebook with the implementation of our mode
0.5
0.8
0.4
Probability
Reward
0.3
0.6
Probability of Correct Solution
Probability of False Positive
Belief Task Easy
0.2
0.4
Beliet lask Hard
0.1
0.2
0.0
Cost -0.7 x TP Benefit
Cost -1 TP Benefit
Cost 2 Benefit
0.0
2
4
6
8
10
12
14
2
8 10 12 14
(Attempts)
K (Attempts)
Figure 13: Even with zero computational cost, the optimal number of samples is finite and very low (K < 3). For this plot, we set the parameters as empirically observed for Llama 3.1 8B on HumanEval (see Table for the exact values). The left plot shows expected value of generating additional candidate solutions ratios. For all cost-benefit ratios, the expected value peaks at very low K, after which it begins to decline, indicating negative returns from additional sampling. The right depicts the probabilities of generating a correct solution vs. a false positive at each step K. As K increases, the likelihood of generating correct solution decreases, while the probability of generating false positive increases. There is a trade-off between continued sampling and increasing risk, emphasizing the limitations of scaling inference compute with imperfect verifiers. Note that when setting the cost of false positive to be 10 times higher than the benefit of true positive, the optimal number of samples becomes K (Figure 14).
function of the number of attempts K for various cost-benefit
*See supplementary materials, file 1 imi iimmta i ing_moeel ipynb
24

---

Published as a conference paper at ICLR 2026
GPT-4o
3
600
Liama 3.1 405B
3
2
2
Llama 3.1 708
60
60
59
57
3
2
2
2
2
Phi-3 Medium
500
Phi-3 Mini
80
80
79
79
78
77
76
73
3
2
Llama 3.1 8B
80
79
76
Command-R
186
184
183
184
400
Code Llama 13B
11
11/
11.
116
116
115
115
11/
116
11.5
Code Llama
0
CodeTS+ 168
200
200
199
198
201
200
199
19/
195
195
0
Command
238
237
236
239
237
235
231
228
StarCoder
198
197
0
Mistral 7B
216
216
215
214
216
217
216
215
213
211
1
0
CodeGen
258
757
261
CodeGen2 7B
459
458
456
4.55
463
462
460
458
455
452
2
1
1100
Vicuna 7B
584
582
592
569
587
583
579
506
579
565
1
Command-Light
0.3
0.4
0.5
0.6
1.0
2
4 10
FP-Cost TP-Benefit Ratio
Figure 14: Heatmap of optimal number of samples for various false positive cost vs. true positive benefit ratios in model with parameters set observed HumanEval+. The y-axis shows models sorted by their Pass accuracy. We observe that for a relative cost of 10 (i.e., the cost of returning a false positive is 10 times more costly than the reward of returning true positive), the optimal number of samples is K for almost all models, effectively making them useless.
C.I MODEL SETUP
The underlying model consists of two components:
Produces candidate solutions to a task, with different success probabilities based on task difficulty.
Generator:
Tasks are either easy (Ti) or hard (T2), with prior probabilities p1 and respectively. The probabilities of generating a correct solution are for easy tasks and r2 for hard tasks, so r1 > n2.
Verifier: An imperfect verifier checks the correctness of generated solutions.
Completeness (c): Conditional probability of accepting a correct solution.
Soundness (s): Conditional probability of rejecting an incorrect solution.
PARAMETER VALUES USED MODEL UNDERLYING FIGURE
PROBABILITY OF REJECTION
The probability that sample is being rejected by the verifier, denoted B3, is given by:
3, = (1 - c)r: + s(1 -r))
where = 1 for easy tasks and = 2 for hard tasks. These probabilities (5₃ and (82) determine how likely a generated solution is be rejected depending on the task type.
BELIEF UPDATES
After each rejection, the belief that the task is of type T2 (hard) increases. The posterior probability that the task is of type Ti or T2 after k rejections is:
(8) PT.
(2)
gk-1
P1 + 135 P2
25

---

Published as a conference paper at ICLR 2026
Parameter
Value
Probability of correct solution (easy task), 71
0.87
Probability of correct solution (hard task), 72
0.13
Completeness, c
Soundness,
0.75
Prior probability of easy task, P1
0.58
Prior probability of hard task, p2
0.42
Benefit for correct solution (true positive), VTP
Cost for false positive, VFP
[-0.7, -1, -2]
Computational cost per attempt, Cs
Table 3: Parameter values used the model setup in Figure 13 as observed for the Llama 3.1 8B model evaluated Hvman These values reflect the empirically observed probabilities and prior settings. Following the observed empirical task difficulty distribution as shown in Figure 66] in this setup we assume tasks with Pass @ 0.5 be easy, and those with Pass 0.5 to be hard.
As more rejections occur, it usually becomes more likely that the task hard (72). In Figure 13 we how the belief that the task is easy decreases, while the belief that the task hard increases the number of attempts K grows.
PROBABILITY OF CORRECT AND FALSE POSITIVE SOLUTIONS
For the k-tth attempt, the probability of generating a correct solution or a false positive depends on the task type. The overall probabilities weighted by the posterior beliefs (k)
The belief-weighted probability of returning correct or false positive at attempt conditional on the k 1 previous attempts being rejected are:
P(N)
(k) Ph
PIP.T
(k) Prz Prr.I
(3)
PF Pr. Pre,T, + PPs
(R)
(8) Pre,T
(4)
where:
Pie,7 c.. Pie,72 C.
Pre,, r1) (1 s)s Pre., r2))
In Figure 13 the right plot shows the evolution of P( and Pf the number of attempts K increases. Initially, the probability of generating a correct solution is higher, but for higher K, the probability of generating a false positive increases.
EXPECTED VALUE OF GENERATING ADDITIONAL SOLUTIONS
The expected value of generating solution at the k-th attempt is:
EV2 = VIP P(P
VEP
[()]
[3
(k) Pr.
+ + PT.
gk-1 (k)
(5)
where:
Vip the benefit for a correct solution.
Vi is the cost for false positive being "accepted"" the solution.
26

---

Published as a conference paper at ICLR 2026
OPTIMAL NUMBER OF ATTEMPTS
The total expected value after K attempts is:
Reward
EV
(6)
The optimal number of attempts, Kopt, is the value of K that maximizes the reward, which shown across models and for various Vpp Vrr--ratos in Figure 14
C.2 INFERENCE SCALING CURVE FOR GPT-40
GPT-4o
0.8
Reward
0.6
0.4
Ratio
Ratio
Ratio
Conditional Pass@1
0.0
100
10a K (Attempts)
10a
Figure 15: Inference scaling curves in the presence of cost for GPT-4o. In addition to the models Figure ! we provide the inference scaling curves for GPT-4o as the model with the highest single-sample accuracy
HymanEyal+ in experiments. We find that the benefits of search minimal (i.e. curves are flat) in line with what we expect from the empirical task difficulty distribution shown in Figure g
27

---

Published as a conference paper at ICLR 2026
D ADDITIONAL DETAILS ON SECTION 5
camelCase
snake_case case
1.0
100
0.00
090339
0ze
0.8
0.00
0A00.0
0.8
Probabiiit
0.2
0.2
Aichevere Gree Qualiy/ vanursnn Caraec)
Ahemerect Gaei buaau icturrann y pul Portwal
0.0
GPT-4o
GFF-3 Turbo
Uama 3.1 708
Mistral Large
Mistrel Small
0.0
GPT-00
GPF-3 Turbo
Uimaa 3.1 708l
Liama 3. 88
Mistral Large
Mistral Small
Uteaa 1.
Line Length
Commenting
1.0
1.0
0.8
0.778777766
0200
8.00
3.5
046-180
0.6
016311
0.4
0.2
0.2
0.0
0.0
GPT 40
GFF-3,8 Tarbo
Liame 3.1 708
Ulama 3.1 98
Misral Large
Mitral Small
GPT-40
GPF-3 5 Rurbo
Liama 3.1 708
Llama 3. 38
Misrral Large
Mitral Snail
Figure 16: False positives tend to be lower-quality code compared to correct solutions across all models and code quality metrics. We evaluated four key code quality metries: adherence to camelCase and snake _naee_case case naming conventions, line-length compliance, and presence of line- level comments. This trend holds consistently across models and for all four code quality instructions we test.
D.1 DETAILS ON DATA AND IMPLEMENTATION
We used the implementation provided by Zheng et al. (024) to collect samples and evaluate the different code readability metrics. Each code quality metric had a separate prompt instructing the model to follow certain guidelines (Section[D.2) For each model and code quality instruction, we generated 50 samples per task on HumanEval+. As for our main experiments, we set the temperature to 0.8. All other parameters were set their default value as provided with the implementation! [
SSee: http://hwhh....o.o/e//////////ee 7ct.99999886777779999979 9c5e
28

---

Published as a conference paper at ICLR 2026
D.2 PROMPT EXAMPLES FOR READABILITY METRICS
Naming conventions
Please generate the Python code to solve the following problem, and use came Cass for both function names and variable names.AnnmPPnn Jumees..inPrrbblemmlinnu (prroollemmmnullppoleeem [roblemm
Please generate the Python code to solve the following problem, and use snake_cas both function names and variable names. ln problemm
Code length
Please generate the Python code to solve following problem, where each line is less than 70 characters long and each function is less than 30 lines long. pongg//mnrrbblmmmnnnnprooooo
Commenting guidelines
Please generate the Python code solve the following problem, and add comments for each line each function. (arrrroolemmmn/ppppooooioo [problem)
Figure 17: Prompt templates for each metric of code readability we consider in our experiments for Section 5 following Zheng et al. (2024).
D.3 QUALITATIVE EXAMPLES OF FALSE POSITIVES
def get_positive (1: list):
return [int (item) for item in if item 0]
a) Example implementation generated by CodeGen-6B which fails additional unit tests from Hu- manEval+.
Standard unit tests
assert get_positive [5, 3, -5, 2, -3, 3, 9, 0, 123, 1, -101) [5, 3, 2. 3, 9, 123, 11
assert get_positive 1-1, 2, -4, 5, 61) == 12, 5, 61
Example unit test from extended test suite assert get vet_positivve(D) 5, 0, -4, 2.5, 5, -2.2, 8, 7.7, 9.9, -10.5]) [0.5, 2.5, 5, 7.7, 9.91
b) Standard unit tests for the HumanEval/30 task and one example test from the extended test suite of HumanEvalt.
Figure 18: Example implementation that passes the standard tests but is incorrect. In HumanEyal/30, the model is asked to Return only positive numbers the list. The code was generated by CodeGin- 6B (Nijkamp et al.) 2023) and passes the tests provided with the original HvmanEval benchmark but fails the extended tests. The implementation contains a logical error due to unnecessarily converting positive floats integers.
29

---

Published as a conference paper at ICLR 2026
def sequence (n: int) -> str:
2. ans n"
3
for i in range (0, n 1): ans ans + str (1) return [:-1]
Figure 19: HumanEval/15 asks to generate space-delimited sequence of numbers. The implementa- tion generated by CodeGen fails on the extended test suite due the inefficient for-loop that times out on larger inputs.
def sort_numbers (numbers: str) str: str_num_dics "zero": C, "one": 1, "ewo": 2, "three": 3, "four": 4, "five": 5, "six": 6, "seven": 7, "eight": 8, "nine": 9,
num_list: [ist[int] dstr_nun_dict [word] for word in numbers. split
7
nun_list.sort
sorted_str_nume dist[str) = word for word, num in Sar_num_dict dim..itmmm ittmm if num in mist
11
return ".join
Figure 20: HumanEval/19 asks sort space-delimited number words. The implementation generated by Code Llama 7B fails to maintain duplicates in the output, incorrectly removing repeated numbers during the dictionary lookup process.
def find Band__har__il whar_llogaaaaaaa (string ): words = string. split () return [word for word in words if 1en (word 41
Standard unit tests
assert set (find_char_lond ('Ploase move back to stream/) )) == sct (I' Please', "ove'' back', x "stream']] assert set (find_cchhr__oo
Jing Eco and Tech ()) set (['Jing', 'Tech']) assert set (find_char_ionn ( Jhingai wulu road
Zone 3')) == set (I' Jhingai', 'wulu', road', 'Zone'
Example implementation generated by GPT-3.5 which fails additional unit tests from MBPP+
6 Example unit tests from extended test suite
assert set (find_char__il
("This is very long string with words that least characters leng.")) set (I' This', 'very', 'long', 'string', 'with', 'words', 'that', 'least', chhracteers', 'Long'])
Standard unit tests for the MBPP/7 task.
Figure Exemplary implementation that passes the standard tests but incorrect. MBPP/7, the model is asked to write function to find all words that are at least 4 characters long in string. The code was generated by gpt-3.5 and passes the tests provided with the original MBPP benchmark but incorrect and fails to pass the extended tests.
30

---

Published as a conference paper at ICLR 2026
def parse nested pazens (paren string: str) -> List lint]:
def sarrs__paee_grroen (paren_group: str) -> int:
counter 0
max_counter
for paren in group:
paren (
counter +=
if counter max_counter: max_counter counter
elif paren :
11
counter -=
12
return max_counter
return [parse_paren_gree (paren_group) for paren_group in caren_string.splic, {)]
Figure 22: HumanEval/6 requires parsing nested parentheses groups. The implementation generated by Code Llama 7B incorrectly processes empty input strings.
def sortea_list_su([st]] assert 1st, "List can not empty.
if (1st str): new_lst [word for word in 1st if len (word) 8 = 0] new_lst.sort/ return sorted (new 1st, key=len)
else:
return sorted (Lst)
Figure 23: HumanEval/149 asks to process a of strings, removing odd-length strings and sorting the result. The implementation generated by CodeTTp 16B incorrectly handles empty input by raising an assertion error instead of returning an empty list.
def countlumberrsiitthnneee.))
if n == 1: # For single-digit numbers, only have return
count = 0
Calculate the range for n-digit numbers start 10*x (n-1) 10=x(n=1)
enc 10**n
11
for num in range (start, end) str_hum str (num)
if str_num [0] '1' str_num[-1] (1f
15
count 1
16
return count
Figure 24:
Example candidate implementation failing to adhere to camelCase naming convention for HumanEval/83 generated by GPT-4o due to variable str num in line 13 not complying.
31

---

Published as a conference paper at ICLR 2026
def get positive (1: :
return [int (item) for item in 1 if item > 0]
Figure 25: Hvl33Eval/30 asks to return positive numbers from list. The implementation generated
by CodeGen fails on the extended test suite because it converts float values to integers before returning the list.
def Lumber_of_subsri11
[0] 128
3
result
for char in s:
[har_count [char)] +=
for in range (1))
result += char_count (s [i])] (len
return result
Figure 26: MBPP/238 asks to count non-empty substrings in string. The implementation generated by Command uses an incorrect algorithm that produces wrong results.
def split : return [char for char in string if char. Ssaarhs
()
Figure 27: MBPP/565 asks to split string into characters. The implementation generated by Command Light incorrectly filters for alphabetic characters only and fails to handle empty strings correctly,
def is_ ss_ssmpppateesss (words, patterns):
if len (words) I= len (patterns):
return False
word_to_pattern (}
battern_to_xord {
for word, pattern in zip (words, patterns):
word word_to_patterm and word_to_pactten [word] !-
pattern:
8
return False
if pattern in pattern_tod to word and pattern to word
!- word:
return False
11
word_to_pattess [word] = pattern
12
patterr_to pattern_to_xxxdppattee (patter))) = word
return True
Figure 28: MBPP/74 asks to check if words follow given patterns. The implementation generated by
GPT-3.5 fails the extended test suite when handling repeated elements, incorrectly returning False.
32

---

Published as a conference paper at ICLR 2026
def find even ever_paiir (nums)
even_count
xor_pairs - set
for i in range (len (numa)):
for in range (i+l, 1en (nums) )
(nums [i] nums (i])
0:
pair tuple (sorted ( (nums [i], nums ) xor_pairs. (pair)
return 1en (xor (xor_pairs)
Figure 29: MBPP/166 asks to count pairs of integers with even XOR. The implementation generated by Llama 3.1 8B incorrectly counts only unique pairs using set.
def diff _are___e______ even_odd (numbers)
None
3
4
for num in numbers:
if num $ 2 == 0 and even is Nonc:
ever num
elif num odd num
== and odd is None:
8
if ever is not None and add is not None:
break return abs (even odd)
Figure 30: MBPP/594 asks to write a function to find the difference between the first even and the first odd number of given list. The implementation generated by GPT-3.5 incorrectly returns the absolute value of the difference.
E DECLARATION OF LLM USAGE AND COMPUTE RESOURCES
In this work, LLMs are the main subject of study and used for sampling solutions for coding benchmarks. For all our experiments using OpenAl models, we utilized the endpoints provided by OpenAI, either directly or through the Azure OpenAl Service. For the analysis on HotPotQA using Llama models, we relied on the endpoints provided by Together.ai. As our work primarily relied on external APIs, we did not use any GPUs for inference and our experiments did not require training of LLMs.
F IMPACT STATEMENT
Our work aims to enhance the technical understanding of the limitations of inference scaling methods. In particular, contribute and how this approach can fail to lift the accuracy of weaker models to match the performance of stronger models. While these findings have implications for the training of reasoning models and the deployment of compound AI pystems--icluding code generation societal implications mirror those already familiar the development and of large-scale language models. We do not identify any additional, domain-specite concerns that arise uniquely from our study. Instead, our results reinforce the importance of reliable evaluation metrics and thorough verification methods, which in turn support safer and more trustworthy applications in coding as well as in other areas where compound Al systems are increasingly adopted.
findings resampling in combination with imperfect verifiers
broader ethical and
33


Reviews:
[{"review_id": "2ZxuTHxgzK", "reviewer": "Reviewer_Tash", "summary": "This paper presents a critical analysis of inference-time scaling, specifically the \"resampling\" approach where solutions are generated until one passes a verifier. The authors argue that this technique is **fundamentally limited** when the verifier is imperfect (i.e., has a non-zero false positive rate).\n\nThe core contributions are:\n1.  An empirical demonstration on coding benchmarks (HumanEval+ and MBPP+) that a model's single-sample accuracy (Pass@1) is **strongly correlated with its false positive rate**. Weaker models produce *more* false positives (solutions that pass limited unit tests but fail comprehensive ones) than stronger models.\n2.  This correlation establishes a **hard upper bound** on the performance of resampling. A weaker model, even with an infinite compute budget, cannot match the single-sample accuracy of a stronger model if the stronger model's accuracy is already higher than the weaker model's *conditional* accuracy (its accuracy on solutions that pass the imperfect verifier).\n3.  By introducing a \"cost\" for false positives, the paper shows that the optimal number of sampling attempts (K) is often **finite and very low** (e.g., fewer than 10), as the expected reward curve bends downward when the risk of a false positive outweighs the benefit of finding a true positive.\n4.  A secondary finding that these false positive solutions are not just functionally incorrect but are also of **lower code quality** (e.g., poor adherence to style conventions).", "questions": "1.  **Scaling Verifiers:** Your analysis relies on a static, human-written verifier. How do you believe your findings would change in a setting where the verifier also scales? For example, what if you used GPT-4o as an \"LM-as-judge\" to verify the solutions from all models, or used each model to generate its *own* unit tests?\n2.  **Domain-Agnostic Claim:** You claim the results are \"domain-agnostic\", yet your own Table 2 shows that \"oracle\" verifiers are common in mathematics. Can you clarify your claim? Does the \"limit\" you've identified primarily apply to domains *lacking* formal, oracle verifiers (like coding with incomplete tests, or creative writing)?\n3.  **Cost-Benefit Ratio:** Your \"optimal K\" analysis is very sensitive to the C/B ratio. Do you have any suggestions for how one might empirically estimate a \"realistic\" C/B ratio for a typical software engineering task, to move this finding from a theoretical observation to a practical prescription?\n4.  **Task Exclusion:** You excluded 78 tasks from MBPP+ due to test harness issues. Can you provide more detail on why you are confident this did not bias the results? For instance, did you check if the Pass@1 or conditional accuracy *on this subset of tasks* showed a different trend?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Novel, Important, and Non-Obvious Finding:** The paper's main strength is the empirical discovery of the strong linear correlation between a model's Pass@1 accuracy and its false positive rate. This is not an obvious or trivial result; one might have assumed the \"imperfect verifier\" was an independent challenge. This finding is the engine for all the paper's other conclusions.\n* **Elegant and Sound Methodology:** As mentioned in \"Soundness,\" the use of the `Benchmark` vs. `Benchmark+` test suites is a perfect experimental setup to test the hypothesis.\n* **Strong, Clear-Cut Results:** The data presented is not ambiguous. The linear relationship is strong ($R^2 \\approx 0.89$), and the performance ceiling is clearly visible. The downward-bending reward curves are also unambiguous.\n* **Practicality of the Critique:** The paper addresses a real-world, practical strategy (resampling) and exposes its fundamental limitations in a way that is immediately useful for both researchers and practitioners.\n* **Secondary Quality Analysis:** The analysis in Section 5, showing that false positives are *also* lower-quality code (e.g., worse style), is a strong supporting argument that adds another dimension to the critique.", "weaknesses": "* **Static, Domain-Specific Verifier:** The paper's *entire* analysis hinges on a **static, human-written verifier** (the original HumanEval/MBPP unit tests) in the single domain of **coding**. The central claim that resampling is \"fundamentally limited\" is very strong, but the evidence is scoped to this specific setup. The paper's claim of being \"domain-agnostic\" is not well-supported, especially since Table 2 explicitly lists \"Math\" as a domain where \"Oracle\" verifiers (like proof checkers) *do* exist, which would make resampling a perfectly valid strategy in that domain.\n* **Doesn't Account for Scaling Verifiers:** The analysis assumes the *generator* scales (from Llama 7B to GPT-4o) but the *verifier* is fixed. In many modern systems, the verifier is *also* an LLM (e.g., \"LM-as-judge\") or is *generated* by the model itself (e.g., model-generated unit tests). A stronger model is likely also a stronger verifier. The paper's conclusions might not hold in a \"co-scaling\" scenario where the verifier's capability (and thus its false positive rate) improves along with the generator's. The paper mentions this but does not investigate it, which is the most significant limitation of this work.\n* **Ambiguity of \"Cost\":** The analysis of the optimal K in Section 4 is highly dependent on the \"cost-benefit ratio,\" a variable for which the paper provides no empirical grounding. While it's intuitive that a bug (false positive) has a high cost, showing a range from 0 to 8 makes the finding abstract. Without a \"realistic\" C/B ratio, it's difficult to conclude whether the optimal K for a real-world application is 3 or 300.\n* **Significant Task Exclusion:** The appendix reveals that a large number of tasks were excluded from the analysis (e.g., 78 from MBPP+ and 14 from HumanEval+) due to issues with the test harness. The paper asserts this \"did not significantly impact our final results\", but this requires the reviewer to trust this claim. It's plausible that these \"broken\" or ambiguous tasks are precisely the ones that would most effectively probe the limitations of weaker models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8apuvz9NHj", "reviewer": "Reviewer_9qjn", "summary": "This paper explores the limitations of scaling resampling in two coding benchmarks and shows that weaker models are more prone to generating false positives—solutions that pass some unit tests but fail when subjected to more comprehensive testing. The study reveals that, in the coding domain, the effectiveness of resampling is fundamentally constrained by the insufficiency of unit tests, a challenge frequently encountered in real-world applications", "questions": "See the weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The main finding is compelling: Limited test cases constrain the scalability of resampling, and weaker models experience a larger generalization gap than stronger ones. Notably, most weak models cannot outperform the single-sample accuracy of stronger models even when resampling is scaled up. These results highlight the inherent limitations of resampling in real-world coding scenarios, where test cases are typically limited.\n\nThe analyses are insightful: By introducing the concept of the cost of false positives and evaluating the quality of generated code, the authors illuminate the potential risks associated with applying resampling techniques in practical coding applications.\n\nThe experiments are comprehensive, spanning various model families and two benchmarks.", "weaknesses": "- The use of a large table (Table 1) to list numerous inference scaling techniques feels excessive, particularly since this paper only analyzes the limitations of one specific technique. Including such an extensive table for all techniques seems unnecessary and distracts from the actual focus of the study. A similar concern applies to Table 2.\n\n- The authors attempt to extend their findings to broader domains; however, the paper’s analysis is limited to the coding domain. While test cases serve as verifiers in coding, other domains (e.g., mathematics) might utilize reward models, which differ fundamentally. Although both are imperfect verifiers, the distinctions between test cases and reward models are substantial, making broad generalization of the findings from \ncoding to other domains unconvincing\n\nInsufficient justification for main finding:  \n\n- (a) The central claim that “the weaker model cannot match the performance of a single invocation of the stronger model” is presented too absolutely. For instance, Figure 3 shows cases where Llama 3.1 70B outperforms a single invocation of GPT-4o, and Llama 3.1 8B outperforms a single invocation of Llama 3.1 70B, contradicting the claim.\n- (b) In line 246, the authors state they generate at least 50 samples for each model and benchmark. The rationale for the “at least” threshold is unclear. Why is it not standardized? How are the actual sampling counts determined?\n- (c) In lines 261–263, the statement that the finding holds “no matter how big the compute budget for the weaker model” is insufficiently supported. Only experiments with “at least 50 samples” are reported, leaving the maximum compute budget used ambiguous and raising the question of whether a larger compute budget could yield different results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d31dc8zDJo", "reviewer": "Reviewer_JXaJ", "summary": "This paper argues that inference-time scaling via repeated sampling + imperfect verifiers (like unit tests) has a hard ceiling. Even if you spend infinite inference compute, weaker models cannot necessarily catch up to stronger models, because false positives (incorrect-but-accepted answers) become the bottleneck. The paper backs this up with theory and experiments on HumanEval+/MBPP+, and also studies utility tradeoffs when false positives are costly, plus code quality effects.", "questions": "Question and suggestion are listed in weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper challenges a widely held assumption (“a weak model can just sample more and pass tests to match a stronger model”) and shows that imperfect verifiers are actually the fundamental bottleneck. This is framed cleanly and feels very relevant right now.\n\n2. Using benchmarks like HumanEval+ and MBPP+, the paper quantifies how weaker models produce many false positives (wrong answers that still pass tests) and shows that even with unlimited resampling they hit a ceiling and still can’t match stronger models.\n\n3. The paper doesn’t stop at benchmark accuracy — it talks about real-world cost of shipping bad code, the risk of contaminating training data with verifier-approved mistakes, and why “just sample more” is not a reliable path to replacing stronger models in production.\n\n4. The figures and tables (e.g. showing the accuracy ceiling line, and how the optimal number of attempts K is actually low when false positives are costly) make the main ideas easy to understand and memorable\n\n5. The experimental ideation in Section 4 is novel. Although the assumptions regarding the cost are simplified, this is inevitable. The experiment provides a practical and insightful analysis of the cost associated with infinite or large-scale sampling.", "weaknesses": "Overall, I appreciate the quality of the paper, though I have several points of concern outlined below.\n\n1. Because the verifier’s effectiveness depends on the specific test suite, its accuracy and false-positive behavior may vary across different sets of test cases. The paper would benefit from an analysis of how the results change when the verifier’s quality or coverage is systematically varied.\n\n2. A considerable amount of the introductory section (up to approximately page 4) is spent discussing scaling methods beyond the imperfect verifier. It is somewhat questionable whether this level of detail is essential, given that the paper’s primary scope focuses on tasks involving imperfect verifiers in the coding domain. Such elaborations might be better suited for the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5jXLkPnx2t", "reviewer": "Reviewer_Tt4Y", "summary": "The paper examines inference-time “resampling with verifiers” and shows that when verifiers are imperfect (i.e., admit false positives), weaker models cannot sample their way up to a strong model’s single-sample accuracy. On HumanEval+ and MBPP+, many solutions from weaker models that pass standard unit tests fail extended tests, so even with unlimited samples, they remain below a strong model's Pass@1. Incorporating the cost of false positives further implies that the optimal sample count is small. The authors also show that false-positive solutions are lower-quality code beyond correctness (naming, comments, length).", "questions": "1. If the verifier can abstain at a tunable threshold to bound FPR$\\leq \\varepsilon$, how does the ceiling shift? Can you train a “selective verifier” with coverage guarantees and show a different optimal K frontier?\n\n2. The cost of a false positive is treated uniformly in the paper. In practice, costs vary by task and downstream impact. Is it possible to learn task-conditioned costs (e.g., via proxy severity scores) and optimize K per instance?\n\n3. How does sampling configuration (e.g., top-p/temperature) affect the conclusions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The central message is clearly motivated and explained: imperfect verifiers induce an upper bound on performance gains from resampling, and weaker models cannot match stronger models’ Pass@1 via increased sampling.\n\n2. The extensive experiment demonstrates a few interesting observations: 1) the ceiling effect persists across model families and sampling budgets, 2) the optimal sample count is typically small when false positives incur penalties, and 3) false positive solutions exhibit systematically poorer code properties (naming, comments, verbosity).", "weaknesses": "1. The scope of the paper is coding-centric. The arguments are persuasive for unit-test verifiers, but other imperfect verifiers, such as LM-judges for QA/reasoning, could also be studied. \n\n2. Resampling changes the distribution of candidate solutions; the verifier’s FPR may not be stationary with K (e.g., later samples are weirder/hackier). Do you observe drift in the verifier errors as K grows?\n\n3. A single verifier (tested in the paper) would create a brittle FPR. What is the ceiling under, for instance, majority ensembles of various verifiers? In that case, how does diversity (e.g., correlation of errors) mathematically translate into ceiling relief?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)