Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
Authors: Eric Zhao
Pranjal Awasthi
Sreenivas Gollapudi
Venue: ICML 2025 poster (2025)
Keywords: reasoning
search
verification
Primary area: deep_learning->large_language_models
TL;DR: We study the scaling trends governing sampling-based search, a test-time compute scaling paradigm.

Abstract: Sampling-based search, a simple paradigm for utilizing test-time compute, involves generating multiple candidate responses and selecting the best one---typically by verifying each response for correctness. In this paper, we study the scaling trends governing sampling-based search. Among our findings is that simply scaling up a minimalist implementation that uses only random sampling and direct self-verification results in sustained performance improvements that, for example, elevate the Gemini v1.5 Pro model's reasoning capabilities past that of o1-Preview on popular benchmarks. We partially attribute the scalability of sampling-based search to a phenomenon of implicit scaling, where sampling a larger pool of responses in turn improves verification accuracy. We further identify two useful principles for improving self-verification capabilities with test-time compute: (1) comparing across responses provides helpful signals about the locations of errors and hallucinations, and (2) different model output styles are useful for different contexts---chains of thought are useful for reasoning but harder to verify. We also find that, though accurate verification can be elicited,  frontier models demonstrate remarkably weak out-of-box verification capabilities and introduce a benchmark to measure progress on these deficiencies.

Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
Eric Zhao 12
Pranjal Awasthi
Sreenivas Gollapudi
Abstract
Wei et al., 2022; Yao al., 2023; Akyürek et al., 2024),
There are many methods for increasing test-time compute usage, including implicitly encouraging longer responses via reinforcement learning (OpenAI, 2024; Team, 2025) or explicitly via prompting (Wei et al., 2022; Yao et al., 2023). However, sampling- -aaspiig-based search an instance of the generate-and-tes approach where model generates many responses in parallel, e.g. via random sampling or delegation, and selects what the model guesses to be the best one-remains one of the most natural and fundamental paradigms. addition to being complementary with other test-time compute scaling strategies, it also has the unique advantage of being embarrassingly parallel and allowing for arbitrarily scaling: simply sample more responses (Cobbe et al., 2021; Wang et al., 2023).
Sampling-based search, simple paradigm for utiliizig test-time compute, involves generating multiple candidate responses and selecting the best one-typically by having models self-verify each response for correctness. In this paper, we study the scaling trends governing sampling- based search. Among our findings is that sim- ply scaling up a minimalist implementation of sampling- search, using only random sam- pling and direct self-verification, provides a prac- tical inference method that, for example, elevates the reasoning capabilities of Gemini v1.5 Pro above that of ol-Preview on popular benchmarks. We partially attribute the scalability of sampling- based search to a phenomenon of implicit scaling, where sampling larger pool of responses turn improves self-verification accuracy. We further identify two useful principles for improving self- verification capabilities with test-time compute: (1) comparing across responses provides helpful signals about the locations of errors and halluci- nations, and (2) different model output styles are useful for different contexts- chains of thought are useful for reasoning but harder to verify. We also find that, though accurate verification can be elicited, frontier models demonstrate remark- ably weak out-of-box verification capabilities and introduce a benchmark to measure progress on these deficiencies.
Though recent works demonstrate the benefits of sampling- based search (Cobbe et al., 2021; Wang et al., 2023; Xue et al., 2023), many questions remain as to what scaling trends govern this fundamental test-time compute scaling strategy. In particular, the practical utility of sampling- search is limited by verification. Whereas prior work have shown that parallel scaling of test-time compute via Pass can be remarkably effective (Brown et al., 2024), sampling remains fundamentally bottlenecked by the availability of external verification or the model's 's ability to verify. We are interested in understanding how this bottleneck evolves with scale and how much of the Pass@k Pass I gap can actually be attained in practice.
To develop this understanding, we study minimalist- yet remarkably effective instantiattonn of sampling- based search that uses a language model (Gemini Team, 2024) to both generate a set of candidate responses via random sam- pling and select the best one by self-verifying each response with natural language. Specifically, we consider the case where models must self-verify their responses select the best answer, and do not make the strong assumption that
1. Introduction
Recent advances in language models highlight the impor- tance of test-time compute scaling wherein one uses more compute during inference enhance reasoning capabil- ities (OpenAI, 2024; Team, 2025; Agarwal et al., 2024;
"The al 2r2view-2002-09-12 numbers in Tahle figures, with MATH and AIME figures sourced from the OpenAl blog post (OpenAI, 2024), and LiveBench figures sourced from the LiveBench leaderboard (livebench.ai) We found ol -Prreview performance through the OpenAI API to slightly differ with pub- licly reported figures, e.g. scoring 26% not 44% on AIME, and scoring 77% not 67% on LiveBench Reasoning.
public
'Google Research PUC Berkeley. Correspondence to: Eric Zhao <eric.zh@ @ berkeloy.edu>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sample, Serutinize and Scale: Effective Inference- Search by Scaling Verification
Method
AIME
MATH
LiveBench Math
LiveBench Reasoning
Pass@1 1
1/15
426 / 500
104 200
63 140
Consistency @ 200
4/ 15
460 / 500
118 200
75 140
Consistency @ 1,000 3/ 15
460 500
120 / 200
73 / 140
Verification @ 200 8/ 15
467 / 500
135 200
97 / 140
ol-Preview @!
7/15
428 / 500
131 200
95/ 140
Table Accuracy rates the Gemini v1.5 Pro model using campling-based search (Verification 200) on reasoning benchmarks. compared to other inference methods and o1-Preview performance. Verification @ 200 consistently improves on Consistency @ 200 and surpasses or-erevie.." 2 Each score reflects a single run, due to the high expense search at this scale (see Section 4).
one can access ground-truth answers or symbolic systems that exactly verify correctness. In this setup. we address the question: what test-time scaling trends emerge as we scale both the number of sampled responses and verification capabilities? In particular, what the limits of scaling this simple sampling- based search paradigm and how much does one need to continuously scale verification capability as one scales up search?
didate responses provide a strong signal for where errors might be located. The latter leverages our observation of output style suitability where chain-of-thought output for- mats are beneficial when generating responses but harder to verify than more formal, mathematically conventional writ- ing styles. Surprisingly, while effective verification can be easily elicited from frontier models by communicating these strategies, we observe that frontier models have remarkably poor out-of-box verification capabilities and introduce a new benchmark to quantify these deficits.
Our findings. We first identify scaling trends demon- strating that reasoning performance continues to improve with sampling-baseer search even as test-time compute is scaled well beyond the point where the performance of self-consistency (Wang et al., 2023) saturates. At sufficient scale, even our minimalist implementation provides a sig- nificant leap in reasoning accuracy, lifting Gemini v1.5 Pro performance beyond ol -r-Previ,w, and Gemini v1.5 Flash beyond Gemini v1.5 Pro, on reasoning benchmarks such as LiveBench (White et al., 2024) and the AIME (MAA, 2024), exhibiting sustained power-law scaling on the latter. This not only highlights the importance of sampling-based search for scaling capability, but also suggests the utility of sampling- based search simple baseline which to compare other rest-time compute scaling strategies and mea- sure genuine improvements in models' search capabilities.
Historically, self-verification has been believed to be an unreliable means of closing the Pass@k Pass @ I gap, mo- tivating the use of process-based reward models (Lightman et al., 2024), the reinforcement learning of verifiers (Snell et al., 2024), and other complicated interventions. Our re- sults indicate that self-verification can actually be made quite reliable at scale and in fact is sufficient to extract ol-level performance from non-reasoning models without finetuning, distillation, RL, or custom-traines verifiers.
Preview and outline. Table summarizes our first find- ing: that, with effective self-verification, simply scaling sampling- based search is sufficient to approach state-of-art performance on reasoning and math benchmarks (AIME 2024 (MAA, 2024), LiveBench Math, LiveBench Reason- ing (White et al., 2024), and the Berkeley MATH dataset (Hendrycks et al., 2021)). It depicts the accuracy of the Gemini v1.5 Pro model (Gemini Team, 2024) when only one solution is attempted per question (Pass@) 1), when 200 solutions attempted and the most common final answer is selected (Consistency @ 200, (Wang et al., 2023)), and under sampling based search, when 200 solutions are at- tempted and scored for correctness with the highest scorer selected (Verification @ 200, Algorithm 1). With sampling- based search (Verification @ 200), Gemini v1.5 surpasses the performance of ol ol-Prree a model explicitly trained on reasoning problems to leverage significant test-time com- pute and perform internal search.
We then attribute much of the strong scaling trends of sampling-basee based search to an implicit scaling phenomenon. Contrary to the intuition that sampling more responses should impose a greater burden on the verifier and reduce verification accuracy, we observe that scaling sampling indi- rectly enhances verification accuracy. At a high-level, this is because well-written responses are easier to verify than poorly written responses, and scaling sampling widens the pool of well-written candidates.
We further identify two effective strategies for scaling verifi- cation capabilities using test-time compute: (1) directly com- paring candidate responses and (2) task-specific rewriting of candidate responses. The former mitigates a core weakness of language models, which struggle to identify mistakes and hallucinations unless given their locations (Tyen et al., 2024), by leveraging the fact that differences between can-
As focus on answering reasoning problems, use "model responses" and "model solutions" interchangeably.
2

---

Sample, Serutinize and Scale: Effective Inference- Search by Scaling Verification
The rest of this paper is devoted to studying the three key factors behind the numbers in Table 1. Section 2.1 analyzes the scalability of sampling-based search, as one varies the compute spent on search and verification; Section 2.2 an- alyzes the phenomenon of implicit scaling that drives this scalability; and Section 3 discusses key principles for scal- ing self- rerficeriiiotionn capability. We also highlight deficits in the verification capabilities of frontier models with a new benchmark in Section 5. Further discussion of related work is found in Appendix A.
In addition clear burn-in costs along both axes of scale, we can observe that the largest performance gains real- ized when search and verification are both scaled. These trends also indicate that the performances of sampling-bases search, as reported in Table 1, have not yet been scaled to saturation on these benchmarks. This scaling trend is strongest on the AIME benchmark, where performance bottlenecked by (search); we attribute this bottleneck to the difficulty of the AIME questions resulting in correct solu- tions only appearing with very low probability (see Table 2). In particular, we note that of the 7/15 AIME questions where exactly 1-4% of solutions are correct, Verification @ 200 suc- cessfully extracts a correct solution for 5/7 questions.
Algorithm 1 Sampling Based Search (Verification @ kinf)
Input: Prompt Q, model LM, params kinf, Kverif, kaii-
Populate& S with kint samples from LM(" LM("Answer Q"). for each candidate response 8, E S do
2.2. Implicit Scaling
Let V; be kverit samples of LM(1 8; conrect?).
Scaling sampling-basd -baned search along the search axis by sampling more solutions, i.e. increasing k, should have two effects on performance that partially cancel out: (1) the ver- ifier must discriminate between more solutions, increasing the likelihood of error and (2) the generator is more likely to produce at least one solution that reaches correct final answer, i.e. Pass @k increases.
end for Gather the highest-scored response
SBest = { {ss Avg(Vi)
>
max Avg(V;) 0.05 je [End
Return SBest if [Steet] = 1. for each (8i, 8;) in (Steat) do Let Ci.j be kitie samples of LM("Is 8; or 8j correct"). end for Return round-robin winner 8,- of [C,, 18; 8y E SBest)
To isolate the first effect, we study the model's Verifica- tion@k accuracy on "ambiguous" questions: questions where at least one of the model's k candidate solutions reaches the correct final answer (note that Pass@k equals the number of ambiguous questions). Figure 2 and Figure 3 do exactly this, plotting Verification @k accuracy measured only on ambiguous questions from each benchmark. To reduce noise in these figures, we deterministically omit benchmark questions that Consistency @2nrunccy@2200 200 answers cor- rectly or where, with high probability, 50 random responses result in either all correct or all incorrect final answers.
2. Scaling Trends
This section examines how reasoning capability scales with two fundamental test-time compute axes:
Search refers to the compute used to discover candidate solutions. In this section, our knob for scaling search is the number of responses sampled for each reasoning problem (kinf in Algorithm 1).
A fteer controlling for the growth of @ass@k, we should ex- pect trend of decreasing accuracy if we increase k but keep the number of verification attempts constant. However, Figure 2 shows the reverse trend: accuracy increases with k. This demonstrates an implicit scaling of verification accu- racy, where increasing the number of generated responses increases not only the chance that at least one response is correct (Pass@k) but also the chance that at least one of the correct responses is of higher quality. Here, quality can be understood as the rigour or flawlessness of a response; a lower quality solution may be generally correct but fail to justify a non-trivial step or make a non-critical error.
Verification refers to the compute used to scrutinize candidate solutions. Our knob for scaling verification is the number of verification scores we compute and average over per solution (kiverit).
For computational reasons, this section uses streamlined form of Algorithm that omits bre-bre. This, for exam- ple, results in significant underestimates of Verification @ on MATH (see Table 3). All figures are averaged over 20 random seeds, where each run subsamples solutions and verification scores from a primary run that sampled 200 so- lutions per question and 50 verification scores per solution.
Implicit scaling suggests that verification should become more accurate, and sampling- more effective, with the use of more capable base models that produce more sound reasoning and compelling proofs of correctness. Because the number of ambiguous questions strictly increases with more candidate solutions, the implicit scaling effect also explains the overall accuracy scaling
search should become
2.1. Scaling Trends
Figure 1 provides a heatmap of Verification @k on each benchmark in Table as we scale search and verification.
3

---

Sample, Scrutinize and Scale: Effective Inference- Tineencc--iiee Search by Scaling Verification
Verification@k on AIME (All QS)
Verification@k on MATH (All Q5)
50
46.7
50
92.6
Accuracy On Questions (%6
of Ventication Attempts
40
44.9
of Verfication Attempts
40
91.0
All 00 A
42.7
89.0
40.0
86.6
36.2
83.1
5
21.5
$/
69.7
20
80
120
200
20
80
120
200
of Inference Attempts
of Inference Attempts
Verification@k on LiveBench Math (All Qs)
Verification@k on LiveBench Reasoning (All Qs)
Accuracy On All Questions (%)
Accuracy On All Questions (%
of Verif catiication Attempts
61.9
of Verificaatinn Attempts
64.1
60.2
60.8
58.1
56.6
55.2
50.7
5
20
80
120
160
200
43.6
$5
20
80
120
160
200
27.8
of Inference Attempts
of Inference Attempts
Figure 1. Hcatmap of Gemini v1.5 Pro accuracy rates using sampling- based search (without tie-breaking) as the number of responses generated (x-axis) and verification attempts (y-axis) increase. Warmer colors indicate higher accuracy (cubic scale). The largest gains occur when scaling both search and verification, with the strongest trend on AIME.
AIME
MATH
LiveBench Math
LiveBench Reasoning
Accuracy on Ambiguous Os (%))
100
Verification@k
80
60
40
50
100
150
200
50
100
150
200
0
50
100
150
200
0
50
100
150
200
Inference Attempts (k)
of Inference Attempts (k)
of Inference Attempts (k)
Inference Attempts (k)
Figure 2. Plot of Gemini v1.5 Pro accuracy rates using sampling-based search (without tie-breaking and with kiverif questions only as the number of responses generated increases. A question ambiguous when the model generates at least one candidate response with correct final answer. Accuracy on ambiguous questions increases with search.
ambiguous

---

Sample, Serutinize and Scale: Effective Inference- Search by Scaling Verification
gains in Figure I: larger k increases both the number of ambiguous questions @Pass@k) and accuracy on the set of ambiguous questions.
combat the fact that language models have low recall (i.e., often overlook) when asked to identify mistakes and hallucinations (Tyen et al., 2024; Kamoi et al., 2024a), as models are able to identify errors when provided their locations (Tyen et al., 2024). We can improve the self-verification of a candidate response by providing the verifier with other responses to compare the candidate against-an instance of implicit scaling.
2.3. The Long Tail of Response Distributions
We can directly observe Verification @k scaling beyond the saturation point of Consistency @k in Figure 4, where we plot their performance after fixing the number of verification attempts at 50. On AIME, the most technically challenging benchmark, Verification @k demonstrates power law scaling even as Consistency @k k begins to plateau. The rapid satu- ration of Consistency@e @k can be attributed to the fact that, while it is effective at small scales in averaging out noisy mistakes, it necessarily plateaus as it converges on the most probable response; for example, Consistency @50 50 has the same accuracy as Consistency @ 10,000 on AIME.
2. Rewrite responses for output style suitability. The opti- mal output style of a language model should depend on the task. Writing in a linear chain of thought which includes detailing reasoning before committing to a claim effective when generating responses (search) (Wei et al., 2022). However, responses are easier to verify when written rigorously, hierarchically, and mod- ularly. This can be leveraged by having verifiers first rewrite candidate responses in, e.g., an expanded math- ematically conventional peeorem-lemma-prod format rather than directly evaluating
Consider cheaply sampling a vast set of solutions from a weak but ergodic model: Consistency @k k is unlikely to return a correct solution, but an effective verifier should still be expected to detect rare but correct solutions in the long-tail of the response distribution. We find an example of this on the AIME 2024 exam, where the Gemini v1.5 model struggles to identify the correct answer to Problem 11 on Exam II. Table 2 shows the final answers from 200 randomly sampled Gemini v1.5 solutions, of which only one is correct (*601," in green).
These principles also provide levers for scaling self- verification capability with test-time compute, including by (1) sampling and providing verifiers with more responses to compare between and (2) rewriting responses with in- creasing rigour and structure.
3.1. Sampling- Based Search Implementation
Consistency returns the incorrect answer of "1" (in red), which appears in over half the responses. In contrast, Ver- ification successfully identifies the solution reaching the correct answer from the response distribution's long-tail, assigning a <36% score to each solution reaching a final answer of "1" but a 98% score to the single solution reach- ing "601". Scaling verification capability is key driving improved search, allowing for discerning between answers that appear correct with 98% vs. 76% confidence. The fact that verification can be used to so effectively leverage the long-tail of model response distributions also suggests that Pass@k, k, not @ass@ 1, should be the key performance metric for search applications. Existing post-training techniques (e.g.. reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022)) which explicitly optimize for Pass @ may potentially be doing so at the expense of Pass @k k and inhibiting search capabiliiy.
We now detail our minimalist implementation of sampling- based search (summarized in Algorithm 1) that uses only parallelizable blackbox queries to a language model. generates candidate responses by randomly sampling from models and select responses by asking models to self-verify; prompts are identical across all benchmarks.
Step 1: Generate Candidate Responses. A language model generates kinf candidate responses (candidate so- lutions) in parallel to each question, using temperature Oif-
Step 2: Verify Candidate Responses. A language model generates kvverif binary "verification scores" for each candi- date in parallel, indicating whether its final answer is correct. Each scoring attempt is a single conversation thread that rewrites the response as theorem, supporting lemmas, and proofs (examples in Appendix F) and systematically scans for errors. The highest scoring response is selected.
3. Effective Self- Verification
Tie-Break: Compare Candidate Responses. When the three highest scoring candidates score within 5% of one another and disagree on the final answer, language model directly compares the responses in pairwise matchups. Each matchup is a single conversation thread that identifies where responses diverge and, at each such point, determines which side is correct. Each matchup is repeated kise = 100 times, and the response with the most wins selected.
In the process of scaling sampling-base search, we identi- fied two general principles for eliciting more accurate self- verification, that may be of independent interest.
1. Compare responses to localize errors. Disagreements between candidate solutions strongly signal the poten- tial locations of their errors. This can be leveraged to
5

---

Sample, Scrutinize and Scale: Effective Inference-Tinee Search by Scaling Verification
Verification@k on AIME (Ambiguous Qs)
Verification@k on LiveBench Reasoning (Ambiguous Qs)
50
100.0
Cambue u0 S092 2200
Accuracy On Ambigunus Queecioon mmm (%)
Verification Attempts
of verification Attempts
30
52.8
35.9
20
20
37.1
29.3
10
5
21.3
5
160
22.8
# Inference Attempts
# of Inference Attempts
Verification@k LiveBench Math (Ambiguous
Verification@k MATH (Ambiguous
50
77.2
63.8
Recunooe On Ambigumus Questions (%)
of Verification Attempts
40
69.0
of Verification Artempts
40
56.0
Somiiguna Quesi (%6
60.9
48.3
30
52.7
40.5
20
44.5
32.8
5
20
40
80
120
160
200
36.3
5
20
40
120
160
200
25.0
Inference Attempts
Attempts
Figure 3. Heatmap of Gemini v1.5 Pro accuracy using sampling-based search (without tie-breaking) ambiguous questions only the number of responses generated (x-axis) and verification attempts (y-axis) increase. Warmer colors indicate higher accuracy (linear scale). question is ambiguous when the model generates least candidate response with correct final answer. Accuracy on ambiguous questions increases with search (x-axis).
AIME (AlI Qs)
(All Qs)
LiveBench (All Qs)
LiveBench Reasoning (All Qs)
140
Nerification@k
20
Poss@kk
100
120
Consistency@k
6
1400
100
400
120
80
350
100
60
3
4'-
Number of Inference Atterpps
Number of Inference Altempts
Number of Inference Attempts
Number of Inference Attempts
Figure 4. Line graph depicting the accuracy rates of the Gemini v1.5 Pro model using sampling-bast bbased search as the number candidate responses generated is scaled upwards. The number of verification attempts is fixed 50 for all plots. The depicted accuracies obtained without tic- hre-breaking and may he lower than reported elsewhere. Verification @k improves with k even when Consistency @1 stagnates on AIME LiveBench Reasoning.
6

---

Sample, Serutinize and Scale: Effective Inference-Tine Search by Scaling Verification
Problem 11, AIME 2024
Verification Score
Final Answer
# Solutions
Find the number of triples of nonnegative integers (a,b,c) satisfying a a+ b b+b 300 and a²b aac + bLa + b'e caa cbb=
0.98
601
(Correct)
0.76
6
(Wrong)
11
0.52
0
(Wrong)
14
6, 000, 000.
0.40
7
(Wrong)
21
0.38
4
(Wrong)
10
0.36
(Wrong)
124
125
0.22
10
(Wrong)
2
0.20
(Wrong)
9
Nonber of Solutons
0.18
301
(Wrong)
1
0.16
45451
(Wrong)
1
0.14
101
(Wrong)
2
0.06
2
(Wrong)
0.04
45151
(Wrong)
1
0.04
303
(Wrong)
303
45151
45451
301
633
0.00
100
(Wrong)
Final Anower
Table 2. The final answers identified by the Gemini v1.5 Pro model to Problem 11 on AIME 2024, sorted by verification score and annotated with their multiplicity in 200 solution generations. The correct final answer (green) only found by generated response whereas Consistency @ 200 selects an incorrect final answer (red) that is found by 124 generated responses.
3.2. Ablation Studies
Main refers to manually written prompts used in our ex- periments. Shortened refers to shorter variant of "Main" that omits, e.g., instructions to avoid truncation. Without Rewrite refers to a variant "Main" that omits instructing the verifier to first rewrite responses. Split-Contexi refers to variant of "Main" that creates separate conversation threads to individually verify pieces of the response.
We can individually ablate the practices of comparing and rewriting candidate responses to confirm their role in elicit- ing greater verification capability.
Ablating comparisons. The step of asking models to di- rectly compare candidate solutions with similar verification scores significantly increases sampling-basd based search perfor- mance. This is demonstrated in Table 3, where we depict the accuracy rates from Table alongside the accuracy rates after ablating the tie-breaking step. These comparisons have the greatest impact when models struggle from low recall and excessively assign high verification scores. On the MATH benchmark, which sees the greatest lift from comparisons, the average verification score of the top 3
The gap between the performance of "Main" and "Without Rewrite" demonstrates that ablating the rewriting of solu- tions negatively impacts verification performance. Similarly, the gap with "Split-Context" demonstrates that splitting the verification process into separate conversation threads sharply decreases performance due to low precision, which attribute to miscalibration.
candidate responses is nearly 90%.
4. Technical Details
All experiments are run on Google Cloud with Gemini vI.5-Pro-002 and Gemini v1.5-Flash-002 models dated to September 2024. Unless otherwise specified, the de- fault parameters for our implementation of sampling-based search (Section 3) are kinf 200, aini 1.5, kverif 50, Verif 1, and maximum of 8, 192 output tokens per query. For all benchmarks, the scoring of candidate responses is performed using a language model rather than literal string comparison; details are in Appendix C.2.
Ablating rewritings. We explored limited number of prompts for velffverification, including prompts which omit instructing the model to rewrite responses. We did not per- form further prompt optimization and expect refinements would boost accuracy. Table 4 shows each prompt's proba- bility of mislabeling correct solutions (false positive) and in- correct solutions (false negative), with the former generally having a more severe impact on downstream performance. We evaluated these prompts 1,080 candidate responses to 54 level-5 questions from the MATH training split, and 120 candidate responses to 6 questions from AIME 2023. response is marked as incorrect if, of 20 verification at- tempts, the number finding an error in the solution exceeds the equal error rate threshold.
Preliminary scoring. When generating kverif = 50 veri- fication scores per candidate solution is too expensive, we first generate kverit = 10 preliminary verification scores and discard candidate solutions with an average score below
7

---

Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
LiveBench Math
LiveBench
MATH
Reasoning
Dataset
Cons@200
Verification @ 200
56
70-
Without
With Tie-Break
48-
Accuracy
MATH
460 / 500
457 500
467 / 500
92-
40
60
LiveBench Math
118 / 200
125 200
135 / 200
90-
32- 55-
LiveBench Reasoning
75 140
94 140
97 140
AIME
4/ 14
7/ 14
8 14
Consistency@@000
Verification2200 Tie-Break)
Verification@2000 (With Tie fie-Break)
Table Accuracy rates Gemini v1.5 Pro using sampling-buss based search, with and without ne-breaking, Tie-breaking provides most Verification @ 200's gains on Consistency @ 200 (Cons @ 200) on MATH and LiveBench Math, and smaller gains on AIME and LiveBench Reasoning.
Prompts on MATH
Prompts on AIME
Prompt Style
MATH
AIME
14
FPR
FNR
FPR
FNR
22
Fabse Negative Rate
Falce Nogative
Main
14%
17%
7%
7%
20
Shortened
17%
17%
7%
7%
18
8
Without Rewrite
16%
18%
11%
12%
Split-Context
19%
23%
11%
14%
12
14
16
18
20
8
10
12
Positive
Positive
Shortened
Without-Rewrite
Spiit-Coneext
Table Verification scoring accuracy rates of Gemini v1.5 Pro model for various prompts. False positive rate (FPR) refers to how often correct response is labeled incorrect; false negative rate (FNR) refers how often incorrect response is labeled correct.
0.2. If final answer is represented by more than can- didate responses, only the top 15-as measured by average preliminary score, tie-breaking randomly kept. This results in smaller pool of candidate solutions for which we compute all Averif = 50 verification scores. Preliminary scoring is used on all datasets except AIME.
from the 150 available of October 21st 2024, including Zebra puzzles, Web-Or-Lies, and Spatial reasoning (White et al., 2024). Our AIME benchmark consists of the 15 questions in the 2024 Exam II (MAA, 2024).
5. A Verification Benchmark
Compute. On AIME, the verification process involves 32,000 characters (roughly 13,000 tokens) of model output. Extrapolating from these figures, running the full sampling- based search pipeline on question for kint 200 and kverif 50 requires 200 50 13, 000 = 130M output tokens. At around S5/1M output tokens (public pricing of Gemini v1.5 Pro), this is approximately S650 in cost. Preliminary scoring reduce output token usage by 70%, with per-question cost $200. The use of Gemini Flash for verification decreases cost to $12 per question.
Frontier language models demonstrate remarkable mis- match between their problem-solving capabilities and poor out-of-box verification capabilities. These limitations have largely been attributed to the inability of current language models to self-diagnose hallucinations or enforce rigour (Zhang et al., 2023; Orgad et al., 2024; Snyder et al., 2024; Kamoi et al., 2024a; Tyen et al., 2024; Huang et al., 2024). However, our findings that models can be directed to accu- rately perform verifications scale suggest that these out- of-box limitations can be addressed with standard methods like instruction tuning. We compiled set of challenging reasoning problems and candidate solutions to provide benchmark for these deficits.
Datasets. Our MATH benchmark consists of 500 ques- tions from the PRM800K (Lightman et al., 2024) test split of MATH (Hendrycks al., 2021). Our LiveBench Math benchmark consists of 200 random questions from the 368 available as of October s 2024, including AMC12 2023, AIME 2024, SMC 2023, USAMO 2023, IMO 2023, and synthetic math questions (White et al., 2024). Our LiveBench Reasoning benchmark consists 140 questions
Each entry in this benchmark consists question, a cor- rect candidate response, and an incorrect candidate response, and is manually curated from the residuals of our sampling- based search experiments (Section 3). Example entries from this benchmark can be found in Appendix E.
8

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
Model
Metric
Scoring Accuracy
Comparison Accuracy
Correct
Wrong
Flawed
@ass@1
76.5%
31.0%
22.2%
43.2%
GPT-4o
Consistency@55 @
77.4%
30.0%
11.1%
35.4%
Pass @1
89.6%
22.5%
33.3%
56.1% 1%
Claude 3.5 Sonnet
Consistency @5
90.3%
17.5%
33.3%
61.2%
Pass
100%
68.8%
80.0%
84.5%
ol-preview
Consistency @5
100%
79.4%
88.8%
92%
@ass@1
73.5%
44.5%
60%
58%
Gemini 2.0 Flash
Consistency @5
77.4%
42.5%
66.6%
58.7% 7%
Pass @1
75.4%
56.5%
53.3%
80%
Gemini 2.0 Thinking Flash
Consistency @5
77.4%
55%
55.5%
89.1% %
Random guessing
80%
20%
20%
50%
Table Accuracy rates of commercial language models verification benchmark. For the task of response scoring (Scoring Accuracy), accuracy rates are broken down for entries that require identifying a correct response as being correct (Correct), entries that require identifying wrong response being wrong (Wrong), and entries that require identifying wrong response that coincidentally reaches the correct answer as being wrong (Flawed). GPT-4o and Claude 3.5 Sonnet only perform marginally better than random guessing across all tasks. ol-Preview performs better, but still fails to identify 20-30% of wrong responses.
Our benchmark studies verification accuracy on two tasks:
cause Consistency simply averages out noise from an output distribution, meaning that a model biased towards produc- ing incorrect answer will do so with higher probability under Consistency. Addressing these deficits in verifica- tion capabilities-whice we see as low-hanging fruit for post-training-wuutt enable not only better sampling-base search, but also other downstream applications of verifica- tion including reinforcement learning (e.g. OpenAI, 2024; Team, 2025), data flywheeling (e.g., Welleck al., 2022), and end-user experience (further discussion in Section A).
1. Scoring task. When given only the question and one of the responses, is the model able to discern the cor- rectness of the response?
2. Comparison task. When provided the whole tuple with the correctness labels of the responses masked and a guarantee that at least one response is correct, is the model able to discern which response is correct and which is incorrect?
The scoring task is also evaluated over separate set of (question, response) pairs where the response reaches the correct final answer by coincidence but contains fatal errors and should be labeled by a reasonable verifier as being incor- rect; an example can be found in Appendix E. In the scoring task, models are provided only with the task description; in the comparison task, models are provided only with the task description and suggestion to identify disagreements between responses in its reasoning.
6. Conclusion
This paper studied the scaling trends governing sampling- based search with self-verification, finding that (1) it scales remarkably well even with simple implementations, (2) im- plicit scaling plays big role in this scalability, and (3) self-verification capability can be scaled with test-time com- pute using two key principles: comparisons localize errors, and responses should be rewritten for output style suitability, To this end, we scaled a minimalist, embarrassingly parallel implementation of sampling baased search that, with suffi- cient test-time compute, is sufficient to attain state-of-art performance on range of reasoning benchmarks. Given that complements other test-time compute scaling strate- gies, is parallelizable and allows for arbitrarily scaling, and admits simple implementations that are demonstrably ef- fective, sampling based search will play a crucial role as language models are tasked with solving increasingly com- plex problems with increasingly large compute budgets.
Table 5 lists the baseline performances of current commer- cial model offerings on this benchmark. Gemini v1.5 Pro is omitted from the benchmark as the entries the benchmark are curated from the residuals of Gemini vI.5 Pro. The prompts used in Table are provided in Appendix C.4.
As we previously observed, and has been noted in prior works (Tyen et al., 2024; Kamoi et al., 2024a), verifica- tion errors are typically due to low recall. Even the casier comparison task, models perform only marginally better and often worse- worse-than random chance. In many cases, Consistency @5 performs worse than one-shot inference be-

---

Sample, Serutinize and Scale: Effective Inference-Tine Search by Scaling Verification
Acknowledgments
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training Verifiers to Solve Math Word Problems, November 2021.
This material is based on work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 2146752 and by Google PhD
Fellowship (EZ).
Collins, M.. Jiang, A. Q.. Frieder, S., Wong, L., Zilka, M., Bhatt, U., Lukasiewicz, T., Wu, Y., Tenenbaum, J. B., Hart, W. Gowers, T., Li, W., Weller, A., and Jamnik, M. Evaluating language models for mathemat- ics through interactions. Proceedings of the National Academy of Sciences of the United States of America, 121 (24) ISSN 0027-8424.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., and Weston, J. Chain-of-verification reduces hallucination in large language models. In Ku, L., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August // 11-16, 2024, pp. 3563-3578. Association for Computational Linguistics, 2024.
References
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., Behbahani, F., Faust, A., and Larochelle, H. Many-shot in-context learning, 2024.
Akyürek, E., Damani, M., Qiu, L., Guo, H., Kim, Y., and Andreas, J. The surprising effectiveness of test-time training for abstract reasoning, 2024.
Gao, K., Cai, H., Shuai, Q., Gong, D., and Li, Z. Embedding Self- Self-Correction Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning, October 2024.
An, S., Ma. Z., Lin, Z., Zheng, N., Lou, J.-G., and Chen, W. Learning From Mistakes Makes LLM Better Reasoner, March 2024.
Gemini Team. Gemini 1.5:
Unlocking multimodal under- standing across millions of tokens of context, 2024.
Anil, C., Zhang, G., Wu, Y., and Grosse, R. Learning to Give Checkable Answers with Prover-Verifte Games, August 2021.
Gou, Z., Shao, Z.., Gong, Y., Shen, Y., Yang, Y., Duan, N., and Chen, W. CRITIC: Large Language Models Can Self-Correc with Inte-Interective Critiquing, February 2024.
Anil, C., Durmus, E., Rimsky, N., Sharma, M., Benton, J. Kundu, S., Batson, J., Tong, M., Mu, J., Ford, D. J., Mosconi, F., Agrawal, R., Schaeffer, R., Bashkansky, N., Svenningsen, S., Lambert, M., Radhakrishnan, A., Deni- son, C., Hubinger, E. J.. Bai, Y., Bricken, T., Maxwell, T., Schiefer, N., Sully, J., Tamkin, A., Lanham, T., Nguyen, K., Korbak, T., Kaplan, J., Ganguli, D., Bowman, S. R., Perez, E., Grosse, B., and Duvenaud, D. Many-shot jailbreaking. 2024.
Havrilla, A., Raparthy, S. C., Nalmpantis, C., Dwivedi- Yu, J., Zhuravinskyi, M., Hambro, E., and Raileanu, Glore: When, where, and how to improve LLM reasoning via global and local refinements. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song. D., and Steinhardr, Measuring math- ematical problem solving with the MATH dataset. In Van- schoren, J. and Yeung, S. (eds.), Proceedings of the Neu- ral Information Processing Systems Track on Datasets and Benchmarks I, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. 2024. URL https:: //at:///avivv////a org/ abs .2.40.27770 21787.
Chen, Y. Pan, X., Li, Y., Ding, B., and Zhou, J. A Simple and Provable Scaling Law for the Dist-Time&& Compute of Large Language Models, November 2024.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, ICLR 2024. Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Chow, Y., Tennenholtz, G., Gur, I., Zhuang, V., Dai, B., Thiagarajan, S., Boutilier, C., Agarwal, R., Kumar, A., and Faust, A. Inference-Aweer Aware Fine- Fine-Tuning for Best-of-N Sampling in Large Language Models, December 2024.
10

---

Sample, Serutinize and Scale: Effective Inference- Search by Scaling Verification
Jiang, W. Shi, H., Yu, L.,, Liu, Z., Zhang, Y., Li, Z., and Kwok, J. T. Forward-back ward reasoning in large lan- guage models for mathematical verification. In Ku, L., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics, ACI. 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 6647-6661. Association for Computational Linguistics, 2024.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Liu, D., Nasscreldine, A., Yang, Z., Xu, C., Hu, Y., Li, J., Kumar, U., Lee, C., Qin, R., Shi, Y., and Xiong. J. Large Language Models have Intrinsic Self-Correction Ability, December 2024.
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain,
D., Perez, E., Schiefer, N., Hatfield- Doddss Z., DasSarma,
N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec. S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. Language Models (Mostly) Know What They Know, November 2022.
MAA, M. A. A. AIME 2024 Problem Set, 2024.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-Refine: Iterative Refinement with Self- Fee--eeeb,,,, May 2023.
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t, Koh, P. W., Tyyer, M., Zettlemoyrr L.,, and Hajishirzi, H. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, October 2023,
Kamoi, R., Das, S.S. S. S., Lou, R., Ahn, J. Zhao, Y., Lu, X., Zhang, N., Zhang, Y., Zhang, R. H., Vummanthala, S. R., Dave, S., Qin, S., Cohan, A., Yin, W., and Zhang, R. Evaluating LLMs at Detecting Errors in LLM Responses, July 2024a.
OpenAL.
Introducing OpenAl https: com/indew
ol ol-preview.
Kamoi, R., Zhang, Y., Zhang, N., Han, J., and Zhang, R. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Celf-Correction of LLMs. Transac- tions of the Association for Computational Linguistics,
Lxtrocuuccngg--ennaa---------eee 2024.
Orgad, H., Toker, M., Gekhman, Z., Reichart, R., Szpektor, I., Kotek, H., and Belinkov, Y. LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations, October 2024.
12:1417-1440, 2024b.
Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. In Oh, A., Naumann, T., Globerson, Advances
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C. L.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F. Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. Advances in Neural Information Processing Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans,
A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Kirchner, J. H., Chen, Y., Edwards, H., Leike, J., McAleese, N., and Burda, Y. Prover-verifier games improve legibility of Il outputs, 2024.
Systems 35: Annual Conference
IA, USA, November 28 December 9, 2022, 2022.
Ko, J. Baek, J., and Hwang, S. J. Real-time Verification and Refinement of Language Model Text Generation, January 2025.
Paul, D., Ismayilzada, M., Peyrard, M., Borges, B., Bosselut, A., West, R., and Faltings, B. REFINER: Reasoning Feed- back on Intermediate Representations, February 2024.
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D.,
Singh, A., Baumli, K., lqbal, S., Bishop, C., Roelofs, R.,
Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y.. Huang, Q., Liden, I.., Yu, 7.,, Chen, W., and Gao, J. Check Your Facts and Try Again: Improving Large Lan- guage Models with External Knowledge and Automated Feedback, March 2023.
Zhang, L. M., McKinney, K., Shrivastava, D., Paduraru, C., Tucker, G., Precup. D., Behbahani, F., and Faust, A.
Training Language Models to Self-Correct via Reinforce- ment Learning, October 2024.
Li, Y., Lin, Z., Zhang, S., Fu, Q.. Chen, B., Lou, J.-G., and Chen, W. Making Large Language Models Better Reasoners with Step-Aware Verifier, May 2023.
Qu, Y., Zhang, T., Garg, N., and Kumar, A. Recursive Introspection: Teaching Language Model Agents How to Self-Improve, July 2024.

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators, June 2022.
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language Agents with Verbal Reinforcement Learning, October 2023.
Wei, J.. Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, E. Chi, E. H., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., and Oh, A. (eds.), Advances in Neu- ral Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A. T., Kumar, A., Alemi, A. A., Rizkowsky, A., Nova, A., Adlam, B., Bohnet, B., Elsayed, G. F., Sedghi, H., Mordatch, I., Simpson, I., Gur, I., Snoek, J., Pennington,
J. Hron, J., Kenealy, K., Swersky. K., Mahajan, K., Culp, L., Xiao, L.., Bileschi, M. L., Constant, N., Novak, R., Liu, R., Warkentin, T., Qian, Y., Bansal, Y., Dyer, E., Neyshabur, B., Sohl-Dickstein, J., and Fiedel. N. Beyond human data: Scaling self-training for problem-solving with language models. Trans. Mach. Learn. Res., 2024, 2024.
Welleck, S., Lu, X., West, P., Brahman, F., Shen, T. Khashabi, D., and Choi, Y. Generating Sequences by Learning to Self-Correct, October 2022.
Wen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., Bowman, S. R., He, H., and Feng. S. Lan- guage Models Learn to Mislead Humans via RLHF, September 2024.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024.
Weng. Y., Zhu, M. Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., and Zhao, J. Large language models are better reasoners with selffverriication. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 2550-25755 Association for Computational Linguistics, 2023.
Snyder, B., Moisescu, M., and Zafar, M. B. On Early Detection of Hallucinations in Factual Question Answer- ing. In Proceedings of the 30th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, pp. 2721-2732, Barcelona Spain, August 2024. ACM. ISBN 9798400704901.
Stechly, K., Valmeckam, K., and Kambhampati, S. Chain of thoughtlessness? an analysis of cot in planning, 2024a.
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv R., Jain, N., Saifullah, K., Naidu, S., Hegde, C., LeCun, Y., Goldstein, T., Neiswanger, W., and Goldblum, M. Livebench: A challenging, contamination- free 1lm benchmark, 2024.
Stechly, K., Valmeekam, K., and Kambhampati, S. On the self-verification limitations of large language models reasoning and planning tasks, 2024b.
Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., and Jiang, M. Large language models can self-correct with key condition verification, 2024a. URL ptattp:///axiv.. org/ abs / 2400 22244.111999 14092.
Team, D.-A. Deepseek-rl: Incentivizing reasoning capabil- ity in Ilms via reinforcement learning, 2025.
Tyen, G., Mansoor, H., Cärbune, V., Chen, P., and Mak, T. LLMs cannot find reasoning errors, but can correct them given the error location, June 2024.
Wu, Z., Zeng, Q., Zhang. Z., Tan, Z., Shen, C., and Jiang, M. Large Language Models Can Self- Ooff-Coreet with Minimal Effort. June 2024b.
Uesato, J., Kushman, N., Kumar, R., Song, F,, Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, Solv- ing math word problems with process- and outcome- based feedback, November 2022.
Xie, Y.. Kawaguchi, K., Zhao, Y., Zhao, X., Kan, M., He, J., and Xie, M. Q. Self-evaluation guided beam search for reasoning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023. NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Valmeekam, K., Marquez, M., and Kambhampati, S. Can large language models really improve by self-critiqiing their own plans?, 2023.
12

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
Xue, T., Wang, Z., Wang, Z., Han, C., Yu, P., and Ji, H. RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thought, October 2023.
Yao, S., Yu, D., Zhao, J.. Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Treee of thoughts: Deliberate problem solving with large language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans,
LA, USA, December 10- 16, 2023, 2023.
Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., and Agarwal, R. Generative Verifiers: Reward Modeling as Nexe-Token Prediction, October 2024.
Zhang, M., Press, O., Merrill, W., Liu, A., and Smith, N. A. How Language Model Hallucinations Can Snowball, May 2023.
13

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
A. Related Work
Test-time compute. Many of the recent advances in language model reasoning capabilities can be traced to increasing use of test-time compute. Inference strategies like shain-of-thoughh reasoning (Wei et al., 2022), tre-of-thouugts (Yao et al., 2023) and self-critique (Valmeekam et al., 2023) result in improved reasoning performance the cost of forming longer responses. Reinforcement learning has emerged as particularly successful strategy for effectively leveraging more test-time compute, wherein models learn from exploration to form lengthy chain-of-thought outputs that incorporate backtracking and search, despite not being explicitly taught to do so (OpenAI, 2024; Team, 2025). inference-time model adaptation, whether through many-shot learning (Agarwal et al., 2024; Anil et al., 2024) or finetuning (Akyürek et al., 2024), provides another avenue when training data is available. We study sampling- -aannd search: obtain a set of candidate responses from a model and apply an aggregation method to select response, such as ceff=consiteencypluraaiii voting (Wang et al., 2023) or selecting a response with a reward/verifier model (Cobbe et al., 2021). These various methods for scaling test-time compute are complementary; for example, sampling-base -ased search can also be used models trained to produce longer outputs. We note that it is possible for models trained to produce long chains of thought to perform something resembling sampling-bans based search internally, in which case we still expect our observed scaling trends to hold. However, we also expect explicit sampling-base search will remain indispensable, due to its greater parallelism and robustness than internally implemented search.
Scaling sampling -based search. The paradigm of sampling-based search provides three main knobs for scaling: gen- eration, sampling, and selection. While the cost of generating each individual response can be scaled with previously mentioned interventions, such as chain-of- ofahhutf- (e.g. Wei et al., 2022), reinforcement learning (e.g. OpenAI, 2024), or inferencc-time adaptation (e.g. Anil al., 2024), the cost of sampling a set of responses can be scaled by increasing the number of responses generated (Wang et al., 2023; Snell al., 2024; Brown et al., 2024). We use random sampling to generate each set of candidate responses, which means the latter corresponds to simply taking more random draws. Whereas prior work on parallel test-time compute scaling via Pass@ k, such as Brown et al. (2024), has focused on understanding how Pass@1 k scales, we are concerned with how much of the @ass@k Pass @ gap can actually be attained in practice. However, this sampling can also be implemented in an agentic fashion, with a central model delegating the generation of responses so as to perform search more systematically. The process of selecting a response can be scaled by using more expensive rules: celf-consistency provides a simple plurality voting rule at the lowest-cost end of the spectrum (Wang et al., 2023), while language model self-verification (e.g. Xue et al., 2023, see below) and learned verification/rewward models (e.g. Cobbe et al., 2021, see below) provide range of selection strategies that vary in cost and capability. For more fine-grained control over the scaling of self-verification in our experiments, we apply plurality voting (Wang et al., 2023) to self-verification and vary our number of verification attempts per response.
Verification of language model outputs. A large body of recent work has studied the self-verification capabilities of
large language models (e.g., Cobbe et al., 2021; Kadavath et al., 2022; Saunders et al., 2022; Kim et al., 2023; Xie et al.,
2023; Weng et al., 2023; Zhang et al., 2023; Xue et al., 2023; Li et al., 2023; Liu et al., 2024; Chow et al., 2024; Jiang et al.,
2024; Dhuliawala et al., 2024; Snyder et al., 2024; Wu et al., 2024b; Huang et al., 2024; Kamoi et al., 2024a;b; Orgad et al.,
2024; Wen et al., 2024: Tyen et al., 2024; Chen et al., 2024; Kumar et al., 2024; Qu et al., 2024; Zhang et al., 2024; Ko et al., ours simply ask models to perform verification and parse the response, others have proposed custom methods of performing self eerfiveriii,tion, including: recreating the problem from the response (Xue et al., 2023; Wu et al., 2024b), masking and re-filling parts of the response (Weng et al., 2023; Jiang et al., 2024), creating a rubric (Dhuliawala et al., 2024), or asking models to choose from options (Xie et al., 2023; Chen et al., 2024). Our work does not focus on optimizing for self-verification or advocate for any particular strategy. However, in the course of performing our scaling study, we did identify several previously unstudied principles of self-verificattio that only arise at sufficiently large scale and may be of independent interest, including implicit scaling, output style suitability, and the importance of directly comparing responses. Other related bodies of work study the learning of verifiers, often on top of a pretrained large language model (c.g. Cobbe et al., 2021: Saunders al., 2022; Li et al., 2023; Havrilla et al., 2024; Kumar et al., 2024; Qu al., 2024; Chow et al., 2024; Zhang et al., 2024), and the use of external tools for verification (e.g. Min et al., 2023; Gou et al., 2024; Gao et al., 2024; Kim et al., 2023). We did not train customized verification models or permit verifier use of external tools in the listed experiments, as we found blackbox model access to be sufficient for effective verification at scale. The limitations of model verfiverrfiattion capabilities are also well-studied (Kamoi et al., 2024a; Tyen et al., 2024; Huang et al.. 2024; Wu et al., 2024a), and can be remedied with external information (Huang et al., 2024) or hints for localizing errors (Tyen et al., 2024). Models especially struggle with self eeff-diagnosin hallucinations (Zhang et al.,
2025: Havrilla et al., 2024). While some works-innlucee
14

---

Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
2023; Orgad al., 2024; Snyder et al., 2024), despite awareness of their own limitations (Kadavath et al., 2022), and often incentivized obfuscate errors (Wen et al., 2024).
Applications of verification. In addition being used select from candidate responses (Cobbe et al., 2021; Li et al., 2023; Weng al., 2023; Jiang al., 2024; Chen et al., 2024; Xie al., 2023), verifiers be used guide iterative improvements
model's output by providing feedback to the generating model (Kim et al., 2023; Xue et al., 2023; Valmeckam et al., 2023; Wu et al., 2024b; Huang et al.. 2024; Dhuliawala et al., 2024; Stechly et al., 2024a;b; Qu et al., 2024; Havrilla et al., 2024; Ko et al., 2025). Another important application of verification is enhancing model capabilities. For example, verification results for model outputs can be fed back into models as feedback via in-context reinforcement learning (Shinn et al., 2023), reinforcement learning (Uesato et al., 2022; Peng et al., 2023; Madaan al., 2023; Kumar al., 2024; Chow et al., 2024), or finetuning (Welleck al., 2022; Paul al., 2024; An et al., 2024; Singh al., 2024), in approach known data flywheeling. Verification has also been explored as means of encouraging models to produce better written responses (Anil et al., 2021; Kirchner et al., 2024). From a product perspective, verification capabilities are also important to the workflow of end users (Collims et al.).
B. Additional Experiments
B.1. Smaller Models
Model
Method
AIME
MATH
LiveBench Math
LiveBench Reasoning
Pass@1
1/
426 500
104 / 200
63 / 140
Pro v1.5
Consistency @ 200
4/15 15
460 500
118 / 200
75 140
Verification 200
8/
467 / 500
135 / 200
97 140
Pass@)
2/15
407 / 500
96 / 200
65 140
Flash v1.5
Consistency @ 200
3/15
440 500
92 / 200
84 / 140
Verification @ 200
5/15
445 500
104 / 200
84 / 140
Pro+Flash v1.5
Verification @ 200
7/15
456 500
/ 200
140
AIME
Verification with Gemini Flash and Pro
100
1000
89%
91%
Accuracy (%)
80
Accurscy (%)
80
100
Pass@11
47%
Verification@200
40-
40
Flash Verification@200
Accuracy (%)
80
Verification@200
20
20
Pro
Pro Flash
Pro Flash
60
LiveBench Math
LiveBench Reasoning
40
Accuracy (%)
67%
Accurecc (%)
700%
60
59%
60
60%
50%
40
40
20
AIME
MATH
LiveBench Math
LiveBench Reasoning
Flashh
Pro
Plash
Flash
Pro
Flash
Ross@1
Consistency@200
Veriiccatio@2000
Table Accuracy rates with sampling-based search using either the Gemini v1.5 Pro model to both generate and verify responses (Pro), Gemini v1.5 Flash both generate and verify responses (Flash), or Gemini v1.5 Pro model generate responses v1.5 Flash to verify responses (Pro+Flash). Verification@ exceeds Consistency @ 200 for all model choices, while Pro+Flash Verification @ 200 matches or exceeds Pro Consistency@ @
We also observe sampling- bumpling-based based search to be a powerful tool for enhancing smaller, lower-cost models. Here, we apply sampling-based search to Gemini v1.5 Flash model, which has a nearly 20x lower inference cost than Gemini v1.5 Pro. Table 6 lists the performance of using the Flash model to evaluate candidate responses generated by the Pro model (Pro+Flash), and the performance of using the Flash model end-to-end for sampling-hee search (Flash). Sampling-hes
15

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
search still provides significant improvement in performance for both Flash and Pro+Flash. Moreover, Verification @ 200 still provides significant improvements over Consistency 200, albeit lesser in magnitude than for end-to-end use of Gemini Pro. In addition, Flash Verification @ 200 using Gemini Flash is competitive with Pro Consistency @ 200, while Pro+Flash Verification 200 exceeds Pro Consistency @ 200. We highlight that Pro+Flash Verification @ 200 has roughly the compute cost of Consistency 500 500-as as our sampling-ber search implementation is minimally optimized for efficiency, we expect costs to further decrease.
B.2. Performance by Subtask
Dataset
Cons@ 200
Verif@ 200
Improvement (%)
Pass@200
Questions
(Abs)
(Rel)
Berkeley MATH
92.0%
93.4%
2% t
20.0%
99.0%
500
AIME 2024
26.7%
53.3%
100%
57.1%
73.3%
15
Web-of-Lies-v2*
75.5%
91.8%
22% 1
66.5%
100.0%
49
Spatial*
33.3%
46.7%
40% 1
21.5%
95.6%
45
Zebra Puzzle*
50.0%
67.4%
35%
36.4%
97.8%
46
Competition
66.2%
83.1%
26% 1
63.1%
93.0%
71
AMPS Hard" 70.6%
77.7%
10%
33.5%
91.8%
85
Olympiad
25.0%
22.7%
9% 4
-11.2%
45.5%
44
Table The Pass@ 200, Consistency 200 (Cons (2ons)200), and Verification 200 (Verif@200 200) accuracy rates of the Gemini v1.5 Pro model using sampling-hased search. LiveBench Math' and LiveBench Reasoning' numbers are divided per task. Absolute % Increase (Abs) is percentage improvement Verification over Consistency @200. Relative Increase (Rel) (Verification 200 Consistency@22 @200) (Pass@ 200 Consistency @ 200).
The LiveBench benchmarks each consist of multiple subtasks. In Table 7, we break down the numbers reported in Table I for each of these subtasks. We also provide in Table the Pass @as@220 200 scores of the Gemini Pro model, which measure the probability that of 200 attempted responses to a question at least one is correct. Pass 200 upper bounds what one can hope to achieve through Verification or Consistency. Verification provides the greatest gains on AIME 2024, Web-of-Lies, Competition, and Zebra Puzzle. In contrast, Verification does not improve on Consistency on the Olympiad task of the LiveBench Math benchmark. We attribute this the unique question design of LiveBench Olympiad task questions, which incompatible with implementation of Verification (see Appendix B.4).
B.3. Temperature Tuning
After attempting temperature tuning on the training split of the MATH dataset, we found that the choices of temperatures Jint and vveri did significantly affect performance. In Table 8 and Table 9, compare the post-veríficatio accuracy of pipeline for various temperature choices. The Verification @2 20 figures obtained without running tie- fie-brekking and with only 20 verification attempts.
Inference Temp oint
Pass 20
Consistency @20
Verification @ 20
0.2
89/100
76/100
82/100
1.0
94/100
73/100
80/100
1.5
89/100
73/100
79/100
2.0
89/100
75/100
82/100
Table 8. Accuracy rates of the Gemini v1.5 Pro model on the training split of MATH using different methods of selecting from 20 generated responses and varying the temperature used to generate the responses. The temperature used for verification attempts fixed Overit = 1.0. Inference temperature does not significantly affect downstrcam performance.
16

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
Verification Temp Overif
Pass@20
Consistency@22 @2
Verification @ 20
0.2
89/100
73/100
76/100
1.0
89/100
73/100
79/100
2.0
89/100
73/100
79/100
Table Accuracy rates of the Gemini v1.5 Pro model on the training split of MATH using different methods of selecting from 20 generated responses and varying the temperature used to for verification attempts. The temperature used for generating responses is fixed at auf = 1.5. Verification temperature docs not significantly affect downstream performance.
B.4. Olympiad LiveBench Math Subtask
The one task for which we saw no lift from verification the Olympiad questions from LiveBench MATH. These questions formatted as open-ended problems. Rather, they take a very specific form of asking one to fill in pre-written proof from menu of expression options, and to output specific sequence of indices corresponding these options. This is incompatible with our verification pipeline, which asks the verification model to rewrite candidate responses in theorem-lemma format where the theorem states the final answer. For example, the final answer to the Olympiad question at the bottom this section is the following sequence:
Representative example of Olympiad task from LiveBench MATH
You are given question and its solution. The solution however has its formulae masked out using the tag imissing X where X indicates the identifier for the missing tag. You are also given a list of formulae latex in the format "jexpression Yú = latex code" where Y is the identifier for the formula. Your task missing tags in the solution. Think step by step loud as what the answer should be. If you not sure, give your best guess. Your answer should be in the form of list of numbers, e.g., 5, 22, 3, .... corresponding to the expression identifiers that fill the missing parts. For example. if your answer starts as 5, 22, 3, ... then that means expression 5 fills mmssing li. expression 22 fills imissing 2i, and expression 3 fills imissing 3g.
match the formulae the
The question is: Find all integers n 2 such that the following property holds: we list the divisors of in increasing order as di < d2 < ... d2 = n!! then we have
d2 di S d3 d2 S S dR da-1-
Find all integers 2 3 such that the following property holds: if we list the divisors of n! in increasing order as 1 = di d2 < ... < du n!, then we have
d2
ds d2 < d2 da-1-
The solution is: We can start by verifying that imissing 1ú and n = work by listing out the factors imisssig 2 and mmissing 3g. We can also see that imissing 4; does work because the terms 15, 20, and 24 are consecutive factors imissiig 5g. Also, imissing 6ú does not work because the terms imissing 7g, and 9 appear consecutively in the factors of jmissing 86.
We can start by verifying that imissing 9; and imissing 10, work by listing out the factors of imissing 11, and 4!. We can also see that imissing 12; does not work because the terms 15, 20, and 24 are consecutive factors of 5!. Also, ;missiig 13, does not work because the terms imissing 14, and appear consecutively in the factors of :missing 15%-
Note that if we have prime number jmissing 16; and an integer imissing 17, such that both and jmissiig 18; are factors missing 19;, then the condition cannot be satisfied.
If imissing 20g is odd, then imissing 21g is factor of mmissnng 22g, Also, imissing 23g is factor of n!. Since ;missing 24; for all (missing 25g, we can use Bertrand's ''s Postulate to show that there is at least one prime number such that imissing 26,,- Since we have two consecutive factors of mmissing 27g and prime number between the smaller of these factors and n, the condition will not be satisfied for all odd imissiig 28;-
If imissiig 29, is even, then (2)("=²) n² + 3 is a factor :missing 31g, Since jmissing 32; for all n 8, we can use Bertrand's Postulate again to
[[("~")( (n 2) = n² 4n + 4 is a factor of mmissnn 30g- Also, - (n-3)(n - - 1) =
17

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
show that there least one prime number p such that missing 33g- Since we have two consecutive factors of ;missing 342 and a prime number between the smaller of these factors and n, the condition will not be satisfied for all even imissing 35
Therefore, the only numbers that work are and jmissiig 36,.
The formulae are:
expression 1g
expression 2/
jexpression 3/ 3!
jexpression 4g + 1
(omitted for brevity)
jexpression 33/, n < p < n²2 4n +3
jexpression 34g > n
jexpression 35g < p < n² 2n
jexpression 36g n = 4
Your final answer should be STRICTLY in the format: Detailed reasoning
Answer: ;comma separated list of numbers representing expression identifiers;
C. Additional Technical Details
C.1. Inference Prompts
For questions from the MATH and AIME benchmarks, we use the following prompt.
MATH and AIME Prompt
Please answer the following question. Think carefully and in step-by-stey fashion. At the end of your solution, put your final result in boxed environment, e.g. 42
The question would be here.
For questions from the LiveBench Math and LiveBench Reasoning benchmarks, which already come with their own instructions and formatting requests, we do not provide any accompanying prompt and simply submit the model the question verbatim.
LiveBench Prompt
The question would be here.
C.2. LM-Based Scoring
Given a tuple consisting of a question, ground-truth solution, and candidate response, we grade the correctness of the candidate response by querying a
model to compare the candidate and ground-truth solutions. This involves repeating the following process five times: (1) send prompt to the model that provides the question, the correct ground-truth solution, and the candidate response, and asks the model deliberate on the correctness of the candidate response; and (2) send followup prompt the model to obtain correctness ruling in structured format. If strict majority of (valid) responses the second prompt evaluate to JSON object with the key-value pair
"student final answer is correct" True rather than "student final answer correct"
False the candidate response
18

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
labeled correct. Otherwise, the candidate response is labeled incorrect. These queries all processed with temperature zero. The prompts, which can be found the end of this subsection, ask the language model to (1) identify the final answer of the given response, (2) identify the final answer of the reference (ground truth) response, and (3) determine whether the final answer of the given response satisfactorily matches that of the reference response. ignoring any non-substantive formatting disagreements. In line with convention, we instruct our scoring system to ignore the correctness of the logic used to reach the final answer and rather only judge the correctness of the final answer. The model is asked to label all non-sensical and incomplete responses as being incorrect.
As form of quality assurance, every scoring output for the Consistency @200 and Verification @ 200 figures depicted in Table manually compared against human scoring. No discrepancies between automated and human scoring were found on the MATH and AIME datasets for both Consistency @ 200 and Verification @ 200. No discrepancies were found on LiveBench Reasoning for Consistency @ 200. For Verification @ 200, one false positive (answer labeled by automated system as being incorrect but labeled by human as being correct) and one false negative (answer labeled by automated system as being correct but labeled by human as being incorrect) were identified on LiveBench Reasoning; three false positives and four false negatíves were identified on LiveBench Math. For Consistency 200, two false negatives were identified on LiveBench Math. This means that LM scoring matched human scoring 99% of the time, and the choice of human versus automated scoring matters little to results.
Prompt
You are an accurate and reliable automated grading system. Below are two solutions to a math exam problem: a solution written by a student and the solution from the answer key. Your task is to check if the student's solution reaches a correct final answer.
Your response should consist of three parts. First, after reading the question carefully, identify the final answer of the answer key's solution. Second, identify the final answer of the student's solution. Third, identify whether the student's final answer is correct by comparing to the answer key's final answer. # The question, answer key, and student solution The math exam question:
The question would be here.
The answer key solution:
The reference solution would be here.
The student's solution:
The candidate solution would be here.
# Your response format
Please structure your response as follows. PROVIDE A COMPLETE RESPONSE.
Answer Key Final Answer
Identify the final answer of the answer key solution. That's all you need to do here: just identify the final answer. "final answer" can take many forms, depending on what the question asking for; can be a number (e.g., "37"), string (e.g., "ABCDE"), sequence (e.g., "2,3,4.5"", letter (e.g.. "Y"), a multiple choice option (e.g. "C"), word (c.g.. "Apple"). algebraic expression (e.g. "x= + 37"),, quantity with units (e.g. "4 miles"), any of number of other options. If solution concludes that the question is not answerable with the information provided or otherwise claims that there is no solution to the problem, let the final answer be "None". If the solution does not produce any final answer because it appears to be cut off partway or is otherwise non-sensical, let the solution's 's final answer be "Incomplete solution" (this could only ever possibly happen with the student solution). YOUR RESPONSE HERE SHOULD BE BRIEF. JUST IDENTIFY WHAT THE QUESTION IS ASKING FOR, AND IDENTIFY THE ANSWER KEY'S FINAL ANSWER. DO NOT ATTEMPT TO ANSWER THE QUESTION OR EVALUATE INTERMEDIATE STEPS.
19

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
Student Solution Final Answer
Identify the final answer of the student solution.
YOUR RESPONSE HERE SHOULD BE BRIEE JUST IDENTIFY WHAT THE QUESTION IS ASKING FOR, AND IDENTIFY THE STUDENT'S S FINAL ANSWER. DO NOT ATTEMPT TO ANSWER THE QUESTION OR EVALUATE INTERMEDIATE STEPS. # Correctness
Simply evaluate whether the student's final answer is correct by comparing it to the answer key's final answer. Compare the student's 's final answer against the answer key's final answer to determine if the student's final answer correct.
* It does not matter how the student reached their final answer, long their final answer itself is correct.
It does not matter how the student formatted their final answer; for example, if the correct final answer is 7/2
14 It does matter if the student's final answer uses the student may write ***3.5*** three and half or
the same specific formatting that the question asks for, such writing multiple choice options in the form "(E)" rather than "***"***"
It does not matter if the student omitted units such dollar signs. & If the student solution appears to be truncated or otherwise incoherent, e.g. due to technical glitch, then it should be treated as being incorrect.
ONCE AGAIN, DO NOT EVALUATE INTERMEDIATE STEPS OR TRY TO SOLVE THE PROBLEM YOUR- SELF. THE ANSWER KEY IS ALWAYS RIGHT. JUST COMPARE THE FINAL ANSWERS. IF THEY MATCH, THE STUDENT ANSWER IS CORRECT. IF THEY DO NOT MATCH, THE STUDENT ANSWER IS INCOR- RECT.
# Summary
Answer key final answer: (The final answer of the answer key solution. Please remove any unnecessary formatting, e.g. provide "3" rather than |(3 provide "E" rather than "*** provide "1, 2, 3" rather than "[1, 2, 3]", provide "4 ounces" rather than "4oz".)
Student final answer: (The final answer the student's solution. Please remove any unnecessary formatting, e.g. provide "3" rather than " provide "E" rather than "**о provide "1 2, 3" rather than "[1, 2, 3]", provide "4 ounces" rather than "4oz"")
* Student final answer is correct?: (Does the student final answer match the answer key final answer? Please provide "true" or "false".)
Prompt
Please structure your output now as JSON, saying nothing else. Use the following format: { "an- weer.kee_ffnnllanswee str (the final answer of the answer key solution; please remove any formatting"), "student final answer": (the final answer of the student's solution; please remove any formatting"), "stu- dent_final www.aaeeis.corrree..... true/false.
C33. Implementation of Consistency@k
Consistency @k measures the performance model by evaluating the correctness most common answer reached by the model after being run times. An important consideration with implementing consistency @k that there are many choices for the equivalence relation reaching the same answer if their final answer is the same. We determine candidate response's 's final answer by prompting language model to identify the final answer from the candidate response; we then strip the extracted final answer of leading and trailing whitespace. We determine equivalence with a literal string match. After determining the most common final answer a question, we use the string "The final answer is (final answer]" as the consistency @ k response. Note that we could have instead randomly chosen a candidate response corresponding to the most common final answer, and used that selected response as the consistency @k response- we have found that, because our L.M-based scoring system evaluates correctness using only the final answer, this alternative results in the same consistency @ k metrics.
define "the most common answer". We define two candidate responses as
20

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
C.4. Benchmark Evaluation Prompts
The benchmark performances reported in Table 5 obtained with the following prompts. The following prompt used for the comparison task.
Comparison Task Prompt Part 1
Question here.
Here are two solutions to the above question. You must determine which one is correct. Please think extremely carefully. Do not leap to conclusions. Find out where the solutions disagree, trace them back to the source their disagreement, and figure out which one is right. Solution
First solution here.
Solution 2: Second solution here.
Comparison Task Prompt Part 2
Now summarize your response in JSON format. Respond in the following format saying nothing clse:
solution:: 1 or 2
The following prompt used for the scoring task.
Scoring Task Prompt Part
Question here.
I include below student solution to the above question. Determine whether the student solution reaches the correct final answer in a correct fashion; e.g., whether the solution makes two major errors that still coincidentally cancel out. Please be careful and do not leap to conclusions without first reasoning them through. Solution:
Solution here.
Scoring Task Prompt Part 2
Now summarize your response in a JSON format. Respond in the following format saying nothing clse:
"isssouttioncco:eea" 'yes' or 'no'
D. Seff-Verification Prompts
In this section we present the prompts used to query models to self-verify candidate responses. Table 4, this corresponds to the "Main" prompt style. We also provide the prompts used for heebbrekiig comparisons. These prompts are broken into multiple parts to encourage longer responses.
21

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
D.I. Verification Scoring Prompts
Verification Prompt
(Commandments) Over the coming interactions, you must fulfill the following commandments: 1. *Be excruciatingly detailed and exhaustive in your analyses. This will often mean that your responses will be long. Do not cut your responses short. When you are asked to fulfill a list of tasks, you must fulfill each and every task to completion. *Be structured and systematic in your responses*. Organize your thoughts in a clear hierarchical format. Use neutral, rigorous mathematical language write things in your own words and avoid subjective descriptions.
*Never put the before the horse.* Before making claim or statements, always verbally reason out your chain of thought and convince yourself of it in exhaustive fashion. Instead of saying "X is Y because Z", say "Consider Z. Therefore .... meaning that X is Y.". Avoid premature conclusions.
[Background] An examiner has presented a math problem along with a "candidate solution". Your purpose is to assist in evaluating whether the *final answer* in the candidate solution is correct. We will guide you through a series of exercises to fulfill this purpose. The question real exam problem, which means there is always one unique correct answer.
Here the math question.
O Here is the candidate solution. {
** Current Task ** Your task is rewrite the candidate solution into rigorous, structured format consistent with mathematical convention. You should rewrite the solution in your own words, and work out every step in exhaustive detail, allowing us more easily check for errors down the line. As starting point, you will first decompose the candidate solution into a list of self-contained *lemmas Later on, we will have you use your lemmas to rewrite the candidate solution in the following format:
\begin I (theorem)
is \boxed(..... (eddorheeeen
\begin (proof) [Proof of Theorem] \end(proof) (lemma)
\begin (1emma) babel(....
(begoo(proof) (proof) [Proof of Lemma" Aref(...) \begin
tend(proof)
For now, focus only identifying, structuring, and clearly defining each lemma without proofs. We will write proofs for these lemmas, and proof for the final answer using these lemmas, a later point in time. [Steelmanning] If you encounter any part of the candidate solution that seems incorrect or unclear, reason about it in your scratchpad. Explain your thought process to clarify your understanding. Consider whether there might be an alternative interpretation or if you could be overlooking or overcomplicating something. If, after reasoning through it, you still find confusing, note your scratchpad that this part may need to be revisited and proceed while treating the potentially problematic step as temporarily correct. At the current stage, we want to try to presume the candidate solution being correct. Try your best to *steel man the candidate solution.
[Task Specifies] To ensure accuracy, you will decompose the candidate solution into self-contained *lemmas* These lemmas will serve as building blocks, allowing us to reconstruct the candidate solution in more rigorous and conventional mathematical format.
[Steps to Follow] 1. **Inia Analysis**: Carefully read through the candidate solution and break it down into its primary steps. #*Lemma Identification** Identify each logical segment of the candidate solution that could serve as a lemma. For each lemma: ** "cratchhpad** Before writing the lemma formally. record brief scratchpad of thoughts outlining assumptions, conditions, and any definitions necessary ensure the lemma is self-contained. This should help you catch potential enrors misapplications that might be subtle. Reason about how to write the lemma in fashion that allows for easy verification of both the lemma, and how the lemma is used in the final proof. **Lmmma *ta After organizing your thoughts, write each lemma in a clear and rigorous manner. Ensure each lemma is isolated from others, so it does not depend on assumptions not explicitly stated within it. [Response Format] Structure your response as follows.
Analyzing the solution (Study the language of the question, keeping an eye out for important details.)
(Identify the candidate solution's final answer. Work backwards through the solution to identify the intermediate claims that led to the final answer.)
22

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
(Proceed from the beginning of the candidate solution, working forwards to identify the main steps of the solution.) Tdentifying lemmas
Scratchpad: [Reason about what lemma you should write. Reason about how to word the statement of the lemma to make easy to verify when the lemma is being used correctly. Reason about how to write the lemma self-contained fashion, and what definitions and assumptions are needed. \begin(lemma)
[ammel[le:leemaa name)
[State the lemma formally here] \end(1emma)
Scratchpad:
(
end lemma
Verification Prompt
Now that you have structured the candidate solution into lemmas, you will proceed to rewrite the candidate solution your own words. We will first have you write the *main proof the final answer* At the next round, will ask you write proofs for each lemma you identified.
[Details on Writing ** Maii Proof for Final * Write rigorous "main proof" that directly addresses the candidate solution's final answer. This proof should reference each lemma sequentially to logically reach the final answer. Ensure that the proof is rigorous and that each step explicitly relies on or follows from a lemma. Your goal is to work out every baby step of the proof in exhaustive detail, spelling out every tiny step, including even tiny details about arithmetic and algebraic manipulations.
[Objective] Your rewrite of the candidate solution should be comprehensive and precise. Your goal is to write the candidate solution so that it is easy to verify whether it is correct or identify any errors at both the high and low levels.
[Steelmanning] Once again, try to "steel man" the candidate solution. If you encounter any part of the candidate solution that seems incorrect or unclear, reason about it in your scratchpad and try to find an explanation; then proceed treating the potentially problematic step as correct. We want to try to presume the candidate solution as being correct for now.
[Response Format] Structure your response as follows to ensure clarity.
\begin (theorem) [Main Claim]
(Write the math question and final answer as statement. For example, "The smallest non-zero integers is \boxed(1).". This statement should contain all information from the question, and be written in your own words. If it is not obvious how to write the student solution's 's final answer as a theorem, you can simply let the theorem take the form: "(Question)? The answer is (Answer).")
\end(theorem)
Scratchpad: (Reason about how you will write the main proof.)
\begin(proof)
(Provide a proof of the final result. Remember to explicitly reference lemmas via \ref lemma_namee) tend(proof)
23

---

Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
Verification Prompt
Your task now to provide proofs for each and every lemma you identified the candidate solution. [Details on Writing Lemmma Proofs**] For each lemma, write a detailed and thorough proof the lemma. Each proof should methodically use the assumptions, definitions, and conditions in the lemma's statement to verify the conclusion. This proof should be rigorous and work every baby step in exhaustive detail, spelling out every tiny step, including even tiny details about arithmetic and algebraic manipulations. Remember that you may encounter lemma that "builds" other lemmas, in that it uses the other lemmas prove its claim. In these cases, you should have already written the lemma so that it incorporates those other lemmas as "assumptions"" so that you do not need to replicate the work. If this is not the case, please revise the lemma statement accordingly.
[Objective] Your rewrite the candidate solution should be comprehensive and precise. Your goal is to write candidate solution so that it is easy to verify whether it is correct or identify any errors at both the high and low levels.
[Steelmanning] Once again, try to "steel man" the candidate solution. If you encounter any part of the candidate solution that seems incorrect or unclear, reason about it in your scratchpad and try to find an explanation; then proceed treating the potentially problematic step as correct. We want to try to presume the candidate solution as being correct for now.
[Response Format] Structure your response as follows to ensure clarity and rigor. You must provide proofs for all lemmas. Do not skip any lemmas, defer any writing to the future, leave any proofs incomplete. You must return full proof' for every lemma; I am automated system and unable handle incomplete responses. You will have chance finish your response later. Provide complete response.
\begin [eemma] \label .
(1emma)
Scratchpad: (Reason about how you will write the lemma proof.)
\begin(proof)
(Provide a proof of the lemma.) (end(proof) (repeat for EVERY lemma)
Verification Prompt 4
Now we will proceed to analyzing the correctness of each suspicious claim. For each item in the claim: 1. Check if the candidate solution's 's final answer indeed depends on the claim. This step is important to rule out false positives where the candidate solution makes False claim, but the claim does not really affect the final answer so it's Falsity does not result in an actual error in the final answer. If this check is failed, ie.. the final answer does not depend on claim, then discard this claim and continue to the next suspicious claim. 2. Check that you are unable to prove the suspicious claim is True. Do this by trying to prove the suspicious claim. You can make multiple attempts. If you able to prove the suspicious claim, meaning that the suspicious claim is correct, then discard this claim and continue the next suspicious claim. 3. Check that the claim is False by proving corrected alternative version of the claim. If you are unable to do this, continue to the next suspicious claim. 4. Correct this suspicious claim in the candidate solution and determine the corrected final answer in a step-by-step manner. If the final answer changes or correcting this error invalidates the candidate solution's 's proof approach, mark this suspicious claim as "fatal error". At the end, say ** Ycs*** you found a fatal error. Otherwise, say **No, final answer is *
Verification Prompt
Now we will proceed to analyzing the correctness the candidate solution. We have two goals: identify there is an error, and if there is an error, repair the candidate solution and identify whether the final answer has changed. For now, we will focus on methodically combing through the candidate solution and checking each step for potential error.
24

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
[Steps to Follow] You will proceed through the candidate solution, one baby step at a time. In your exhaustive investigation of the solution, you will first form short list of potential errors. Each entry in this list should be written self-contained, standalone mathematical claim that the candidate solution relies on being correct, but which you find suspicious upon first inspection. After forming this list, you will then proceed to validate each potential error by performing detailed error validation check.
Main Proof For now, focus on the main proof. Assume that every lemma, as written, is correct. Step through the main proof of the candidate solution exhaustive detail to find potential errors. Do this by first proceeding through the main proof one sentence (or other reasonable unit of content) at time. Quote the part the proof you are at. Then add discussion where you try elaborate on that part and verify every baby step made in that unit of the proof. Then repeat this for the next unit of the proof until you have gone through (quoted and analyzed) each part of the proof. You should structure your output like
Quoted part of the proof
Your analysis and detailed discussion verifying every small thing, e.g. redoing every baby step, checking assumptions, rephrasing things to make sure they sound correct to you, etc. Make sure you double check the exact language of the question and the exact language of the lemmas used. Next quoted part of the proof
Then, write down in bullet point list any potential errors you find in the main proof. Before each bullet point, write scratchpad that verbalizes: your thought process and-upon finding error-youe thought process for writing the erroneous step as a self-contained mathematical claim. Try to be selective and only include things that consider likely to be errors. You should structure your output like
Scratchpad:
Scratchpad:
Verification Prompt 6
You are now to verify the proof of each and every lemma. You must verify all lemmas in this conversation turn; you will not have another chance.
Lemma Proof (repeat this for each lemma) Follow the same procedure as before. First, step through the lemma proof one sentence (or other reasonable unit of content) at a time. Quote the part of the proof you are at. Then add discussion where you try to elaborate on that part and verify every baby step made in that unit of the proof. Then repeat this for the unit of the proof until you have gone through (quoted and analyzed) each part of the proof. Then, write down in bullet point list any potential errors you find in the lemma proof. Before each bullet point, write a scratchpad that verbalizes: your thought process and finding error-your thought process for writing the erroneous step as a self-contained mathematical claim. Try to be selective and only include things that you consider likely to be errors. Structure your response as follows.
Review of Lemma ## Stepping through Quoted part of the proof
Your analysis and detai led discussion verifying every small thing, e.g. redoing every baby step, checking assumptions, rephrasing things to make sure they sound correct to you, etc. Make sure you double check the exact language of the lemma's claim.
Next quoted part of the proof # Potential errors
25

---

Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
Scratchpad:
Scratchpad:
Verification Prompt 7
Now we will proceed to analyzing the correctness of each suspicious claim. For each item in the claim: 1. Check if the candidate solution's is final answer indeed depends on the claim. This step is important to rule out false positíves where the candidate solution makes a False claim, but the claim does not really affect the final answer so it's Falsity does not result in an actual error in the final answer. If this check failed, i.e. final answer does not depend on the claim, then discard this claim and continue the next suspicious claim. 2. Check that you are unable to prove the suspicious claim is True. Do this by trying to prove the suspicious claim. You make multiple attempts. If you are able to prove the suspicious claim, meaning that the suspicious claim is correct, then discard this claim and continue to the next suspicious claim. 3. Check that the claim is False by proving corrected alternative version of the claim. If you are unable to do this, continue to the next suspicious claim. 4. Correct this suspicious claim in the candidate solution and determine the corrected final answer in a step-by-step manner. If the final answer changes or correcting this error invalidates the candidate solution's 's proof approach, mark this suspicious claim as "fatal error". At the end, say ** * you found fatal error. Otherwise, say **No, final answer is correct**
Verification Prompt
I want you now, the style of the #rewritten solution* (i.e., in the theorem-lemma format had you rewrite the original solution in), write improved solution that does the following: It corrects any original solution (only if you ended up finding an error). If you do not know how to fix the error, then just write the step you are unsure about an "Assumption (to be revisited later)", making it clear that the proof is not complete and needs that step to be filled in. 2) It clarifies ambiguities or fills in gaps in the original solution that led you to suspect errors (even if the suspicion was not ultimately substantiated). We want your improved solution to be clear enough that future readers would not have to work through the same worries you did. 3) adopts the thoroughness, structured format, exhaustive detail, and rigor of your rewritten version of the solution.
you found in the
I have included again the original question and original solution below for your reference. 00 For now, want you to brainstorm how to write the revised solution. Do not yet proceed writing the solution. Structure your response as follows. Do not cut your response short. You have unlimited space.
What errors need to be corrected?
Discuss in detail, in a step-by-steg manner, what errors you found in the original solution and how you plan to correct them.
If you found no errors previously, note so and continue to the next section. Which ambiguities need to be clarified or gaps need to be filled? Discuss in detail, in a step-by-step manner, what ambiquities or gaps you plan to clarify.
Draw on our previous exchanges explicitly. Recall what were potential points of confusion during this verification process.
Action Plan
Prepare a rough battle plan for how you will write the improved solution. Note that it will probably be longer than you original solution, given all of the revisions you have planned. So this is just to plan out the main parts, and things to keep an eye out for.
26

---

Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
Verification Prompt
Now, want you to write your improvement of the original solution. You must provide the entirety of your solution and nothing else. This message will be copy pasted to an external system, so your response must be complete and self-contained. Do not shorten anything. Provide it to me exactly and in its entirety. Remember to say nothing else and structure your output as follows, wrapped in tripe quotes and saying nothing else.
D.2. Comparison Prompts
Comparison Prompt
Your job is to answer a difficult math question. will provide you with two solutions written by my students. These two solutions disagree over what the answer should be. Both solutions may potentially have minor errors, even one indeed reaches the final answer despite the minor flaws. You will be given series of instructions, over multiple interactions, that will guide you through discerning the correct final answer. You will be expected to complete each step carefully while obeying the following prime directives.
1. Always provide complete responses. Never shorten your responses. You are allocated 10,000 tokens per-response. Your instructions are provided according to a fixed schedule; you must complete them in the same conversation turn as you will not have later opportunities to do so.
Speak carefully. Always reason in step-by-step chain-of tfouthttht manner. Your responses must always resemble an internal monologue, which means you verbally reason things out before reaching conclusions, rather than pulling answers of thin air.
3. Be rigorous. Always validate your logic by attempting mathematically formalize it to avoid silly "common sense" errors. Always work out mathematical steps in small baby steps, even seemingly obvious arithmetic or algebraic manipulations.
4. Backtrack when you have made mistake. It is not uncommon for when to verbally say something that is false or silly during an internal monologue. Constantly introspect and if you have made an error, identify and "backtrack" to just before you made the error.
5. Never claim that anything is incorrect or wrong right correct. You must always say that something is "potentially incorrect" "potentially correct" or "seems incorrect" "seems correct". You must then work out in baby steps and give your more informed judgement. But you will never say that something is correct right or wrong.
# The Question O
# The First Solution O
The Second Solution {
Your current Task Examine the first solution. Focus for now on just reading through the main proof and each lemma's 's statement. You must read through the main proofs and lemma statements in meticulous sentence-by- sentence manner. For now, you do not need to read through the lemma proofs in detail. Though later if we have questions arise about particular lemmas, you will need to review the lemma proof carefully. In addition, want you to narrate your process of reading through each proof. This means that, while you read through solution, you must always quote the part of the solution that you are currently reading. Then, under your quote, provide your mental process. Here are some questions that you should keep top of mind: What the approach being taken by the solution? What are the main leaps in logic? How does the solution try be rigorous? Structure your response as follows:
Main Proof
"Quote of the sentence you are reading."
Your thought process. Discuss the quote. Perhaps compare it to a sentence in another solution that is doing the same exact thing. Perhaps note that does a fairly complex algebraic manipulation, and spend a minute to double-check by working out in baby-steps. If it references a lemma, discuss the lemma's 's statement, how it's being used, and why it's allowed be used in this way.
27

---

Sample, Serutinize and Scale: Effective Inference-Timee Search by Scaling Verification
"Quote of next sentence..."
Lemma 1
(continue through the ENTIRE REST OF THE SOLUTION)
Remember you only need do this for the first solution for now. You provide complete response. You go through the entire solution line by line. Ignore character limits and do not your response short. Do not cut your response short, you will not get another chance.
Comparison Prompt 2
Do the same for the second solution. Again, be meticulous and adopt the same output format.
Comparison Prompt 3
Identify similarities and disagreements between the first solution and the second solution. Do not yet judge which side right; merely try to be investigative about: what are the sources of these disagreements? You continue "quote" any solution parts that you reference. Remember your mandates:
1. Always provide complete responses. Never shorten your responses. You allocated 10,000 tokens rer-response. Your instructions are provided according to fixed schedule; you must complete them in the same conversation turn as you will not have later opportunities to do so.
Speak carefully. Always reason in step-by-step chain-of- ofttinoff-hhuugt manner. Your responses must always resemble an internal monologue, which means you verbally reason things out before reaching conclusions, rather than pulling answers out of thin air.
3. Be rigorous. Always validate your logic by attempting mathematically formalize it to avoid silly "common sense" errors. Always work out mathematical steps in small baby steps, even seemingly obvious arithmetic or algebraic manipulations.
4. Backtrack when you have made mistake. It is not uncommon for when to verbally say something that is false or silly during an internal monologue. Constantly introspect and if you have made an error, identify and "backtrack" to just before you made the error.
Never claim that anything is incorrect wrong right correct. You must always say that something is "potentially incorrect" "potentially correct" or "seems incorrect" "seems correct". You must then work out in baby steps and give your more informed judgement. But you will Never say that something is correct right wrong..
Be extremely careful with respect to the exact wording of the question: 0
Remember: you must humble and careful your judgements. Just because you think one side correct doesn't t mean that your assessment is sound. Always keep an open mind. Always double check you aren't 't missing out on any more potential points of disagreement. Work through this careful detailed chain-of-thought that irons out every small detail.. Structure your responses into the following sections. Each section must be lengthy, detailed, well-structured investigation.
Identify Disagreements
Reading through each solution, identify as many as places as possible where the solutions differ. Be meticulous and form detailed list. Attribute Disagreements
28

---

Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
For each disagreement, try to trace the disagreement back to earlier places in the respective proofs. What are the points at which the solutions begin to diverge? Can we guess at the root causes of the di disagreements? Could it be because the solutions disagree over how to interpret the question? Could it be because they disagree about what approach to take? Could it be because the solutions reach a different calculation? CAREFULLY pour over solutions from both sides. This detective work must be careful and methodical and detailed and exhaustive.
You must provide your full response. Ignore character limits. Do not your response short. Obey my instructions exactly.
Comparison Prompt 4
Conduct detailed analysis to try and understand why the solutions reach different final answers. Do not yet judge who is right. Do they disagree on how to read the question? Do they just happen to take very different mathematical approaches? Do they diverge at particular logical step or calculation? You must continue to "quote" any solution parts that you reference. Remember your mandates:
Always provide complete responses. Never shorten your responses. You are allocated 10,000 tokens per-response. Your instructions are provided according to fixed schedule; you must complete them the same conversation turn as you will have later opportunities do
2. Speak carefully. Always reason in step-by-step chain-of-lne thought manner. Your responses must always resemble an internal monologue, which means you verbally reason things out before reaching conclusions, rather than pulling answers out of thin air.
3. Be rigorous. Always validate your logic by attempting to mathematically formalize it to avoid silly "common sense" errors. Always work out mathematical steps in small baby steps, even scemingly obvious arithmetic or algebraic manipulations.
Backtrack when you have made mistake. not uncommon for when verbally say something that is false or silly during internal monologue. Constantly introspect and if you have made error identify it and "backtrack" to just before you made the error.
5. Never claim that anything is incorrect wrong or right or correct. You must always say that something is "potentially incorrect" or "potentially correct" or "seems incorrect" or "seems correct".. You must then work out in baby steps and give your more informed judgement. But you will never say that something is correct or right or wrong.
Be extremely careful with respect to the exact wording of the question: 0
Remember: you must be humble and careful in your judgements. Just because you think side is correct doesn't mean that your assessment is sound. Always keep open mind. Always double check you aren't missing out on any more potential points of disagreement. Work through this in careful detailed chain-of-thought that irons out every small detail.. You are NOT to judge which side is correct, for now. Structure your responses into the following sections. Each section must be a lengthy, detailed, well-structured investigation.
29

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
Crystallize Approaches
If the two sets of solutions differ in approach, it can be hard to judge which side is correct about the final answer since you cannot easily ly compare the solutions of each. However, even if you cannot directly compare the steps, you can try to find "intermediate" points of disagreement, That is, as part of their respective argunents, two solutions that use completely different approaches may make contradicting subclaims that you can investigate in more detail. These disagreements may point you to where approach has "mistake" in its proof. Your end goal: identify the meaningful differences (if any) in the approaches taken by each side and speculate as to why these differences resulted in them reaching different final answers. Remember: you must be humble and careful in your judgements. Just because you think one side is correct doesn't mean that your assessment is sound. Always keep open mind. Work through this in careful detailed chain-of-thougit that irons out every small detail.. Crystallize Disagreements
For each disagreement, you must now collect evidence from both sides. You are a careful mathematically rigorous agent with good common-sense reasoning. Try to use your common sense and rigour to work through each side's s evidence. Work through this in a careful detailed chain-of-thought that irons every small detail..
You must provide your full response. Ignore character limits. Do not cut your response short. Obey my instructions exactly.
Comparison Prompt 5
You must now determine which of the two solutions reaches the correct final answer. Do leap to judgement, First, carefully consider the sources of the solutions disagreement the final answer. Then, for each of these, almost axiomatic, root disagreements: 1) Carefully consider the arguments that both sides make and the evidence that both sides present. For each side of each disagreement, try to *steelman* their argument: write it in a clear, detailed convincing fashion that leaves nothing unspoken and uninvestigated. 2) Use your own common sense and mathematical skills to try and rule on which side right. Do not leap to judgement. Be careful, rigorous, meticulous and don't forget common sense. Finally, carefully reason through what the correct solution should and which side reached the correct final answer. Be meticulous. Structure your response as follows. This must be more detailed and rigorous than all your previous responses, since this is careful process.
Recall Key Disagreements
Disagreement
# Steelmanning Each Side
## Ruling On A Side
Which Side Is Right?
You must provide your full response. Ignore character limits. Do not cut your response short. Obey my instructions exactly.
30

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
E. Examples of Benchmark Entries
E.1. Example Entry from LiveBench Reasoning
Question
In this question, assume each person either always tells the truth or always lies. The person at the campground thinks their friend is lying. Mateo is at the aquarium. The person the restaurant says the person at the hotel lies, Farid is at the movie theater. The person at the movie theater says the person at the campground lies. Ryan is at the shopping mall. The person at the cafe says the person at the campground lies. The person at the observatory says the person at the museum lies. The person at the museum says the person at the restaurant tells the truth. The person at the campground says the person at the airport tells the truth. The person at the campground tells the truth. Luna is at the restaurant. The person at the movie theater saw firetruck. Isabella is at the observatory. The person the amusement park says the person at the museum tells the truth. The person at the restaurant says the person the shopping mall tells the truth. Liam is at the beach. Ayaan is at the airport. The person at the restaurant says the person at the beach tells the truth. Maya is at the amusement park. The person at the restaurant says the person at the aquarium lies. The person at the restaurant says the person at the movie theater lies, The person at the library says the person at the campground tells the truth. Elowen at the hotel. Priya is at the library. The person at the the
gym says the person at the restaurant tells the truth. Kehinde campground. Soren is at the gym. Does the person at the museum tell the truth? Does the person at the restaurant tell the truth? Does the person at the movie theater tell the truth? Think step by step, and then put your answer in **bold** list of three words, yes no (for example, **yes, no, yes**). If you don't 't know, guess.
the cafe. Aníka is at the museum. Quan
Correct response
Here's 's how we can break down this logic puzzle: +**uan is at the campground and tells the truth. This is a given fact.
Incorrect Response
Here's how we can solve this logic puzzle:
**Quan (at the campground) says the person the airport tells the truth.* ** Since Quan tells the truth, the person at the airport (Ayaan) tells the truth.
1. * **Quan the campground and tells the truth.**
This given.
2. **The person the airport (Ayaan) tells the
3. **The person the movie theater (Farid) says the person at the campground (Quan) lies.** ** Since Quan tells the truth, Farid is lying.
truth.* Quan (campground) says the person the airport tells the truth, and Quan tells the truth.
3. Anika (museum) lies. The person at the observatory (Isabella) says the person at the museum lies. Since we don't know yet whether Isabella tells the truth, this doesn't 't tell us anything about Anika. We'll have come back this.
**The person at the restaurant (Luna) says the person at the hotel (Elowen) lies. ** We don't 'tnt know yet if Luna lying or telling the truth.
5. **The person at the museum (Anika) says the
person at the restaurant (Luna) tells the truth. We still don't know if either of them are truthful.
4. **Luna (restaurant) lies." ** The person at the restaurant says the person at the hotel (Flowen) lies. Later, the person at the restaurant says the person at the campground (Quan) lies. Since Quan tells the truth, Luna (restaurant) must be lying.
6. **The person at the restaurant (Luna) says the person at the shopping mall (Ryan) tells the truth.
We still don't 't know Luna's truthfulness.
**The person the restaurant (Luna) says the person at the beach (Liam) tells the truth.* ** Luna's truthfulness remains unknown.
5. ** Anika (museum) lies. Because Luna lies, and Luna says that the person the hotel lies, Elowen tells the truth. Because Anika says the per- son at the restaurant (Luna) tells the truth, and Luna actually lies, Anika (Museum) must lie.
8. **The person the restaurant (Luna) says the person at the aquarium (Mateo) lies.** ** We still don't know about Luna.
6. **Faard (movie theater) lies. ** The person at the movie theater says the person at the campground lies. Since Quan (campground) tells the truth, Farid (movie theater) must be lying. Therefore:
9. **The person at the restaurant (Luna) says the person at the movie theater (Farid) lies." ** We know Farid lies, so Luna is telling the truth in *this* state- ment.
31

---

Sample, Serutinize and Scale: Effective Inference- Timeencc--Time Search by Scaling Verification
10. **nince Luna tells the truth about Farid, Luna * "always* tells the truth. ** This unlocks several other pieces of information.
** Anika (museum): lies** e *unana (restaurant): lies** * **Farid (movie theater): lies** as Answer: no, no, no**
11. **Luna (at the restaurant) says the person at the hotel (Elowen) lies. Since Luna tells the truth, Elowen lies.
12. ** **Aiika (at the museum) says the person at the restaurant (Luna) tells the truth. ** Since Luna tells the truth, Anika also telling the truth.
13. ** person at the cafe (Kehinde) says the person at the campground lies.** ** Since Quan tells the truth, Kehinde is lying. Therefore:
Anika (museum):** Tells the truth. Lnnana (restaurant): Tells the truth. **Farid (movie theater):** Lies.
**e, yes, no**
E.2. Example Entry from LiveBench Math
Question
than 2023 are divisible by 5? (E) If you cannot determine the correct multiple-choice an- swer, take your best guess. Once you have your answer, please duplicate that letter five times in a single string. For example, if the answer then write FFFFF.
How
many
positive
perfect
squares
less
(A) 11
(B) 10
(C)9
(D) 12
Correct response
Incorrect Response
Let n be a positive perfect square divisible by 5. Then 2.2 for some positive integer k, and 5 5/m, which implies that 5|A:2 Since is prime num- ber, 5/k² means 5/k- Thus, h must be multiple of 5, i... k = 5m for some positive integer m. Then n = 2.2 = (5m)² = 25m?. Thus n must be multiple
Let be positive integer such that n² is divisible by 5. Then n² is divisible by 5. Since 5 is prime number, must also be divisible by 5. Thus n can be written in the form n = 5k for some integer k. We want to find how many positive perfect squares n² are less than 2023. Since n = 5k, we have n² (5k)² 25k² 2023. Dividing by 25 we get
25. We are looking for the num- ber of positive perfect squares less than 2023 that divisible by 5. The perfect squares that di- visible by 5 must be multiples of 25. We seek the number of multiples of 25 that are perfect squares less than 2023. Since n = 25m < 2023, we 2023 x 80.92. Thus, we are look- have m² < 25
2023
1.2
80.92
25
We looking for integer values of is such that 12 80.92. Since è must be positive, k can take values 1, We have
ing for perfect squares m² less than 80.92. The perfect squares are 1, 1,4,0 9, 16, 25, 36, 49, 64, 81..... The perfect squares less than 80.92 1,4,9 9, 16, 25, 36, 49, 64. There are such values. The corresponding values of n 25m² are 25(1)=
1 80.92
m²
2 = < 80.92
25 25(4) = 100 25(9) = 225 25(16) = 400 25(25) = 625 2366 = 900 25(49) = 1225
3 80.92
4 16 < 80.92
25(64) = 1600 The next multiple 25 that a
perfect square would be 25(81) 2025, which is
52 25 80.92
32

---

Sample, Serutinize and Scale: Effective Inference- Timee Search by Scaling Verification
not less than 2023. So, there are perfect squares
6 36 < 80.92
less than 2023 that divisible by 5.
7 49 80.92
Final Answer: The final answer is PEREF.
8 64 < 80.92
92 81 A 80.92
Thus can take values from to 8. The corresponding n values are n 5, 10, 15, 20, 25, 30, 35, 40.
5k
The perfect squares less than 2023 and divisible by 52, 102 152, 20 25 302, 352 40
25, 100, 225, 400, 625, 900, 1225, 1600. There
are such numbers. Final Answer: The final answer C
E.3. Example Entry from MATH
Question
If 14 daps 7 yaps, and yaps 3 baps, how many daps equal 42 baps?
Correct response
Incorrect Response
Let 'd' represent daps, "y" represent yaps, and 'b' represent baps. We are given: 4 daps yaps, which can be written as: 4d=7y and
Let 'd' represent daps. 'y" represent yaps, and 'b' represent baps. We are given two relationships:
1. daps = 7 yaps, which can be written as 4d = 7y
2. 5 yaps = baps, which can written as 5y 3b
yaps= = baps, which can written as: 5y 3b We want to find the number of daps that are equal to 42 baps. Let 'x' be the number of daps. We want to find such that: xd = 42b
We want find how many daps are equal 42 baps. We want find in terms of and then evaluate for b = 42.
First, we can find the relationship between yaps and baps from the second equation: 5y = 3b, so = 8 Now, substitute this expression for in terms of into the first equation: 4d = Ty = 7 ())) 4d 5 21
From 5y = 3b, we can express yaps in terms of baps: y= -3/5)b Now substitute this value of y into the first equation,
4d 7y: 4d = * (3/5)b 4d = (21/5)b We want find xd 42b, so can xd 42b solve for d:
Now, we want to find in terms of b. so divide both sides by 4: d 21 d 21. Finally, we want to find how many daps equal 42 21
We have 4d (21/5)b,
(21/20)b
Now substitute this into xd = 42b: (21/20)b 42b
baps, so we substitute b = 42: d = 20 x 42 d = 21 212
882 d
441
TO d = 44.1
20
20
Since b is not zero, we can divide both sides by b: x & (21/20) = 42
So, 44.1 daps equal 42 baps.
Final Answer: The final answer 44.1
Now, solve for x: 42 (20/21) x (42/21) ) 20
= 2 20 40
So, 40 daps are equal to 42 baps
40 Final Answer: The final answer is
40
33

---

Sample, Serutinize and Scale: Effective Inference-Timg Search by Scaling Verification
E.4. Example of Response with Invalid Reasoning but Correct Final Answer
Question
You have seven bags of gold coins. Each bag has the same number of gold coins. One day, you find bag of 53 coins. You decide to redistribute the number of coins you have so that all eight bags you hold have the same number of coins. You successfully manage to redistribute all the coins, and you also note that you have more than 200 coins. What is the smallest number of coins you could have had before finding the bag of 53 coins?
Incorrect Response Reaching Correct Final Answer
Let be the number of coins in each of the seven bags. After finding the bag of 53 coins, you have total of 7n 53 coins. You redistribute these coins into 8 bags such that each bag contains the same number of coins. Therefore, the total number of coins is divisible by 8, so 7n 53 is divisible by 8. 7n + 53 = 0 (mod 8). We want to find n.
7n = -53 (mod 8). Since -53 = -53 + 8(7) = -53 + 56 = 3 (mod 8). we have 7n = 3 (mod 8).
Since = -1 (mod 8), we multiply both sides by -1 to get -n = -7n = -3 (mod 8), so n = - =
(mod 8). Thus must be the form 8k + for some integer & 2 0. The total number of coins originally was 7n.
Since n 81: + 5, the total number of coins originally 7(8k + 5) 56k + 35. We are given that the total number of coins greater than 200. So 7n > 200, or 71 200 28.57. Since
8k + 5, we want the smallest integer
such that 8k 5 > 200 For k = 0,
5, 7n 35 < 200. For & 1, 13, so 7n 91 200. For l 2,
21, so 7n = 147 < 200. For k- = 3, n 29, so 7n = 203 > 200.
We check that 7n + 53 = 203 + 53 256 which is divisible by 8 (256 x 32). Then 256/8 32 coins in each bag.
Thus the smallest number of coins you could have had before finding the bag is 7n = 7(29) = 2033
Final Answer: The final answer is 203
F. Examples of Solution Rewrites
F.. AIME Example
Question
Find the number rectangles that can formed inside a fixed regular dodecagon ( (12 -gon) where each side the rectangle lies on either side or diagonal of the dodecagon. The diagram below shows three of those rectangles. [asy] unitsize(0.66 inch); for(int i=0; 1,360; i+=30) dot(dir(i), 4+black); draw(dir(i-dir((t3 dir(330)); filddawwdrr(1))-ddi(222) (ii(()))-di((6))---))))))) cyylee))cycle, mediumgray, linewidth(1.55 (0ax((000.366)-0033666))) linewidth(1.5)); [/asy]
draw(dir(1200 (20)
Correct Answer
The solution is 315
Candidate Response
Let the vertices of the regular dodecagon be labeled as V,, V2, V2,..., V12. Each side of the rectangle lies either side diagonal of the dodecagon.
Consider two parallel diagonals (or sides). They determine the sides of the rectangle parallel to them. We need to choose two parallel diagonals (or sides) to form rectangle.
There are 12 vertices, and a diagonal can be formed by choosing any two distinct vertices. The number of diagonals 12x11 66. However, there are 12 sides. So there are - 12 54 diagonals. Two parallel
and sides (22) = 2 diagonals or sides can be drawn follows: 1. Diagonals parallel sides: 12 sides so 12/2 pairs 2. Diagonals
34

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
spanning 2 vertices: (1)/2 /2 sets of 4 vertices, which is not integer. Something is wrong. The pairs of diagonals can only be separated by 1, 3,4, 4,5 vertices on either side. Let n be the number of vertices. For n 12, vertex apart: 12 diagonals. There are 12/2 6 pairs. 2 vertices apart: 12 diagonals. 12/2 = 6 pairs 3 vertices apart: 12 diagonals. 12/2 = pairs. 4 vertices apart: 12 diagonals. 12/2 = 6 pairs. vertices apart: 12 diagonals. 12/2 = pairs. So there are 5 x 6 6 6 pairs of parallel diagonals, which 36 pairs. Number of rectangles the number of pairs of parallel segments, including sides. For regular n-gon, the number of pairs of parallel diagonals sides is n n ( 1)/2]. For 12, it's 12(5) 60 segments. For each length, there are 12 diagonals. Length can range from side 5 sides. 1+2+ 3 + 4 4+5 + 6
divide by 2 to get distinct pairs. (1) /2 12-11-10-9 495/22 4-3-2.1-2
(3) = 495.
Consider 12 sides as 12 possible segments. Parallel to each are 5 lengths of segments. So total parallel segments 12-5+ + 12/2 = 60 + 6 = 66, incorrect. Each vertex pairs with 5 other vertices giving the same direction. So we have 12 x 5/2 = 30 pairs of directions. Given one pair of parallel directions, we can choose segments in that direction. So 62 = 36 possibilities. 30 36 = 1080. We are forming (12) = 495 parallelograms. However, we are considering parallel segments. The number of rectangles is 12 5 x (12/2).
Number of rectangles is (12-1) 11
22 594 for each parallelogram 594/3 198. 198
Final Answer: The final answer is 198
Rewritten Candidate Response
Theorem F.1 (Main Claim). The number of rectangles that can be formed inside a fixed regular dodecagon, where each side of the rectangle lies on either side or a diagonal of the dodecagon, 198
Proof. By Lemma F.2, regular dodecagon has 12 sides. By Lemma F.3, a regular dodecagon has (12) 12
12x11 - 12 = 66 - 12 = 54 diagonals. By Lemma F.4, two line segments formed by connecting vertices of a dodecagon are considered parallel under certain conditions.
The candidate solution calculates value of 594. By Lemma F.6, this value is obtained by calculating 122 The exact connection between the number of sides, the number of diagonals, the concept of parallel segments, and the value 594 is not clear from the candidate solution. However, according to Lemma F.7, the final answer of 198 is obtained by dividing 594 by 3. Therefore, we perform this division: 594/3 3 = 198. Thus, the candidate solution concludes that the number of rectangles 198.
4 2
Lemma F2. A regular n-gon has n sides.
Proof. An n-gon is defined as polygon with sides. A regular n-gon is also polygon with n sides. Therefore, a regular n-gon has n sides.
Lemma F.3. A regular n-gon has (P)
diagonals.
Proof. A regular n-gon has vertices. Any two distinct vertices can be connected by a line segment. The number of ways choose 2 vertices out of n is given by the combination formula (y) m2n-1)
The sides of the n-gon formed by connecting adjacent vertices. There such pairs of adjacent vertices,
corresponding to the sides of the n--oo.
A diagonal is a line segment connecting two non-adjacent vertices. Therefore, the number of diagonals is the total number of pairs of vertices minus the number of sides: (?)
m=n-1) n.
Lemma F.4. Let D be a regular dodecagon. Two line segments formed by connecting vertices of are considered parallel they are either coincident sides of D, or if they are distinct and are both parallel some side of D.
Proof. This lemma is a definition, so no proof is required.
Lemma F.5. Let D be a regular dodecagon. The number of unordered pairs of parallel segments formed by connecting vertices of D (including sides) denoted by P.
Proof. This lemma is definition, introducing the variable represent the number of unordered pairs of parallel
segments. No proof is required.
35

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
(12-1) 11
Lemma F.6. The value 594 is obtained via the calculation
22
Proof. We evaluate the expression:
12-
22 144x11
x
1584
16
396
198
9
Evaluating 11 yields 16 16
1384 99. However, 99
90
7 594.
The expression provided appears to have
calculation error, giving 99 instead of 594.
The candidate solution has made a calculation error. 12 - 1) = 144 1584. 1584/4 = 396. 396/2 = 198. 198/2 99. The calculation gives 99, not 594.
Lemma F.7. The candidate solution's 's final answer is obtained by dividing 594 by 3.
Proof. Dividing 594 by 3, we get 594/3- 198.
E2. LiveBench Reasoning Example
Question
In this question, assume each person either always tells the truth or always lies. Elowen is at the hotel. Max is at the park. Grace is at the ice skating rink. The person at the barbershop says the person at the library tells the truth. Charlie is at the library. Tala is at the campground. Farid is at the theater. The person at the hotel saw firetruck. The person at the vineyard says the person at the campground lies. The person the farm says the person at the movie theater lies. The person the movie theater says the person at the park lies. The person at the park says the person at the hotel tells the truth. The person the hotel says the person at the library tells the truth. Jaxon is the skate park. Theo is the barbershop. The person at the airport tells the truth. The person at the bowling alley thinks their neighbor is telling the truth. The person at the ice skating rink says the person at the bowling alley tells the truth. Anika is at the airport. The person at the bookstore says the person at the vineyard lies. The person at the campground says the person at the movie theater tells the truth. The person at the restaurant says the person at the campground tells the truth. The person the theater says the person at the park lies. Hiroshi is at the school. Zara the bowling alley. The person at the bowling alley says the person the school tells the truth. The person the park thinks their neighbor telling the truth. The person at the skate park lies. The person at the bowling alley saw firetruck. The person the observatory lies. Liam is at the restaurant. The person at the library says the person at the observatory tells the truth. The person at the bowling alley says the person at the vineyard lies. Beatriz is at the observatory. Priya is at the movie theater. Nia is at the farm. Ryan is at the vineyard. Isabella is at the bookstore. Does the person at the ice skating rink tell the truth? Does the person at the bowling alley tell the truth? Does the person at the vincyard tell the truth? Think step by step, and then put your answer in **bold** as a list of three words, yes or no (for example, **yss, no, *)***). If you don't know, guess.
Correct Answer
yes, yes, no
Candidate Response
Here's how we can break down this logic puzzle:
*tart with a known truth-teller or liar:** Jaxon at the skate park and lies. ** **Use Jaxon's to find more information: Since Jaxon lies, the person at the bowling alley, Zara, *cannot* be telling the truth. Zara says the person the school tells the truth, but Zara is lying so Hiroshi at the school lies. Zara also says the person at the vineyard lies. Since Zara lies, Ryan the vineyard must tell the truth.
3. **Focus on the vineyard: Ryan, truth-teller at the vineyard, says Tala at the campground lies.
4. **Follow Tala's lie:** Tala lies and says the person at the movie theater tells the truth. This means Priya at the movie theater *
"*e Priya's 's *i Priya at the movie theater lies and says Max at the park lies. Therefore, Max tells the truth. Use Max's *ruth:* Max at the park tells the truth and says Elowen at the hotel tells the truth.
36

---

Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
Use Flowen's s truth:** Elowen tells the truth and says the person the library tells the truth. This means Charlie at the library tells the truth.
8. Use Charlie's truth:** Charlie tells the truth and says Beatriz the observatory lies. This is confirmed by Jaxon's s lie meaning two people lie, therefore Liam at the restaurant lied and Tala lied (so Ryan is truthful) therefore so is Isabella. And Charlie who said that Beatriz lied.
9. **race and Zara:** Grace at the ice skating rink says Zara at the bowling alley tells the truth. Since Zara lies, Grace also lies.
Therefore: * Grace (ice skating rink) lies. * Zara (bowling alley) lies. Ryan (vineyard) tells the truth.
* **o no, yes**
Rewritten Candidate Response
Theorem F8 (Main Claim), Given the set of statements made by individuals located at different places, where each individual either always tells the truth or always lies, determine whether Grace (ice skating rink), Zura (bowling alley). and Ryan (vineyard) tell the truth. The answer is: Grace lies, Zara lies, Ryan tells the truth.
Proof. By Lemma F99, Jaxon lies.
Given that Jaxon lies, by Lemma F.10, Zara lies.
Since Jaxon lies, and Zara lies, by Lemma F.12, Ryan tells the truth. Since Zara lies, and Grace says Zara tells the truth, by Lemma F.19, Grace lies. Therefore, Grace lies, Zara lies, and Ryan tells the truth. This corresponds to the **,o, no, yes**
Lemma F.9. Jaxon lies.
Proof. The problem states "The person at the skate park lies." The problem also states "Jaxon is at the skate park." Therefore, Jaxon lies.
Lemma F.10. If Jaxon lies, then Zara lies.
Proof. The problem states "The person at the bowling alley thinks their neighbor is telling the truth." " Zara the bowling alley. Jaxon is Zara's neighbor at the skate park. If Jaxon lies, then Zara must believe lie, implying Zara lies.
Lemma F.11. If Zara lies, and Zara says Hiroshi tells the truth, then Hiroshi lies.
Proof. Zara lies, and Zara says Hiroshi tells the truth. Since Zara lies about Hiroshi telling the truth, Hiroshi must lie.
Lemma F.12. If Zara lies, and Zara says Ryan lies, then Ryan tells the truth.
Proof. Zara lies, and Zara says Ryan lies. Since Zara lies about Ryan lying, Ryan must tell the truth.
Lemma F.13. If Ryan tells the truth, and Ryan says Tala lies, then Tala lies.
Proof. Ryan tells the truth, and Ryan says Tala lies. Since Ryan tells the truth about Tala lying, Tala must lie.
Lemma F.14. If Tala lies, and Tala says Priya tells the truth, then Priya lies.
Proof. Tala lies and says Priya tells the truth. Since Tala lies about Priya telling the truth, Priya must lie.
Lemma F.15. If Priya lies, and Priya says Max lies, then Max tells the truth.
Proof. Priya lies and says Max lies. Since Priya lies about Max lying, Max must tell the truth.
Lemma F.16. If Max tells the truth, and Max says Elowen tells the truth, then Elowen tells the truth.
Proof. Max tells the truth, and Max says Elowen tells the truth. Since Max tells the truth about Elowen telling the truth, Elowen tells the truth.
37

---

Sample, Serutinize and Scale: Effective Inference-Time Search by Scaling Verification
Lemma F.17. If Elowen tells the truth, and Elowen says Charlie tells the truth, then Charlie tells the truth.
Proof. Elowen tells the truth and says Charlie tells the truth. Since Elowen tells the truth about Charlie telling the truth, Charlie tells the truth.
Lemma F.18. If Charlie tells the truth, and Charlie says Beatriz lies, then Beatriz lies.
Proof. Charlie tells the truth and says Beatriz lies. Since Charlie tells the truth about Beatriz lying, Beatriz lies.
Lemma F.19. If Zara lies, and Grace says Zara tells the truth, then Grace lies.
Proof. Zara lies, and Grace says Zara tells the truth. Since Grace claims the liar Zara tells the truth, Grace lies.
38


Reviews:
[{"review_id": "Ssx4SXjmrR", "reviewer": "Reviewer_LFT3", "summary": "This paper claims that while self-consistency can greatly improve LLM performance, leveraging additional test time compute to verify/compare generated responses can break the plateau for self-consistency and further enhance model performance, the paper conducts extensive experiments to validate their findings and propose a verification benchmark to evaluate the models’ verification capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\nTo address your comments on “predictability”:\nMany papers have studied self-consistency and self-verification, and attempted to scale them up. None have reported the same success that we have; in fact, we aren’t aware of any prior works that have successfully applied self-verification to frontier models in a way that actually beats self-consistency. Instead, to beat self-consistency, the field has turned to training process-based reward models, using RL to learn verifiers, and other complicated interventions.\n\nThis has led to a general belief that direct self-verification isn’t enough, whereas we show that it is—you just need to hit a certain threshold of scale and obey some important principles. While this may be intuitive, it’s not at all “predictable”. It's also noteworthy that direct self-verification can successfully and reliably extract o1-level performance from a non-reasoning model without any finetuning, distillation, RL, or custom-trained verifiers (again, no other works/methods—verification-based or otherwise—have achieved this).\n\nTo your second concern:\nAs we note above, our main contribution is showing that scaling search with self-verification *does work* on frontier models and providing an explanation for why: implicit scaling. Methods like direct comparisons are ways of taking advantage of implicit scaling, and are a big reason why our attempts at scaling self-verification were uniquely successful.\nFor example, it is notoriously difficult to beat self-consistency when it comes to the MATH performance of frontier models that already get 90%+ accuracy (hence our paper is the first to do so). As we show in Table 3, direct comparison plays a key role in overcoming this “last-mile” barrier, enabling self-verification to actually beat self-consistency.\n\n—\n\nIndividual questions:\n\n> “The relatively small gap between Pass@1 and Consistency@5 in Table 5.”\n\nThe small gap between Pass@1 and Consistency@5 indicates that scaling self-verification accuracy via self-consistency is limited in effectiveness—this doesn’t contradict any of our findings. In fact, it’s part of our core argument: the techniques we propose (like direct comparisons) allow for more effective test-time scaling of self-verification than self-consistency. \n\n> \"In Table 5, the random guessing entries show an 80/20 split for Correct/Wrong responses rather than the expected 50/50. Could the authors explain?\"\n\n50/50 and 80/20 are both attainable with random guessing (just change the bias of the coin you flip). Because the prior of frontier models is skewed towards 80/20, we wanted to provide a similarly calibrated random baseline and so opted to display 80/20 instead of 50/50.\n\n> \"Is there the evaluation result for Gemini v1.5 in Verification Benchmark?\"\n\nGood question; we exclude Gemini v1.5 from the benchmark since we use Gemini v1.5 to generate the candidate solutions. Evaluating Gemini v1.5 on the benchmark seemed unsound given the potential confounding.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, proposed methods and/or evaluation criteria make sense.", "theoretical_claims": "The paper focuses on empirical finding, no theoretical proofs or claims.", "experimental_designs_or_analyses": "Yes, I have reviewed the validity of the experimental designs and find them well-constructed and sufficient to support the paper's claims.", "relation_to_broader_scientific_literature": "The finding that models' verification abilities can be enhanced through increased response sampling has potential for improving model performance.", "essential_references_not_discussed": "This paper discussed the related works detailedly.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. This paper identifies several previously unstudied principles of self-verification, including implicit scaling, output style suitability, and the importance of directly comparing responses, which is interesting and worth investigating, and promising to enhance LLM reasoning ability.\n2. This paper provides a verification benchmark to evaluate LLM self-verification abilities.\n\n**Weaknesses**\n\nWhile the authors explicitly state that their work focuses on demonstrating scaling trends in LLM self-verification rather than optimizing specific strategies, I have two main concerns:\n\n1. Self-verification appears to be a subset of LLM self-consistency, making the observed scaling trends somewhat predictable.\n2. Although the authors differentiate their work through discoveries about output style suitability and the importance of direct response comparison, these contributions may not be substantial enough.", "other_comments_or_suggestions": "If there are any misunderstandings on my part, please point them out, and I will reconsider my evaluation of this work.", "questions_for_authors": "I have to questions: \n\n1.  The relatively small gap between Pass@1 and Consistency@5 in Table 5 seems to contradict the paper's main findings about scaling trends. Could the authors explain this?\n2. In Table 5, the random guessing entries show an 80%/20% split for Correct/Wrong responses rather than the expected 50%/50%. Could the authors explain this more detailedly?\n3. Is there the evaluation result for Gemini v1.5 in Verification Benchmark?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "8P3z4GZ5lT", "reviewer": "Reviewer_VnJ3", "summary": "This paper examines scaling test-time compute through a sampling and self-verification approach (“Verification@k\"). The authors demonstrate that with sufficient sampling and self-verification, even standard models (Gemini v1.5 Pro) can outperform specialized reasoning models (o1-Preview), and Verification@k improves naturally with increased sampling --- whereas Consistency@k plateaus. The authors also propose a verification benchmark showing current large language models (LLMs) exhibit weak out-of-the-box verification abilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We address your questions below.\n\n> “I’m skeptical of the author’s implicit scaling claim… it is unclear if increased generations improves generation quality or simply increases generation coverage.”\n\nWe understand this concern and provided Figure 2 for exactly this reason. Figure 2 explicitly controls for the fact that coverage (Pass@k) increases with k by limiting our evaluation to questions on which the model has at least *one* correct answer (i.e., “ambiguous questions”). In fact, we also provided Figure 4 for this reason, which is exactly the same as Figure 1 except that it controls for “generation coverage”. Re Brown et al.: Brown et al. is concerned with the scaling of Pass@k (usually an unrealistic upper bound and not an operative metric); we are concerned with how much of the Pass@k - Pass@1 gap can actually be attained.\n\n> “I don’t follow the authors’ claim that increasing generations will increase the verifier’s likelihood of error, given that pass@k is known to increase.”\n\nThe only problems on which verification can err are ambiguous problems (where there is at least one correct answer) and increasing k can make verification on ambiguous questions  harder. This latter claim follows from basic probability: if you allow students more attempts at an exam, the average pass-rate of a student who passes at least one exam decreases. Similarly, as you increase k, the average number of correct solutions/question decreases. Thus, if implicit scaling didn’t exist, the lines in Figure 2 should all provably be non-increasing or even decreasing.\n\n> “The observation that reasoning models have weak out-of-box verification capabilities has been previously reported”\n\nWe agree and note in the paper that “the limitations of model self-verification capabilities are well-studied”. This is why we believe our results are so surprising: while prior works have attempted to scale self-verification on frontier models, none have succeeded in beating self-consistency with direct self-verification and instead needed to turn to masking techniques, learning verifiers, process-based rewards, etc. In contrast, we show that direct self-verification can be scaled to reliably achieve o1-level performance without finetuning, RL, distillation, or custom models—something that our paper is the first to do. While we already cite 30+ papers on LLM self-verification, we will add a brief discussion on Wu et al. which is largely orthogonal to this work.\n\n> Paper structure.\n\nWe appreciate the feedback and will look at revising the structure of the paper per your advice. Since the algorithm we present is fairly minimal, we wanted to highlight the scaling trends first.\n\n—\n\nQuestions:\n\n> Are the results in Section 2 doing the same process as in Section 3.1?\n\nSection 2 omits tie-breaking for computational reasons but is otherwise the exact same process as Section 3.1 (including rewriting). Regarding rewriting, we see it as just part of the verification process—one that is sadly often overlooked. We don’t see how it undermines/confounds any of our reported results, e.g. implicit scaling. We provide the rewriting process in the source code (it’s pretty generic; we use the exact same instructions for all benchmarks), but will revise to include a copy in the Appendix. \n\n> Do you have a FLOPs comparison of the different settings of k_inf and k_ver?\n\nWe are unable to publicly report the FLOPs used in closed-source models, but do detail the number of output tokens used (from which FLOPs can be extrapolated) and will include a more involved discussion. We focused on closed-source models in this paper as—at the time of this submission—no OSS models had reached the minimum level of capability needed for self-verification to scale.\n\n> For the questions in which the number of solutions which reached the correct are very low (eg. 1-2 generations), what fraction does Verification@k get correct?\n\nWe will run a more formal analysis, but we can give some preliminary numbers. In particular,  the AIME exam falls in the regime you’re asking about: on 12/15 questions, less than 4% of Gemini Pro solutions are correct. Nonetheless, Verification@200 gets >50% accuracy. On the 7/15 questions where exactly 1-4% of questions are correct, Verification@200 gets 5/7 correct. We agree that this would be impactful to include and appreciate the suggestion.\n\n> Could the authors clarify what is prompted during self-verification...? Why are the FP and FN rates missing for the LiveBench questions? Does the difference in FP and FN rates translate to... Verification@k accuracy?\n\nThe full prompts for all four options referenced in Table 4 can be found in the source code; we will revise to include a copy in the Appendix. We added LiveBench at a later date and had not yet re-run Table 4, but will include them in our revision. We informally observed Verification@k accuracy to be especially sensitive to FN rates, and will add a formal table to this effect.", "claims_and_evidence": "I’m skeptical of the author’s implicit scaling claim— that increasing the number of generated responses increases the quality of responses. It is unclear if increased generations improves generation quality or simply increases generation coverage [1], and thus even if you have a suboptimal verifier, it is more likely to select a solution that reaches a correct final solution. I don’t follow the authors’ claim that increasing generations will increase the verifier’s likelihood of error, given that pass@k is known to increase– thus, instead of focusing on ‘ambiguous’ questions in Figure 2, I believe it is more interesting to identify the number of instances where the correct solution appears very few (1-2 times) in the k_inf generations, but the verifier still selects the correct answer (the authors provide a singular example of this in Table 2, but there lacks further analysis on this).\n\n[1] Brown, Bradley, et al. \"Large language monkeys: Scaling inference compute with repeated sampling.\" arXiv preprint arXiv:2407.21787 (2024).\n\nEdit: After discussions with the authors, I believe that while the results are not very surprising (ie. you need model scale and proper output formatting for verification to scale with inference generations) I am more sympathetic to the authors demonstrating that this is possible. I still believe the paper requires restructuring for clarity and more rigorous definitions of eg. implicit scaling, but authors have said they will implement these changes.", "methods_and_evaluation_criteria": "Evaluation criteria is straightforward (correctness on the reasoning benchmarks considered). I'm seeking clarification for one part of the methodology: the authors propose a “Verification@k” process detailed in Algorithm 1, but upon reading the details in Section 3.1 the authors state that the candidate responses are rewritten in a format with Theorems/Lemmas/Proofs before being passed for self-verification. This is not obvious from the introduction, and this confounds the verification process with an additional step that may be aiding self-verification. There are not many details about how the rewrites are prompted, even though the authors do ablate over variations of this prompt.", "theoretical_claims": "N/A, there are no theoretical claims.", "experimental_designs_or_analyses": "See my comment in “Claims” regarding the experiments around ‘implicit scaling’. I believe the analysis regarding the accuracy rates relative to scaling number of generations and number of verifications could be improved with a better FLOPs comparison to judge the best allocation of verification and generation attempts; the authors mention compute for a fixed setting of the parameters on AIME in section 4, but this should be present across their results in Section 2.", "relation_to_broader_scientific_literature": "There is a growing trend of using additional computation at inference to enhance performance. The paper directly compares to prior scaling test-time methods like Consistency@k. There has also been recent attention in AI research aimed at making LLMs more trustworthy and less prone to errors. Self-verification is a prominent approach under exploration.", "essential_references_not_discussed": "Authors have not cited related work [1] which was previously mentioned above.\n\n[1] Brown, Bradley, et al. \"Large language monkeys: Scaling inference compute with repeated sampling.\" arXiv preprint arXiv:2407.21787 (2024).", "other_strengths_and_weaknesses": "Careful studies of how search should be scaled at inference time and whether self-verification can be effectively leveraged is a relevant area of research. The authors present analysis on several standard math and reasoning datasets. The potential of self-verification for models to reach reasoning-model performance is promising. \n\nThe observation that reasoning models have weak out-of-box verification capabilities has been previously reported (eg [1]). Further, I believe the paper is organized poorly to the point that it is confusing for a reader to follow the paper. The introduction presents a result (Table 1) before the methods are described, and only Algorithm 1 is given before the authors dive into more results in Section 2, only to loop back to talking about the ‘Sampling-Based Search Implementation’ in subsection 3.1 and technical details in Section 4. Further, the search method employed in Section 2 appears to not be the same as that in Algorithm 1; the tiebreaking was omitted, which I only understood in full after reading Line 275 in Section 3.2. I ask the authors to reorganize their paper such that the methods and setup are described before results are presented. \n\nGiven my feedback about the exposition, the claims, and the insufficient analysis, I do not recommend acceptance of the paper in its current state. I am willing to raise my score if several points raised in my review are clarified, if the paper’s structure is reorganized, and some additional analyses are provided (see Questions).\n\n[1] Wu, Zhenyu, et al. \"Large language models can self-correct with key condition verification.\" arXiv preprint arXiv:2405.14092 (2024).", "other_comments_or_suggestions": "No other comments, see below for questions.", "questions_for_authors": "1. Are the results in Section 2 doing the same process as in Section 3.1? In Step 2, it is mentioned that the scoring attempt is a rewrite of the response as a theorem, lemmas and proofs. How are these rewrites done? If this was done for Section 2, this should be explicitly stated because it is not clear from Algorithm 1 that this is what is done, and  this confounds the verification process by presenting it in a standardized format which potentially makes verification easier.\n2. Can the authors clarify their point about implicit scaling and how their results suggest higher quality of generations with multiple samples? \n3. Do you have a FLOPs comparison of the different settings of k_inf and k_ver? It personally is not surprising to me that scaling up both k_inf and k_ver will show increases in performance, and the analysis feels incomplete without showing a FLOPs tradeoff between inference and verification generations to show, given a fixed compute budget, what the tradeoff should be between generation and verification.\n4. On each benchmark, for the questions in which the number of solutions which reached the correct are very low (eg. 1-2 generations), what fraction does Verification@k get correct?\n5. Could the authors clarify what is prompted and given to the model during self-verification for the 'main' prompt and each of the three variations? Why are the FP and FN rates missing for the LiveBench questions? Does the difference in FP and FN rates across the different prompts translate to significant differences in the Verification@k accuracy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tW3m7cKnok", "reviewer": "Reviewer_VNHP", "summary": "Overall Evaluation  \nThis paper investigates the scalability of sampling-based search methods in inference tasks and proposes a minimal yet effective Sampling-based Search with Self-Verification approach. The key contributions of this work include:\n\n1. A systematic analysis of inference performance scaling with computation, revealing that both search scale (`k_inf`) and verification capacity (`k_verif`) jointly influence reasoning accuracy. The experiments demonstrate that verification plays a crucial role in improving inference reliability.  \n2. The introduction of the implicit scaling phenomenon (Implicit Scaling), showing that increasing the search scale unexpectedly improves verification accuracy—i.e., as the search space expands, the probability of generating high-quality answers increases, leading to overall performance gains.  \n3. An evaluation of the out-of-the-box self-verification capability of leading models, showing that GPT-4o and Claude 3.5 Sonnet perform only marginally better than random guessing in self-verification tasks, indicating a significant limitation in current LLM reasoning capabilities.  \n\nThe paper conducts large-scale experiments on Gemini v1.5 Pro, demonstrating the advantages of Verification@200 over Consistency@200 in complex reasoning tasks. Furthermore, it proposes Comparison-based Verification and Response Rewriting as optimization strategies to enhance model verification.  \n\nOverall, the paper is well-structured, with a clear motivation, rigorous experimental design, and insightful conclusions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We address your questions below.\n\n> Computational cost compared to alternative reasoning frameworks such as Tree-of-Thoughts (ToT) and reinforcement learning-based approaches.\n\nOptimizing computational efficiency was not the main focus of this paper, which focuses on understanding the broader scaling trends underlying sampling-based search via self-verification. While we chose to study a minimalist implementation of sampling-based search, we expect our insights to generalize to alternative implementations that place a premium on computational efficiency. In fact, there are several avenues along which the effectiveness and efficiency of our minimalist sampling-based search implementation can be improved: prompt caching, delegated search, training smaller verifiers, etc.—all exciting directions for future work. Regarding ToT and RL, these are complementary techniques: you can always apply sampling-based search and self-verification to RL-trained models that use ToT for inference. As a result, comparing between two methods that can be used in concert imposes an unnecessary trade-off.\n\n> Application of Verification@200 on other LLMs.\n\nIn Appendix B.2, we include results that apply Verification@200 to the significantly smaller Gemini 1.5 Flash model. Computational concerns were a limiting factor in the number of LLMs studied, but we find no reason for the trends we observed with 1.5 Pro and 1.5 Flash to not generalize to models like GPT4o.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Enhance the trustworthy of  LLMs", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "Comments and Suggestions  \n- Limited application of Verification@200 on other LLMs: maybe due to constraints in computational resources, the authors did not apply Verification@200 to models such as GPT-4o, which would have provided a broader validation of the method's generalizability. It would be beneficial to explore its impact on multiple state-of-the-art models.  \n- Positioning relative to alternative reasoning enhancement methods: While Verification@200 demonstrates performance gains, I am concerned about its computational cost compared to alternative reasoning frameworks such as Tree-of-Thoughts (ToT) and reinforcement learning-based approaches. Given the increasing focus on efficient inference, should Verification@200, which requires large-scale computation, be the primary direction for improving model reasoning? A comparative discussion with these alternative methods would strengthen the paper's positioning.  \n\nOverall, the work presents a compelling approach to enhancing reasoning through scalable verification but would benefit from additional discussion on computational efficiency and comparisons with alternative reasoning paradigms.", "other_comments_or_suggestions": "No", "questions_for_authors": "See the Comments and Suggestions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6sl1n6FVh4", "reviewer": "Reviewer_CzVL", "summary": "This paper studies the inference-time scaling of LLMs for reasoning tasks in a sampling-based search setting. The authors first study the test-time scaling along two important dimensions, search (number of sampled candidates) and verification (number of verification scores computed). While scaling in both axes improves the accuracy, the authors additionally discovered an implicit scaling where larger sampling pools actually increase the verification accuracy. Built upon their findings, this paper identified two principles to boost self-verification of LLMs, candidate pair comparison and response rewriting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We address your questions/comments below.\n\n> “I would suggest the authors to also report the token consumption of baselines (e.g., Consistency@k) as well for a more fair comparison”.\n\nWe appreciate the suggestion and agree it would be useful. As our focus was identifying broader scaling trends rather than computational efficiency, we chose to study a minimalist implementation of sampling-based search that leaves significant room for optimization e.g. via prompt caching, delegated search, small verifiers, etc.—all exciting directions for future work as the reviewer has accurately noted. \n\n> Algorithm 1\n\nWe appreciate the feedback and will revise the algorithm description accordingly. The notation 1[is correct] would indeed benefit from clarification.\n\n> Adding analyses of R1/R1-Zero models.\n\nThis is a great suggestion and something we hope to do. Unfortunately, since R1 was released only 3 days before the ICML deadline, we were unable to include an analysis in this submission but agree that it’s valuable to analyze R1’s sampling-based search trends.\n\n> On Table 4\n\nTable 4 depicts the implications of ablating the rewriting step from the verification process, showing a significant drop. We include in Appendix 3 a few examples of the outcomes of the rewriting step, which illustrate the rigor and structure that result. We will add a more detailed discussion to this effect as well.", "claims_and_evidence": "1. I am confused about the claim \"Frontier language models demonstrate a remarkable mismatch between the problem-solving capabilities and poor out-of-box verification capabilities\". I am wondering how the \"poor out-of-box verification capabilities\" are proved in the paper using experiments.", "methods_and_evaluation_criteria": "1. The authors proposed Verification@k as a variant in sampling-based search to study self-verification in LLMs, which is sound and interesting.\n2. The proposed two principles are well supported by experiments on the newly constructed benchmark.\n3. Benchmark selection for MATH, AIME, and LiveBench is reasonable.", "theoretical_claims": "This paper does not contain theoretical claims and corresponding proofs.", "experimental_designs_or_analyses": "The experiment design in this paper is valid and reasonable. I have some additional suggestions for experiments in the following weaknesses/question section.", "relation_to_broader_scientific_literature": "This paper provides another comprehensive investigation in the area of test-time computing, specifically scaling at inference time. \n\nOne of the key contributions is the exploration of scaling with \"verification\". The authors propose a very intuitive \"verification@k\" algorithm, which could serve as baseline for follow-up works.\n\nThis paper also offers insights that are not seen in previous works, and would be valuable to the community, such as the conclusion in \"Pass@k v.s. Pass@1\".\n\nThe proposed verification dataset and the two design principles are also new and interesting.", "essential_references_not_discussed": "The literature discussion/related work is thorough in my opinion.", "other_strengths_and_weaknesses": "1. I appreciate the author's reporting compute/token consumptions in Section 4. Given the expensive cost of the proposed verification scaling, I would suggest the authors to also report the token consumption of baselines (e.g., Consistency@k) as well for a more fair comparison.\n2. The paper offers interesting insights for researchers and practitioners that may be beneficial for downstream tasks. For example, the insights on Pass@k v.s. Pass@1.\n3. The paper is overall well-written and is a joy to read.", "other_comments_or_suggestions": "1. Algorithm 1 may need some improvement to make it easier to understand for readers. For example, it took me some time to understand that \"1[issicorrect?]\" is actually a binary classification of the given query for verification.\n2. This may sound a bit demanding, but I think the paper would be further benefited from adding analyses of R1/R1-Zero models. For example, the distilled version of R1, and analyze their behaviours w.r.t. sampling-based search.\n3. Since the authors mention that \"rewriting responses with increasing rigour and structure\", I am not sure if the current implementation/design in Table 4 actually reflect this.", "questions_for_authors": "See the above reviews.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LOChfHInjo", "reviewer": "Reviewer_uCGZ", "summary": "The paper studies a minimalist implementation of test-time scaling that uses only random sampling and direct self-verification. The contributions include:\n1. the paper shows that the sample-verification method is surprisingly effective, and it is beneficial to scale both number of solution samples per question and number of verification samples per solution.\n2. the paper proposes two effective methods to improve self-verification: pairwise response comparison and response style rewriting.\n3. the paper introduces a benchmark to evaluate the verification accuracy of LLMs.\n\n## update after rebuttal\nI agree with the clarification of the main contributions. Therefore I raised my score to 3. However, I still think the implicit scaling is not reasonable and the dataset lacks diversity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n> The main contribution of the paper.\n\nWe view our main contribution as showing that scaling search with self-verification works on frontier models and providing an explanation for why: implicit scaling. While it may seem obvious that “increasing verification samples leads to higher accuracy”, many papers (including Snell et al.) have studied self-consistency and self-verification and attempted to scale them up, but none have reported the same success. In fact, we aren’t aware of any prior works that have successfully applied self-verification to frontier models in a way that actually beats self-consistency. Instead, to beat self-consistency, the field has turned to training process-based reward models, using RL to learn verifiers, and other complicated interventions. “Increasing verification samples” sounds like a simple panacea but making it work is a different matter.\n\nThis has led to a general belief that simple self-verification isn’t enough, whereas we show that it is—you just need to hit a certain threshold of scale and obey some important principles. It's also noteworthy that direct self-verification can successfully and reliably extract o1-level performance from a non-reasoning model without any finetuning, distillation, RL, or custom-trained verifiers (again, no other works/methods—verification-based or otherwise—have achieved this).\n\nWe therefore respectfully, but strongly disagree with the assessment that “the key problem is what is the optimal allocation between the two factors given a limited compute budget”; prior to this work, it wasn’t even known that near-unbounded compute budget is sufficient to recover o1-level performance without interventions like RL. \n\n> “The difference with Snell et al., 2024.”\n\nOur paper differs from Snell et al. in several important ways. First, we study entirely different questions: Snell et al. asks what the optimal test-time scaling trade-off is between pretraining versus inference-time scaling; we ask what the scaling trends and limits of sampling with self-verification are. Second, we study qualitatively different regimes: making verification improve on self-consistency is easy when trying to improve e.g. MATH performance from 10-40% (as is done in Snell et al. 2024); making verification improve on self-consistency on the last 90% of MATH or on actually challenging benchmarks like AIME is significantly more difficult (we aren’t aware of any works prior to ours that were able to do so without RL). Third, Snell et al. 2024 trains a process-based verifier reward model; we show that direct self-verification works fine.\n\n> “The benchmark consists of only responses from Gemini-1.5-pro. I think it is insufficient to evaluate frontier models.”\n\nGemini 1.5 Pro-002 was released in September 2024, making it one of the main frontier models at the time of this submission. Our paper also includes results on Gemini-1.5-Flash (a significantly smaller model with major qualitative differences) in Appendix B.1. We also note for context that the Snell et al. 2024 paper ran experiments only on the Palm-2 model and on a single benchmark (MATH).\n\n> Omit questions that Consistency@200 answers correctly to reduce noise. It might also change the results.\n\nOn MATH for example, 98% of “ambiguous questions” are easy questions that Consistency@200 answers correctly and hence Verification@200 answers correctly. When we plot Figure 2 without omitting these questions, we end up with a line that goes from 98% to 99% which entirely misses the point of the plot (which is to highlight implicit scaling). To fulfill the reviewer’s request, we will include an analog of Figure 2 in the Appendix where Consistency@200 filtering is not done. However, no trends change. For example, on AIME, Consistency@200 only gets 1-2/15 questions correct—omitting those questions has no material impact on the trends observed in Figure 2.\n\n> MATH trend on Figure 2.\n\nIncreasing the number of generated samples makes verification on ambiguous questions  harder; this follows from basic probability: if you allow students more attempts at an exam, the average pass-rate of a student who passes at least one exam decreases. Similarly, as you increase the number of attempts, the average number of correct attempts/question decreases. Thus, if implicit scaling didn’t exist, *all* lines in Figure 2 should be strictly decreasing. The fact that we observe the opposite proves that implicit scaling must be happening. The fact that there is a 63->61% drop on the right-hand side of the MATH plot is not evidence against this and is—if anything—expected: we can’t expect implicit scaling is universally powerful enough to completely reverse the natural bias of increasing verification difficulty.", "claims_and_evidence": "The paper claims the existence of implicit scaling of verification accuracy: increasing the number k of generated responses increases the conditional probability of at least one of the correct responses is of the highest verification score given that at least one response is correct (Pass@k=1). This is not intuitively or theoretically verified and only supported by the limited evidence on a single model. I note that the trend on the MATH benchmark in Figure 2 is downwards when k is larger than 80, contradicting the claim. Also, the figure omits questions that Consistency@200 answers correctly to reduce noise. It might also change the results.", "methods_and_evaluation_criteria": "The idea of scaling number of verification samples is sensible to improve the verification accuracy. The two methods to improve self-verification also make sense. \n\nThe benchmarks used to evaluate reasoning ability are AIME, MATH, and LiveBench. They are common benchmarks in mathematical reasoning. Including reasoning benchmarks in other areas like LiveCodeBench (for coding reasoning) and GPQA (for science reasoning) will be better.", "theoretical_claims": "No proofs in the paper.", "experimental_designs_or_analyses": "Yes. The experimental setting is simple and solid.", "relation_to_broader_scientific_literature": "The effectiveness of search against a verifier is already demonstrated in Snell et al., 2024. They also show that search against a verifier can improve accuracy when majority voting saturates.\n\nThe verification benchmark is in fact a benchmark for reward model on mathematical reasoning. It is known that the accuracy of reward model is sensitive to the distribution of responses (Zheng et al., 2024). The benchmark consists of only responses from Gemini-1.5-pro. I think it is insufficient to evaluate frontier models.\n\nSnell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024.\nZheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J. and Lin, J., 2024. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559.", "essential_references_not_discussed": "The difference with Snell et al., 2024. is not sufficient discussed.", "other_strengths_and_weaknesses": "I think the paper is a mixture of contributions to three rather independent directions, while the contribution to each direction is insufficient. \n\nFirst, the paper highlights the impact of scaling verification compute besides inference compute. However, it is obvious that increasing verification samples leads to higher accuracy, since verification is also a reasoning task and majority voting improves its accuracy. Since there are two scaling factors (number of solution samples per question and number of verification samples per solution) and they both require computing resources, the key problem is what is the optimal allocation between the two factors given a limited compute budget. \n\nSecond, the paper proposes two effective methods to improve self-verification. However, their contribution to the end-to-end accuracy is not measured. Also, the pairwise comparison is only applied to the tie-break stage for three highest scoring candidates. \n\nThird, the paper introduces a benchmark to evaluate the verification ability of LLMs. The verification candidates in the benchmark are all sampled from Gemini-1.5-pro, which means that it cannot comprehensively evaluate verification accuracy on diverse solutions.", "other_comments_or_suggestions": "The authors should place section 4 (Technical Details) in the appendix and appendix A (Related Work) in the main paper.", "questions_for_authors": "How is the threshold 0.05 selected in the algorithm?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies test-time compute, and finds that generating lots and lots of candidate solutions combined with self verification allows smaller LLMs to outperform state-of-the-art reasoning models.  The findings here are interesting and topical.  Given the interest in reasoning and test-time compute right now, this paper will contribute to ICML.  The reviewers largely voted for acceptance, with the one dissenting reviewer only presenting two comments, neither of which I feel are disqualifying.  For example, the reviewer suggests that this paper should investigate computational efficiency.  While this would be interesting, it is not the focus of the paper and I think the paper stands without that.  Therefore, I vote for acceptance.
Decision: Accept (poster)