When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning
Authors: Nishad Singhi
Hritik Bansal
Arian Hosseini
Aditya Grover
Kai-Wei Chang
Marcus Rohrbach
Anna Rohrbach
Venue: COLM 2025 (2025)
Keywords: test-time scaling
self-consistency
generative reward models
compute-matched analysis
TL;DR: We perform test-time compute matched comparison between scaling solutions via self-consistency and verifications via GenRM to find useful insights for the practitioners.

Abstract: Scaling test-time compute has emerged as a key strategy for enhancing the reasoning capabilities of large language models (LLMs), particularly in tasks like mathematical problem-solving. A traditional approach, Self-Consistency (SC), generates multiple solutions to a problem and selects the most common answer via majority voting. Another common method involves scoring each solution with a reward model (verifier) and choosing the best one. Recent advancements in Generative Reward Models (GenRM) reframe verification as a next-token prediction task, enabling inference-time scaling along a new axis. Specifically, GenRM generates multiple verification chains-of-thought to score each solution. Under a limited inference budget, this introduces a fundamental trade-off: should you spend the budget on scaling solutions via SC or generate fewer solutions and allocate compute to verification via GenRM? To address this, we evaluate GenRM against SC under a fixed inference budget. Interestingly, we find that SC is more compute-efficient than GenRM for most practical inference budgets across diverse models and datasets. For instance, GenRM first matches SC after consuming up to $8\times$ the inference compute and requires significantly more compute to outperform it. Furthermore, we derive inference scaling laws for the GenRM paradigm, revealing that compute-optimal inference favors scaling solution generation more aggressively than scaling the number of verifications. Our work provides practical guidance on optimizing test-time scaling by balancing solution generation and verification.

Published as a conference paper at COLM 2025
When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning
Nishad Singhi*!, Hritik Bansal Arian Hosseini**A4
Aditya Grover?, Kai-Wei Chang", Mareus Rohrbach', Anna Rohrbach' ITU Darmstadt, Aessian.AT 2UCLA
Google Deepmind
+Mila "Equal Contribution
Abstract
Scaling test-time compute has emerged as key strategy for enhancing the reasoning capabilities of large language models (LLMs), particularly traditional approach, Self-
in tasks like mathematical problem-solving. Consistency (SC), generates multiple solutions to a problem and selects the most common answer via majority voting. Another common method involves scoring each solution with a reward model (verifier) and choosing the best one. Recent ad advancements in Generative Reward Models (GenRM) reframe verification as next-token prediction task, enabling peereence-timee scaling along new axis. Specifically, GenRM generates multiple verifica- tion chains-of-thoughh to score each solution. Under limited inference budget, this introduces a fundamental trade-off: should you spend the budget scaling solutions via SC generate fewer solutions and allocate compute to verification via GenRM? address this, we evaluate GenRM against SC under fixed inference budget. Interestingly, we find that SC is than GenRM for most practical inference budgets across diverse models and datasets. For instance, GenRM first matches SC after consuming up to 8x the inference compute and requires significantly more compute to outperform it. Furthermore, we derive inference scaling laws for the GenRM paradigm, revealing that compute-optimm inference fa- vors scaling solution generation more aggressively than scaling the number of verifications. Our work provides practical guidance on optimizing test- time scaling by balancing solution generation and verification. The code available at https::/ //gp::://i/i thub. wwt.:/ccooinniiieeeee www.sshiiiiiisscccccc..o. genrm- soarrm--caaing..
more compute-efficien
SC
V=20
V=2+
V=2²
V=2²
V=24
V=25
SC vs GenRM-FT at Fixed Num. Solutions
SC vs GenRM at Fixed Compute Budget
60
128x
4x fewer solutions
8x compute
55
55
Success (a)
(b)
Success
45
45
40
40
35
35
2
2
2
2
2
Number of Solutions
Inference Compute per Problem (FLOPs)
Figure 1: Left: The prominent approach is to compare GenRM and Self-Consistency (SC) at fixed number of solutions, suggesting that GenRM is more efficient matches SC with fewer solutions. Right: When evaluated under a fixed compute budget, including verification costs, SC outperforms GenRM at lower budgets, using up to 8x less compute, while GenRM excels at higher budgets. Each curve corresponds to fixed number of verifications; the number of solutions doubled at each point along the x-axis. The solutions are generated by Llama-3, 1-8B-Insttttt (Grattafiori et al., 2024), which also performs verifications after being fine-tuned as GenRM, on the MATH dataset (Hendrycks et al., 2021).

---

Published as a conference paper at COLM 2025
[b)
Self- Soafff-onsittencyyyooc (SC)
Compute-Matchann Analysis
Generative Reward Models (GenRM)
Scaling Solutions (SC)
Scaling Verifications
Solution
Sols.
Majority Voting
?
(GenRM)
Solution
V
P
Sale
Generative Verifier
per Sol.
BoN
Conerator
[isere sakelone)
(lwwr
Compute
Fixed Budget B
Compute = (B/V)
Figure 2: Compute-Matched Analysis. Given a fixed inference budget B, our analysis (b) compares the performance of (a) scaling the number of solutions (S = B) with Self- Coniisonnittency vs. (c) generating fewer solutions (5 = B/V) while relying verifications (V) using Generative Reward Models.
1 Introduction
Large Language Models (LLMs) have shown substantial improvements in their reasoning capabilities with increased test-time compute (Guo et al., 2025; Goyal et al., 2023; Snell et al., 2024) across diverse domains such as math and science. One of the most straight- forward and effective ways to scale test-time compute is by generating multiple solution mhains-of-though (CoTs) for a given problem and performing majority voting over them, technique known as Self- Consisonnstyy (SC) (Wang et al., 2022). Alternatively, a reward model (aka 'verifier') can be used to score the solutions and identify the best one. This strategy, commonly referred to as Best-of-N (BoN), has been widely employed to enhance LLM reasoning capabilities (Cobbe et al., 2021). A robust reward model can detect errors and discard incorrect solutions, even when they are overrepresented among the generated solutions, making this approach particularly effective.
Recent studies (Zhang et al., 2024; Mahan et al., 2024; Ankner et al., 2024) have framed verification as a next-token prediction task, leading to a new class of reward models known as Generative Reward Models (GenRM). These models enable test-time scaling along new axis: generating multiple verification Co'Is and aggregating their verdicts to score a given solution. Prominently, prior work compares the performance of GenRM (BoN) and SC at fixed number of solutions. For instance, Llama GenRM (BoN) with 32 verification CoTs surpasses SC across different solution counts (Figure ((a)). This comparison suggests that GenRM is more efficient, achieving the same performance as SC with 4x fewer solutions. However, this conclusion is misleading gets (FLOPs) are limited, as the current comparisons overlook the significant computational cost of generating many verifications for several candidate solutions of a given problem. This gives rise to the question: At fixed inference compute budget, is it more effective to perform SC over many solutions, or use GenRM to select the best solution from a smaller set of solutions by scaling the number of verifications?
practical scenarios where inference compute bud-
To address this, we present a framework to estimate the inference budget for Self- Consistency and GenRMs. Subsequently, we conduct a compute-matched analysis, compar- ing the effectiveness of these test-time scaling strategies under a fixed computational budget ($3.1). Specifically, we assume an LLM is used for both problem-solving (solution generator) and generative verification (reward model). The generative verification capability can be leveraged through either prompting or task-specific fine-tuning. Consequently, the inference compute comparison between SC and GenRMs is based on the total number of solutions and verifications generated by the LLM, as illustrated in Figure 2.
In our experiments, we compare the performance of scaling solutions and verifications with GenRM against simply scaling solutions via SC across various inference compute budgets. Our results indicate that SC outperforms GenRM at lower compute budgets, while GenRM performs better at higher budgets (Figure 1(b).. In particular, GenRM first surpasses SC after 8x compute and requires an additional 128x inference compute to achieve a 3.8% performance gain over SC. While prior work realized GenRM in limited settings, we demonstrate the robustness of our findings actoss various model families (e.g., Llama (Grattafiori et al., 2024) and Qwen (Yang et al., 2024)), model sizes (e.g,, 7B and 70B), thinking models (e.g., QwQ-32B (Qwen, 2025)), and reasoning tasks (e.g., math) (55.1).
2

---

Published as a conference paper at COLM 2025
As the compute budget is scaled and GenRM starts outperforming SC, second challenge emerges: with GenRM, the available compute can be split between generating solutions and verifying them in different ways, leading to varying performance. For example, in Figure 1(b), using 8 verifications (red curve) performs better than 4 verifications (green curve) at a budget proportional to 2 FLOPs. This shows a key tradeoff in GenRM: sampling too few solutions may lower the chances of generating correct one (low coverage), while performing too few verifications can make it harder to identify the correct solution (low precision). Hence, this raises the question: Under GenRM, how allocate a given compute budget between generaling solutions and verifying them to achieve optimal performance?
To address this, we derive inference scaling laws for GenRM, which describe how the optimal number of solutions and verifications scales with the total compute budget ($3.2). We observe that while both need to be scaled in tandem, the solutions should be scaled more rapidly than verifications, by a factor of 1.5 2x, x, for optimal performance (S 5.2). Overall, our work provides a solid foundation for understanding the trade-offs associated with test-time scaling of solutions and verifications, offering key insights for practitioners seeking to optimize their inference strategies and budgets.
2 Background
Repeated sampling. A prominent approach for scaling test-time compute is repeated sam- pling, which involves generating multiple potential solutions from the LLM, G, and selecting a final answer as the prediction. Specifically, we study two common methods for choosing the final answer: (a) Self-Consistency (SC) selects the most common answer-determing by majority voting-as as the final answer (Wang et al., 2022). This strategy benefits from ex- ploring diverse reasoning pathways and marginalizing over them, increasing the likelihood of correct answers. (b) Best-of-N (BoN) scores each candidate solution independently and selects the highest-scoring one (Cobbe et al., 2021). This method relies on reward model capable of accurately assessing problem-solution pairs for correctness.
Generative Verification. Unlike traditional reward models that are discriminative, recent works have developed Generative Reward Models (GenRM) that pose verification as next token prediction task (Zhang et al., 2024; Mahan et al., 2024). Concretely, the verifier takes as input the problem and the step-by-step solution, and provides a verification chain-of- thought (CoT) followed by its verdict (e.g., 'Is this answer correct? Yes/No)) This approach enables RenRM to inherit all the advantages of LLM reasoning, most notably test-time scaling, along a new axis: verification. In particular, we can sample multiple verification CoTs for each solution, and average over their verdicts to obtain a more accurate score. ?
GenRM-Base.
The simplest form of GenRM involves prompting an instruction-tuned LLM to verify a solution step-by-step and predict whether the candidate solution/ maationnansserr answer is correct for the given problem. We refer to this as GenRM-Baa Base since it uses an off-the-shelf
LLM without specialized fine-tuning to act 3 3
F.nMt-Finetuiing. Zhang et al. (2024) train an LLM to perform generative verification via supervised fine-tuning, which we denote as GenRM-FT. We start with a solution generator 9, a student verifier studdet which will be fine-tuned to obtain GenRM-FT, and a teacher verifier /teach which is used to generate synthetic data to fine-tune Tstudent- The fine-tuning data is generated in the following way. We take a training dataset, D = {(x, y; a,),, consisting of problems x,, ground-truth solutions y,, and ground-trun -tuth answers a, We use G to generate N₂ solutions for each problem in the dataset, where 91, and à; are the jth solution and answer generated by G for problem í. Then, we use the teacher verifier Tteach (usually a strong model like GPT-4o (Hurst et al., 2024)) generate N₂, synthetic verification
We will release the data, model, and code in the uamera-ready version.
Whill other sophisticated inference strategies exist (e-g. process-level scoring or tree search (Wu et al., 2024; Snell et al., 2024)), their practical use is constrained by the necessity of high-quality process-level training data and the inherent complexities associated with their training.
"This is akin to self-verification in Zhao al. (2025) where an I.I verifies its own solutions.
3

---

Published as a conference paper at COLM 2025
rationales for the generated solutions 91,- The teacher verifier has access to the ground-truth solutions in its prompt, allowing to generate accurate, high-quality verifications. Every verification generated by the teacher consists of chain-of-thought (CoT) reasoning about the solution's ss correctness, followed by a final verdict ('Yes' or 'No'). We filter these synthetic verifications, retaining only those whose final verdict matches the ground-truth correctness of the generated answer ā. Further, we balance this data to have an equal number of 'Yes' and 'No' verifications, leading to the final fine-tuning dataset D GenRM-FT = {(VCOT. V x. x) consisting of verification rationales VCoT and final verdicts v. Finally, we fine-tune rstudett on this dataset to obtain GenRM-FT. During inference, we generate a verification rationale via the GenRM (-base or -FT) and use the probability of the 'Yes' token the final score
"GenRM (x, y) Po ye,|),,,,ve y,CoT)
Test-time scaling with GenRM. For every problem x; in the test dataset, we generate samples ((R.,. و
} using the solution generator 9, where 91, and å.j refer to the 5th solution and answer, respectively. For every problem-solution pair, we generate V verifications (Viijjv k using the generative verifier, where fi,j ~ "GenRM (x; Vi,)) E [0,1] is the verification score. We obtain the final score for the solution Y1,1 by averaging
the verification scores from all V verifications as Pt,j = (1/V) Ex_n Pi.x. Finally, we pick the solution with the highest score Pr, as the final solution. To sum up, GenRM allows time-test scaling along a new dimension by increasing the number of verifications V in addition to increasing the number of solutions S.
Thinking Models. An emerging approach to scaling test-time computation training mod- els to generate longer chains-of-thouge (CoTTs) involving deep thinking before producing solution. These models, e.g, DeepSeek-R1 (DeepSeek-A) AI, 2025), QwQ-32B (Qwen, 2025), are trained using reinforcement learning and can leverage additional test-time compute for self-verification, reflection, and backtracking to arrive at a final solution. However, the amount of compute they allocate to find and verify solution in their thought process remains uncontrollable. Moreover, their reasoning mechanisms are not yet well understood in the current literature. Here, we focus on complementary inference strategies, which can also benefit thinking models, rather than sequentially refining a single solution. We present detailed related work in Appendix A.
3 Methodology
3.1
Compute-Matched Analysis of Test-time Scaling Strategies
Given the option to allocate a fixed inference budget toward either scaling solutions via Self-Consistency or verifying them using GenRMs, it remains unclear which approach is compute-optimal for LLM reasoning. To address this question, we conduct a compute- matched analysis of their respective scaling behaviors. We consider autoregressive LLM with P parameters that will perform 2P FLOPs per output token during inference (Kaplan et al., 2020). Hence, the number of inference FLOPs for generating T tokens is 2PT.
Let the number of tokens required for generating a solution and verification be Ts and Ty, respectively. Following Zhang et al. (2024), we use the same model for moblem--olvingg and generative verification. For instance, one might use Llama-8B to generate solutions and fine-tuned version of the same model as Gen (or the same model without fine-tuning as GenRM-Base). Hence, the number of model parameters for the solution generator and verifier is identical, say P. Thus, the total inference compute (FLOPs) required to solve reasoning problem with S solutions and V verifications 2P + TySV) | Further, we consider Ty ATş where À is the ratio of the number of tokens per verification and solution. In our analysis, we use the formula C(S.V) = 5(1 AV) to measure inference
We find that counting the number of Yess across multiple verifications also works well.
4

---

Published as a conference paper at COLM 2025
compute for simplicity, as it is proportional to the total inference FLOPs for given LLM. For Self-Consistency (SC), we set the number of verifications to 0.
We evaluate SC by sampling S solutions and performing majority voting over them, for all S S {20, 21, ,..,,2),, where 2 is the maximum number of solutions. Similarly, we evaluate GenRM by varying the number of solutions S E S and verifications E == = 2', ,.., 2.., where 2 is the maximum number of verifications per solution. For every combination S, V E {S V}, we sample the corresponding number of solutions and verifications, and pick the final answer via Best-of-N. We compare the final answers against the ground-truth to compute success rates (SR), and plot them against the total compute, C).,V). Thus we compare the performance of GenRM and SC at the same compute budget.
3.2 Inference Scaling Laws for GenRM
As an emerging paradigm, GenRM lacks formalized inference scaling laws. Scaling test-time compute with GenRM involves scaling two independent axes: solutions and verifications. Further, the same amount of compute can be allocated between these two in different ways, leading to different performance outcomes. Hence, understanding the tradeoff between scaling solutions and verifications is crucial for compute-optimal inference. To address this, we extend the approach from Chinchilla (Hoffmann al., 2022) inference-time scaling. Specifically, we follow these steps:
Compute the success rate SRa, for an increasing number of verifications v, while keeping the number of solutions per problem constant.
Plot gS,g S0 SR.22
against the inference compute budget C s(1 + Av), and gener- ate such plots for all values
3. Smooth and interpolate curves to obtain a mapping from inference compute to SR.
For each budget, determine the optimal number of solutions (Sttt) that maximizes SR
(Appendix] Fig. 10). This results in trend of Sopt as function of the inference budget C.
5. Fit a power law, Sopt C", to establish relationship between the optimal number of solutions and the inference budget.
6. Similarly, repeat steps 1-5 for verifications to compute Vopt & C..
A higher value of (b) indicates that as compute increases, the number of solutions (verifi- cations) must be scaled more rapidly compared to the number of verifications (solutions). If a b, implies that solutions and verifications should be scaled the same rate.
Experimental Setup
Tasks. We use MATH (Hendrycks et 2021), dataset of high-school competition problems to evaluate mathematical reasoning. Additionally, we utilize the MATH train split to train GenRM-FT models. To evaluate generalization of GenRM-FT to harder math tasks, use the AIME24 dataset (AoPS, 2024), which consists of advanced high-school math problems. Finally, to evaluate reasoning beyond math, we use the GPOA-Diamond dataset (Rein et al., 2024), which consists of problems pertaining to physics, chemistry, and biology. Following Brown al. (2024), we perform all experiments on subset of 128 problems randomly sampled from the MATH test set. Similarly for GPQA, we perform experiments on a subset of 64 problems randomly sampled from the diamond split.
Models. Following prior works (Brown et al., 2024; Mahan et al., 2024; Ankner et al., 2024), we perform our experiments with Lama-3.1-8B-Instreee coverage across model families and sizes, we also experiment with Qwen-2.5-7B-Insttre (Yang et al., 2024) and 3.0ma-3.-77B-IInstrue on the MATH dataset. We sample solutions with temperature of 0.7 and with maximum of 1024 tokens.
(Grattafiori al., 2024). To ensure
GenRM-FT. We fine-tune as GenRM-FT. To create the GenRM fine-tuning data, use the corresponding models to generate solutions to problems from the MATH training split. Then, we use a stronger model, GPT-4o (Hurst et al., 2024), to generate verification rationales for these solutions.
Biiaaa-..-------t-htt and
models to serve
5

---

Published as a conference paper at COLM 2025
MATH (Easy)
MATH (Hard)
GenRM:FT
GenRM. Base
Normalized GenRM Improvement Over SC
GenRM-FT vs GenRM-Base
30
55
% Improvement
Success Rate (%6
compute
20
45
10
40
35
210
2
212
2
2
2
Inference Compute Per Problem (FLOPs)
Inference Compute per Problem (FLOPs)
(a)
(b)
Figure Left: Relative improvement achieved by Llama-3, 1--ma-3.1-BB--ll 1-St-unntttuct GenRM-FT (32 verifications) over SC for different difficulty levels in MATH. Hard problems benefit more from GenRM-FT, with up to 30% relative improvement over SC. Right: Comparing GenRM-FT against GenRM-Bas that GM-RM-FT consistently performs better, requiring much less compute to match the performance of GenRM-Base. This highlights the importance high-quality verifications.
find
Further details are available in Appendix C. We find that the verifications generated by GPT- 4o tend be lengthier than the solution itself, it analyzes several steps in the solution before making its verdict (Appendix L). Hence, during inference, we sample up to 2048 tokens from our GenRM-FT models with a temperature 0f7, i.e,, sample up to 32 verifications for 256 solutions, ie., S = {22),.., 2"] and = 25...
for GenRM-FT. We
GenRM-Base. We use Llama-3.3-70B-Inttuu to generate solutions and verifications with no fine-tuning, i.e, GenRM-Base, due to its strong cntrruction-oolloiinne and reasoning ca- pabilities. We sample solutions and verifications using temperature of 0.7 and maximum length of 1024 tokens, i.e,, A = We experiment with MATH and GPQA for this model.
We measure performance in terms of success rate (SR), iee, the average percent- age of problems solved on a test set. Following (Zhang et al., 2024; Hosseini et al., 2024), we use Best-of-N to select the final answer with verifier. Unlike SC, Best-of-N can detect rare but correct solutions with an effective verifier (Zhao et al., 2025). We provide the verifier success rate Appendix Eq.
Evaluation.
Experiments
First, we address the question whether at given budget one should scale both solutions and verifications via GenRM or only scale solutions via Self-Consistency (SC). Hence, in $5.1, we compare the performance of these two approaches across range of computational budgets. Further, using GenRM poses another question, as the same budget can be distributed between solutions and verifications in different ways, leading to different performance outcomes. Hence, we develop inference-time scaling laws for GenRM $5.2.
5.1 Fixed Budget Comparison between Self-Consistency and GenRM
Following prior work (Zhang et al., 2024; Mahan et al., 2024), we compare the success rates of Self-Consistency (SC) and GenRM-FT (w/ 32 verifications) across different number of on the MATH dataset in Figure 1(a). We that GenRM-FT outperforms SC when both have the same number of solutions. Further, GenRM- FT matches the performance of SC with 4x fewer solutions. However, this comparison does not account for the cost of generating verifications, and may give the misleading impression that GenRM is generally more efficient than SC.
solutions using
For a fair comparison, we plot the success rates against the total compute used to generate solutions and verifications, and compare SC and GenRM at the same budget in Figure 1(b).

---

Published as a conference paper at COLM 2025
V=2
V=2
V=2²
V=2
V=24
V=2
V=2
Transfer AIME24 (GenRM-FT Trained on
GenRM-Base QwQ-32B AIME25
12
256x compute
84
16x compute
4x compute
12.5%
Success Rate (95
6x compute
(b)
Success
76
2
25
2
2
2
2
2
2
Inference Compute per Problem (FLOPs)
Inference Compute per Problem (FLOPs)
Figure 4: (Left) Evaluation of GenRM-FT (Llama-3.1 trained on MATH) generalizing AIME24. GenRM-FT provides significant improvements over Cell-Connistency (SC) on these harder problems, demonstrating its generalization ability, though it requires substantially more compute to outperform SC. (Right) Comparison of GenRM-Bas versus SC for RL-tuned QwQ-32B model. This confirms previous observations: SC performs better lower budgets, while GenRM shines at higher budgets. We extrapolate the SC curves (dashed lines) because their performance saturates beyond a certain point.
Interestingly, we find that SC outperforms GenRM-FT at lower compute budgets. Notably, GenRM-FT requires 8x more compute to match the performance of SC.
Similar to Brown et al. (2024), we find that the performance of SC plateaus at around 128 solutions, and sampling solutions further does not provide additional benefits. Conse- quently, allocating test-time compute to GenRM-FT only starts to yield benefits beyond certain budget. Finally, we find that GenRM-FT achieves 3.8% improvement over the best performance of SC, but requires 128x more compute achieve this performance.
Takeaway
At lower inference-compute budgets, scaling solutions using Self-Consistency leads to better performance than scaling both solutions and verifications with GenRM. However, at higher budgets, GenRM surpasses Self-Consistency in performance.
Impact of Problem Difficulty. When GenRM outperforms SC at higher inference budgets, we investigate which types of problems benefit the most. Figure 3a, we analyze the rela- tive improvement achieved by applying Best-of-N with GenRM-FT over Self-Consistency, computed as Improvement = -SRRenRM
SRsc) / /SRsc- Specifically, we evaluate Llama-3.1- 8B-Instruct on two difficulty levels from the MATH dataset: level and level 5, which we denote as MATH (Easy) and MATH (Hard), respectively. Our results indicate that GenRM is particularly advantageous for more challenging problems, yielding up to 30% relative improvement in performance. These findings can inform the choice between GenRM and SC, with GenRM offering greater benefits for harder problems.
Impact of Verifier Quality. We also compare the performance of GenRM-FT against GenRM- Base at a fixed compute budget in Figure 3b. We find that GenRM-FT consistently outper- forms GenRM-Base, requiring up to 16x less compute to reach the same performance. This highlights the benefit of fine-tuning LLMs for verification, especially for smaller models with weaker perrructioo-oolowiinn and reasoning capabilities. Additionally, this suggests that as the verification capabilities of LLMss improve, GenRM based methods might become more compute-efficient More details are available in Appendix E. We explore alternative approaches to improve GenRM efficiency in Appendix F.
Hasy-To-Hard Generalization. In practice, GenRM-FT may encounter unseen problems with higher difficulty than the ones seen during training (Sun et al., 2024). Hence, we extend analysis to harder dataset (AIME-2024) for GenRM-FT trained on easier dataset (MATH). In particular, we compare SC and GenRM-FT for Figure 4(a). Interestingly, find that our previous observations still hold: SC outperforms
in
7

---

Published as a conference paper at COLM 2025
V=2
V=21
V=
V=21
V=24
V=2
GenRM-Base Llama-3-78 (MATH)
GenRM-Base Llama-3-708 (GPQA-D)
64x compute
256x compute
Bx compute
Sucress Rate (%)
76
Success Rate (%)
50
compyte
+21%
72
48
(a)
(b:
68
64
44
2
2
2
Inference Compute per Problem (FLOPs)
Inference Compute per Problem (FLOPs)
(a) MATH, and (b) GPQA- Diamond, respectively. These results highlight that across model sizes and reasoning domains, GenRM outperforms Self-consistency (SC) only at high compute budgets. We extrapolate the SC curve (horizontal dashed line) as its performance saturates after point.
Figure 5: Comparing GenRM-Baste with Llamm-3.3-70B-Instte
GenRM-FT at lower compute budgets, whereas GenRM-FT outperforms SC at higher budgets. For instance, SC achieves its peak performance using 16x less inference compute than GenRM-FT needs to reach the same level. However, GenRM almost doubles the performance of SC, but but requires 256 more compute to do so. This highlights that generative verifiers effectively generalize to much harder reasoning tasks.
Trends Across Model Families. To study whether our findings generalize to other model families, compare GenRM-FT and SC with Dwwen225-77-Innstruu
the MATH dataset. Our results in Appendix Figure 9 are consistent with our previous findings: SC outperforms GenRM-FT for most of the lower end of the budget spectrum, achieving its peak performance with 64x less compute than GenRM-FT. However, GenRM-FT shines at higher budgets, achieving an improvement of 5.4%, but by utilizing 512x more compute.
Trends for Thinking Models. We extend our analysis to the emerging class of RL-tuned reasoning models that are capable of deep thinking involving self-reflection (e.g., 'aha moments') before generating the final solution (DeepSeek-Al 2025). We evaluate QwQ-32B (Qwen, 2025) on the challenging AIME 2025 (AoPS, 2025) dataset as a solution generator and verifier (GenRM-Base) We provide more inference details in Appendix I. We present the results in Figure 4(b). Interestingly, we observe very similar trends for these models as well. In particular, GenRM requires 4x more compute to match the performance of SC, and achieves a 2.5% improvement with 16x more compute.
Trends Across Model Sizes. Here, study whether our findings apply to larger LLMs. To this end, we experiment with 1iama-3.3-70B--nstrue as GenRM-Base, which has 9x more parameters than Llama-3-8B. The results in Figure 5(a) consistent with our previous findings: SC performs better at lower budgets and achieves its peak performance with 4x less compute as compared to GenRM-Base
the same performance. Further, GenRM-Ban performs better at higher budgets, using 64x more compute to improve performance by 1.7%. This also highlights that using strong models like Llama-3.3-70B as Generative Reward Models can improve reasoning performance even without fine-tuning them for verification.
Trends Across Reasoning Domains. Test-time scaling can benefit reasoning in domains beyond math, such as physics, chemistry, and biology. Hence, we compare SC against GenRM-Base at a fixed compute budget on GPQA-Diamond using Llaama3.3-70B-IIsttru Our results in Figure 5(b) show that GenRM-Base can provide a boost in reasoning abilities across domains, but only at a large budget. For instance, it uses 256x more compute than SC to yield 2.5% improvement on SC. At lower budgets, however, SC performs better. Additionally, we observe similar trends in the domain of code generation. Specifically, in Figure we find that Llama 3 8B GenRM-Base requires 16x x more compute to match SC and 64x more compute to improve its performance by 3.5% on the LCB Benchmark (Jain et al., 2024).
8

---

Published as a conference paper at COLM 2025
Optimal Solutions Budget
Optimal Verifications Budget
Optimal # (Vad)
Optimal #Solutions (Sxd
24
25
24
2
Shondos
(a)
(b)
essegefes
aggarbrososes
2a
rotoeds
22
Sapt at given budget
at given budget
Power law: Supe
C
Power law: Vap
2
2d 2
2
2
2
2
Inference Compute per Problem (FLOPs)
Inference Compute Problem (FLOPs)
Figure 6: The optimal number of (left) solutions and (right) verifications for given compute budget, using Llama-3-8B and GenRM-FT. Every point corresponds to a compute budget. The plots show that as the budget scales, the optimal number solutions and verifications follows power law, with the number of solutions increasing more rapidly. See Appendix J Fig. 10 for Sopt and Vopt computation.
Takeaway
Our findings about and tasks. Improving verification quality (GenRM-FT) benefits performance at fixed compute.
SC and GenRM hold across model families, sizes,
5.2 Inference Scaling Laws for Generative Reward Models
Our experiments so far have shown that GenRM becomes a favourable choice as the compute budget increases, However, it is important to strike a careful balance between the number of solutions and verifications to achieve optimal performance at a given budget. This raises the question: What is the optimal way to allocate given compute budget between generating solutions and verifying them? To address this, we derive inference scaling laws, which describe how the optimal number of solutions and verifications scales with compute budget.
We sample up to 128 solutions per problem and up to 128 verifications per solution for the MATH test split using Llama-3. 1-8--.1-88-nntrucc and GenRM-FT. Then, we compute the performance at various values of S and (Figure Appendix 11a) and identify the optimal number of solutions and verifications for given budget. Subsequently, we study how the optimal number of solutions and verifications must be scaled as the budget is increased by fitting power law curves. Our findings in Figure 6 show that the optimal number of solutions scales as Supt & C while the optimal number of verifications scales as Vopt C0.39 The larger exponent associated with Sopt indicates that while both solutions and verifications should be scaled in tandem, solutions should be scaled at faster rate for compute-optimal performance. Further experiments in Appendix show that this finding holds across models.
Takeaway
We derive inference scaling laws for generative reward models along two axes: the number of solutions verify and the number of verifications per solution. For compute-optimal inference, the number of solutions should scale 1.5-2x faster than the number of verifications.
Discussion
In this work, we conduct compute-matched comparison between self-consistency and generative reward models. Specifically, we analyze performance under a fixed inference FLOP budget. A relevant direction for future iork to extend this analysis to fixed latency (tokens/ sec) setting across diverse hardware (e.g., GPUs/TPUS different VRAM configu- rations) and algorithmic approaches (e.g., batching, efficient attention). Additionally, our

---

Published as a conference paper at COLM 2025
compute budget calculations focus on the widely used decoder-only autoregressive large language models (LLMs). However, emerging LLM architectures, such as sub-quadratic models (Gu & Dao, 2023) and language diffusion models (Nie et al., 2025), present promis- ing directions for future exploration. A key bottleneck is the current lack of competitive aasoning-capable LLMs in these architectures. Finally, we evaluate the Best-of-N strategy with outcome-based verification using a generative verifier. Future work could extend compute-matcher analyses to alternative inference strategies, such as beam search or lookaround search (Snell et al., 2024). It is also important to note that self-consistency can only be applied to problems with definitive final answer. Therefore, domains such theo- rem proving may not be amenable to self-consistency, necessitating the use of alternative approaches like Best-of-N.
Conclusion
Generative Reward Models (GenRMs) introduce a novel approach to scaling test-time compute through verifications. While prior work demonstrates that scaling both solutions and verifications can surpass Self-Consistency (SC), it often overlooks verification costs. In this study, we investigate whether scaling verifications improves performance under a fixed budget. We find that SC outperforms GenRMs at lower budgets, whereas GenRMs excel at higher ones. Our conclusions regarding the compute-optimality of SC and GenRMs across different budgets remain robust across various model families (including thinking models), sizes, and reasoning tasks. Furthermore, we derive inference scaling laws to optimize budget allocation between solutions and verifications in GenRM. Overall, our findings provide practical guidance for compute-efficient scaling to achieve optimal performance.
Acknowledgements
Hritik Bansal is supported in part by AFOSR MURI grant FA9550-22-1-0380. Nishad Singhi supported by LOEWEStarr-Profesuu (LOEWE /S1//////////9////// /519 /1/5//900000 001.//0000...000000000000 (0000000000)/94 /94). Marcus Rohrbach is supported in part by Alexander von Humboldt Professorship in Multimodal Reliable AI, sponsored by Germany's Federal Ministry for Education and Research. We thank Ashima Suvarna, Xueqing Wu, Hector Garcia Rodriguez, Jonas Grebe, Tobias Wieczorek, and the anonymous reviewers for their helpful comments. For compute, gratefully acknowledge support from the hessian.Al Service Center (funded by the Federal Ministry of Education and Research, BMBF, grant no. 01IS22091) and the hessian.Al Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/003,
References
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Am- manabrolu. Critique-out-loud reward models. arv preprint arXtt:2408.11791, 2024.
AoPS, Aime 2024 dataset, 2024. URL /tt.tt/t//rrreooooooomm///////ee 2024 AIME I,
AoPS. Aime 2025 dataset, 2025. URL httptp///arrroooooooomo////////t 2025 AIME I, II.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08003,, 2022.
Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training 1lm reasoners via compute-optimal sampling, arvio preprint Xi1:2408.11673, 2024.
10

---

Published as a conference paper at COLM 2025
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. ar preprint ar Xix:: 2024.
Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, and Sercan Ô. Arik. SETS: leveraging self-verification and self-correction for improved test-time scaling. CoRR, abs/ 2bb01.21199306 2025. doi: 10.48550/ A2I.A URL
48550/arXiv. 2501. 19306.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. arXiv preprint ar xrxx::2311.17311, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ar Xrv preprint arXio:2110..4168 2021.
Jonathan Cook, Tim Rocktäschel, Jakob N. Foerster, Dennis Aumiller, and Alex Wang. Ticking all the boxes: Generated checklists improve LLM evaluation and generation. CoRR, /b1000000036688 2024. doi: 10.48550/ 10.4855/AARXIV224111 ARXIV.2410.0300. URL httpp:://oi.. org/10 28550/arXiv.2410 03608.
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https://arxivi wttpaaawwwi...ga/////ba////////a 12948.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. 12 2023. URL https://zenodo. wwtp:/nnddddogor/ooeeeoreeee org//records/1225636
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: large language models can self-correct with tool-interactive critiquing. In The Twelffh International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openrevieww wtwprreeiiieeeeeoouuiiiiice. Sx038axjek.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. Xrv preprint arXiv:2310.0222) 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint ar arXiv:2407.21788 217.787, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arv preprint arXiv:2312.00752, 2023.
Etash Guha, Negin Raoof, Jean Mercat, Ryan Marten, Eric Frankel, Sedrick Keh, Sachin Grover, George Smyrnis, Trung Vu, Jon Saad-Falcon, Caroline Choi, Kushal Arora, Mike Merrill, Yichuan Deng, Ashima Suvarna, Hritik Bansal, Marianna Nezhurina, Yejin Choi, Reinhard Heckel, Seewong Oh, Tatsunori Hashimoto, Jenia Jitsev, Vaishaal Shankar, Alex Dimakis, Mahesh Sathiamoorthy, and Ludwig Schmidt. Evalchemy: Automatic evals for Ilms, November 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. ar arXiv preprint ar arXiv:2501.1194. 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ar Xiv preprint ar arXin:2103.03874, 2021.
11

---

Published as a conference paper at COLM 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXio preprint ar xrx::22003.15556, 2022.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXie preprint arXio:2402.06457, 2024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. ar Xiv preprint arXio:2410.21276 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contami- nation free evaluation of large language models for code. arXio preprint arXtv:2403.07974, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural 2020.
language models. Xiv preprint ar Xiv::2001
Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sang- doo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine- gen-ggr evaluation capability in language models. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:: /otpnreie/peeevieww... net
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Kuang-Huei Lee, Ian Fischer, Yuch-Hua Wu, Dave Marwood, Shumeet Baluja, Dale Schuur- mans, and Xinyun Chen. Evolving deeper LLM thinking. CoRR, abs/ /2501.0909,, 2025. doi: 10.48550/ ARXIV.2501.0988 09999.. URL https: //doi. org/10. 48550/arXiv..... 2501. 01551.09891.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision. CoRR, abs/ babb:/240..06e 2406.0659, 2024. doi: ARX1V.24000006222 URL / /ttt:::://oii.or//ee.. 48550/ arwiv. 2406. 06592.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castri- cato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arxiv preprint ar Xiv:2410 1.8322............ 2024.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint ar arXiv:2502.09992, 2025.
OpenAI. Learning to reason with 1lms, 2024. thamiing-t-
URL https::/openii http:///ppenaa.ccoomnee//e///
Qwen. Qwq-32b:
Embracing the power of reinforcement learning, 2025. URL https: //qwenlm. gi thub. 3//b
12

---

Published as a conference paper at COLM 2025
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024.
Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or is suboptimal, 2025. URL https://arxiv.orr org//bs/ 2502. 12118.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters. arv preprint arXiv:2408.03314, 2024.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. arxiv preprint ar Xrio:2403.09472 2024.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 9426-9439. Association for Computational Linguistics, 2024. doi: (0.18653/V1/22 /tt//////22..AA-LOG...... URL https://doi.org org/ 10. 18653/v1/20244 /865/////222..cc11ooog.--- acl-long..11.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. Xiu preprint ar axXtt:2208.11171 2022.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. Xrviv preprint ar Kkt::2408.00722 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint ar dXXiv22412.1511, 2024.
Fei Yu, Anningzhe Gao, and Benyou Wang. Ovm, outcome-supervised value models for planning in mathematical reasoning. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 858-875 875. Association for Computational Linguistics, 2024. doi: 10.18653 WTA///2.FFNIINNGNNAAAAAAA v1/2024. f findings-naacl. 55.
18653/ URL https://doi. org/10 /
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels, 2024. URL
01981
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint ar XrXiv:2408.55200, 2024.
Eric Zhao, Pranjal Awasthi, and Sreenivas Gollapudi. Sample, scrutinize and scale: Effective http://rxxiv..rrr/////22/2
inference-time search by scaling verification, 2025. URL 01839.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lim-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gon- zalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In
13

---

Published as a conference paper at COLM 2025
Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10 16, 2023, 2023b. URL http: http://aappppepeaaaaee nips. ccpppppppppppeeppppprepppp files/ /ales/paper/223/// 2223/h/222//ash/ 91f18a1
Dasss _nd Bedchhansssrrs..... html.
14

---

Published as a conference paper at COLM 2025
A Related Work
Test-Time Compute Scaling. Leveraging more test-time compute to improve the per- formance of LLMs has gained a lot of popularity. Recent studies have explored various methods to scale test-time compute. A widely recognized baseline technique is repeatedly sampling candidate solutions from a model to choose the most frequent answer (aka self- consistency or majority-voting) (Wang et al., 2022). However, recent studies are pushing beyond this, investigating methods that leverage LLMs to iteratively refine their generated outputs (Gou et al., 2024; Cook et al., 2024; Lee et al., 2025). Reasoning models, such as OpenAI o3 series (OpenAT, 2024) and DeepSeek R1 (DeepSeek-A) AI, 2025) have enabled sequential scaling of test-time compute by scaling the length of the generated CoT (rather than parallel scaling by generating multiple shorter candidate solutions). While these long CoTs may implicitly incorporate forms of reflection, verification, refinement within their extended reasoning sequence, such models and previous studies do not primarily address the compute optimality of their proposed methods, the main focus of our investigation.
Verification. Another common method to scale inference-time time compute is to score candidate solutions through verification, which can be achieved via several techniques. Traditionally, discriminative models are employed, trained via binary classification (Cobbe et al., 2021; Luo et al., 2024; Yu al., 2024) or preferences (Hosseini et al., 2024; Yuan et al., 2024). Generative verifiers frame verification as a next-token-prediction task, enabling the use of CoT reasoning and another axis increase inference-time compute, either with trained verifiers (Zhang et al., 2024, Mahan et al., 2024; Ankner et al., 2024) or simply via prompting an off off-the-shelf LLM (aka LLM-as-a-judge) (Bai et al., 2022; Zheng et al., 2023a; Chen al., 2025; Kim et al., 2024; Zheng et al., 2023b) or self-verification based sampling (Zhao et al., 2025). These studies evaluate candidate solutions at outcome level, rather than process level verification (Lightman et al., 2023; Wang et al., 2024). Discriminative verifiers have become less favored due to the inherent challenges in their training and their tendency to exhibit lower performance compared to generative approaches (Zhang et al., 2024). Existing verification-basse scaling studies focus on improving accuracy but ignore the overall cost of adding more verifications. They are not concerned with the compute optimal setting about spending one's budget on generating more candidate solutions or more verifications for existing solutions.
Inference Scaling Laws. We can better allocate resources by understanding the characteris- tics of different inference scaling strategies. Snell et al. (2024) study how scaling test-time compute, through search against dense rewards and adaptive response updates impacts reasoning performance, revealing prompt difficultly as key factor. In contrast, our work takes into account the verification budget and studies the trade-off between allocating compute to generate new candidate solutions versus verifying existing ones. Wu et al. (2024) investigate compute-optimal scaling, specifically examining the trade-off between model size and generating multiple samples. Consistent with findings in Wu et al. (2024); Brown et al. (2024); Bansal et al. (2024), they demonstrate that sampling multiple times from a smaller model can outperform a larger and stronger model within a fixed budget. However, we focus our scaling analysis m the trade-off between generating additional solution candidates and generating verifications for existing solutions. Setlur et al. (2025) argue that test-time compute scaling without verification is suboptimal which is consistent with our overall findings.
B Inference Details
Following Brown et al. (2024), we use vLLM (Kwon et al., 2023) with up to 16 A100 GPUs to generate solutions and verifications from LLMs.
Math Datasets Following Brown et al. (2024), we use 4-shot prompt to generate solutions (Prompt K.1). We use the minerva_mati function from LMEval (Gao et al., 2023) to extract the final answer from the solution, and the is_equiv function to check if the answer matches the ground-truth answer. For GenRM-base, we use a 2-shot prompt to generate verifications (Prompt K.2). For G-n we use Prompt K.3 for both fine-tuning and inference.
15

---

Published as a conference paper at COLM 2025
GPQA We perform experiments on subset of 64 randomly sampled problems from the diamond split of GPQA. We use zero-shot prompt to generate solutions (Prompt K.5). We sample with temperature of 0.7 and with the maximum number of tokens set to 1024. For GenRM-Base, we use a zero-shot prompt to generate verifications (Prompt K.6) with temperature of 01. and maximum of 1024 tokens.
C GenRM Training Details
We use the solution generator, e.g,, 1aama-3.1-8B-Instuude to generate 4 solutions for prob- lems in the MATH training split (7500 problems), using the prompt described in Appendix B. Then, we provide these solutions to GPTT-4o along with the ground-truth solutions, and generate verifications for every solution using Prompt K.4. We filter out the verifications whose verdict doesn't t match the ground-truth correctness of the solution. Further, sample verifications such that the number of correct and incorrect samples is balanced. We LoRA (Hu et al., 2022) fine-tune the same model this dataset of synthetic verification rationales for 3 epochs. We pick the learning rate from {5e 7, le 6, le 5} based on accuracy on validation split (10% of training split).
D Evaluation Details
Reliable Estimation of Best-of-N Following Hosseini et al. (2024), we estimate the average success rate by estimating the probability that when sampling k out of N(> k) solutions, the one with the highest score is correct, and averaging it over K repetitions:
N- N-i-11 Best-of-k = Σ
(1)
1
where [no, 41, -N- are the binary correctness scores (0 1) for the candidate solutions sorted in decreasing order of their verifier scores.
E Impact of Verifier Quality
Our results indicate that using a small number of verifications does not lead competitive performance. In most cases, at least 32 verifications are needed to achieve good results, which significantly increases computational cost. This suggests that individual verifications may be noisy. If verification quality improves, however, then fewer verifications might be required to achieve the same performance.
To investigate this, we compare two generative verifiers: (1) "strong" verifier: Llama- 3.1-8B-Instruct fine-tuned on GPT-4o verification rationales (GenRM-FT), and (2) "weak" verifier: 1amma-3..-BB-IIstteee
using a two-shot verification prompt (MenRM- on the MATH dataset. We use À = for GenRM-Base and À = 2 for GenRM-FT.
Figure 3b shows how these verifiers perform as the number of solutions and verifications increases. For clarity, we only plot the best-performing configurations (number of solutions and verifications) for given compute budget. Our findings reveal that across the board, the strong verifier achieves similar performance with significantly less (up to 16x) compute than the weak verifier. Moreover, the strong verifier outperforms the weak verifier overall.
These results underscore the importance of high-quality verification rationales. As the verification capabilities of LLMs improve the future, the compute required for GenRM to achieve strong performance may decrease.
F Improving Verifier Efficiency
In Section 5.1, we observed that improving the verification capabilities of GenRM via fine- tuning reduces its inference compute requirements, thereby enhancing efficiency. In this
16

---

Published as a conference paper at COLM 2025
section, we conduct preliminary explorations of alternative ways to improve GenRM's efficiency. Specifically, we focus on two approaches to reduce inference compute: using smaller verifiers and training verifiers to generate fewer tokens.
First, we use LLaMA 3 70B to generate solutions on the MATH benchmark. Keeping these solutions fixed, we consider two verifiers: LLaMA 3 70B GenRM-Base and LLaMA 3 8B as GenRM-FT. As discussed in Section 4, Gen has À = 1, so the compute required to generate a verification using GenRM-Base is equal to that required to generate a solution. In contrast, the smaller GenRM-FT has À = 2, meaning the compute required to generate verification is 2 x 8/70 0.228 times that required to generate solution. Thus, generating verifications using GenRM-FT is significantly cheaper than generating solutions. We compare these two verifiers against SC in Figure We find that while the smaller verifier reduces the compute needed to match SC. from 8x to 4x, it still does not outperform SC at low compute budgets.
As discussed in Section 3, the inference compute for verification is proportional to the number of tokens generated. Hence, we can further reduce the compute required for verification-anny and consequently for GenRM-by generating fewer verification tokens. Next, we analyze the impact of improving verifier efficiency through shorter verification outputs.
To estimate the upper bound on potential efficiency gains from generating fewer tokens, vary the value of Å while assuming constant verification quality. Note that this assumption is made solely for analytical purposes; in practice, training verifiers that maintain the same accuracy while generating fewer tokens is non-trivial. In Figure 1, we used LLaMA 8B as GenRM-FT on the MATH benchmark, where each verification uses 2048 tokens on average. In Table 1, we compare GenRM against SC while varying the number of verification tokens, assuming constant verification performance. We find that GenRM would need to reduce the number of verification tokens from 2048 to 256 (i.e., an 8x reduction) to match the compute requirements of SC. Note that here we use the same model sizes (SB parameters) for generating solutions and verifications and only change the number of verification tokens.
The results in these sections show that training verifiers that are smaller and generate fewer verification tokens is promising direction to improve the efficiency of GenRM. However, further research is required to bridge the gap between GenRM and SC.
Table 1: Comparison of GenRM performance against SC with varying values of verification tokens. We find that current GenRMs would have to generate 8x fewer tokens, while maintaining their performance, to match the performance of SC.
Tokens per verification
Compute to match SC
Compute to outperform SC
2048
8x
128x
1024
4x
64x x
512
2x
32x
256
1x
16x
G Results on Code Generation
In this section, we extend our comparison between Self-consistency and GenRM to code generation tasks. We note that is not straightforward to apply Self-consistency to code generation because the task is to generate an entire piece of code, not just final answer. To apply Self-consistency to this task, we follow the methodology of Chen et al. (2023) and perform majority voting over the outputs of the generated code for the given test inputs. Specifically, we use Llama 3.1 8B Instruct to generate solutions and verifications as GenRM-Base on 128 problems from the LiveCodeBench (LCB) benchmark (Jain et al., 2024). Our results in Figure show that GenRM requires 16x more compute to match the performance of SC, and 64x more compute to achieve a 3.5% improvement.
17

---

Published as a conference paper at COLM 2025
SC
-
Large GenRM (Liama 708)
- Small GenRM (Llama 8B)
Comparison between Small vs Large Verifier for Fixed Generator
78
76
4s more compout
Figure 7: Comparing small (Llama 3 8B as GenRM-FT) and large (Llama 3 70B as GenRM- Base) verifiers against SC on MATH. While the smaller ver- ifier more efficient than the larger verifier, it still does not outperform SC at lower budgets.
74
Bc Micoooe For pure
Success Rate
72
70
66
64
62
2'
2
211
Inference Compute per Problem (FLOPs)
Liama 3.1 LCB128
21
6ds more compute
Figure 8: Comparing Re Base against Self-consistency (SC) with Llama-3.1-BB-Instruue
20
+3.5%
SC is better at lower budgets while GenRM outperforms at higher budgets, suggesting that our find- ings generalize to code genera-
18
more compute
8
17
GenRM (32 CoT)
16
tion.
14
inference Compute per Problem (FLOPs)
H Trends Across Model Families
In Figure 9, we present a compute-matchess comparison between SC and GenRM-FT with on the MATH dataset. Similar to previous observations, we find that SC outperforms GenRM-FT at lower budgets, whereas GenRM shines higher budgets.
I Inference Details For Thinking Model
In this work, we evaluate QwQ-32B on the challenging AIME2025 benchmark." We use Prompt K.7 (Guha et al., 2024) and Prompt K.8 to generate solutions and verifications, respectively. Since this model generates much longer reasoning traces than traditional instruction-tundd models, we generate maximum of 32768 tokens (A 1) with tem- perature of 0.7 for both solution and verification tasks. We note that the compute required to generate the thought process within the <think> tokens is included in the solution and verification token budget.
J Additional Details and Results on Compute-Optimal Scaling Analysis
In this section, we derive scaling laws for Owwen-2.5-B-IInstruu and 1lama-33-77B-Instruu the MATH dataset. For Qwen-2.5-7B-Instruus cations using GenRM-FT, Our findings in Figure 12 show that as the budget is increased, the optimal number of solutions scales as S & C and Vopt & C0332 For Llama-3.3- 70B-Instruct, we sample 64 solutions and 64 verifications using GenRM-Base. Figure 13 shows that Sopt & C and Vopt & CO.43 These results show that as test-time compute is scaled, the number of solutions should be scaled more rapidly as compared to the number of verifications.
sample 128 solutions and 128 verifi-
We observed significant performance drop on AIME2U25 in comparison to MATH and AIME2024. This highlights that the scope of improvement is higher with verification in comparison to saturated datasets.
18

---

Published as a conference paper at COLM 2025
SC
V-21
V=2
V-2*
V=2
V=2²
V=24
GenRM-FT Qwen-2.5-7B-Insttee (MATH)
512x compute
Figure 9: Comparing GenRM-FT against Self-consistency (SC) with Qwen-2.5-7B- Instruct. SC is better at lower budgets while GenRM outperforms at higher budgets, suggesting that our findings hold across model families.
70.0
Success Rate (%6
67.5
5.44
64x compute
65.0
62.5
60.0
2
2
2
2
2
Inference Compute per Problem (FLOPs)
Scaling GenRM at Fixed Num. Solutions
The curve for Sw2? has highest success rate at budget=2'. Honce,
54
Success Rate (%)
52
$ =2
50
S=23
S=24
48
S=
46
=2* this range of budgets
44
42
24
25
26
2
2
29
Inference Compute per Problem (FLOPs)
Figure 10: Toy illustration of how compute Sopt for given budget. Every corresponds to fixed number of solutions, with the number of verifications increasing along x-axis. any budget on x-axis, we find the curve that has the highest success rate at that budget. For instance, a budget of 26, the value Sppt increases in step-changes, as the number optimal solutions must be an integer. The
of 2 has the highest success rate, hence, Sopt 2 this budget. Further, the
figure shows that Supt = 24 for budgets between x 27 and ss 28. Vopt can computed analogously from a plot of fixed verifications and increasing number of solutions.
19

---

Published as a conference paper at COLM 2025
Llama-3-8B w/ G-RRM-FT MATH
S=2
S=2
S=21
S=2?
V-21
V-
V=
V-2
S-22
S-2'
S-2
V=22
V=2
V=26
GenRM Inference Scaling -Varying Num. Verifications
GenRM Inference Scaling -Vayiing Num. Solutions
55
Success Rate (%)
Success Rate (%)
2
20
21
2
210
Inference Compute per Problem (FLPSs)
Inference Compute per Problem (FLOPs)
(a) Scaling trends of GenRM at (Left) a fixed number of solutions and increasing the number of verifications, and (Right) fixed number of verifications and increasing the number of solutions.
Optimal Num. Solutions at Budget
Optimal Num. Verifications at Budget
Optimal #Verifications (o))
Optimal (Sue))
24
Pengous
24
Steefie
(a)
(b)
23
Supt at given budget
Vept at given budget
Power law: Sux C
Power law: Van C0.30
2n
2
2
210 21
2
2
Inference Compute per Problem (FLOPS)
Inference Compute per Problem (FLOPs)
(b) The optimal number (a) solutions and (b) verifications for a given compute budget. Every point corresponds to compute budget. The plots show that as the budget scales, the optimal number of solutions and verifications follows power law, with the number of solutions increasing more rapidly.
Figure 11: Compute-optimal scaling of solutions and verifications GenRM-FT with Llama- 3.1-8B-Instruct on MATH.
20

---

Published as a conference paper at COLM 2025
Qwen-2-7B w/ GenRM-FT MATH
S=21
S=2²
S=2²
S=2?
V-21
V-2
V-2²
V-2
S=2?
S=21
S=24
V=22
V=2*
V=2
GenRM Inference Scaling Varying Num. Verifications
GenRM Inference Scaling Varyyng Num. Solutions
70
68
Success Rate 1%6
68
Success Rate
66
64
62
22
2+
2
2
220
2
2
2
2
2550
2
2
Inference Compute Problem (FLOPs)
Inference Compule Problem (FLOPs)
(a) Scaling trends of GenRM at (Left) fixed number of solutions and increasing the number of verifications, and (Right) fixed number verifications and increasing the number of solutions.
Optimal Num. Solutions Budget
Optimal Num. Verifications Budget
Opümal #Verificat (Vose)
Optimal #Solutions (Snge)
2
24
24
Soess
2
22
6600
Power law: Sup C
00000000
Power law: Vaea 0.33
2 2
2
2
2
2
n
2
2
Inference Compute per Problem (FLOPS)
Inference Compute Problem (FLOPS)
(b) The optimal number of (a) solutions and (b) verifications for a given compute budget. Every point corresponds to a compute budget. The plots show that as the budget scales, the optimal number of solutions and verifications follows a power law, with the number of solutions increasing more rapidly.
Figure 12: Compute-optimal scaling of solutions and verifications in GenRM-FT with Qwen- 2.5-7B-Instruce on MATH.
21

---

Published as a conference paper at COLM 2025
Llama-3-70B w/ GenRM-Base on MATH
V=2
V=2
V=2
V=2*
S-2²
S-2²
V-2²
V-2!
S=2²
S-24
GenRM Inference Scaling Varying Num. Solutions
GenRM Inference Scaling Varying Num. Verifications
78
78
Success Rate (%6
Success Rate 1%))
76
76
74
74
72
72
70
70
2
2
2
2
2
2
2
Inference Compute per Problem (FLOPs)
Inference Compute per Problem (FLOPs)
(a) Scaling trends of GenRM at (Left) fixed number of solutions and increasing the number of verifications, and (Right) fixed number of verifications and increasing the number of solutions.
Optimal Num. Solutions Budget
Optimal Num. Verifications Budget
Optima #Solutions (Snge)
26
Power law: S 0.69
Optimal #Verilicatonn (Var)
26
24
24
22
22
Power law: V 0.43
20
20
2
2
2
2
2
Inference Compute per Problem (FLOPS)
Inference Compute Problem (FLOPS)
(b) The optimal number of (a) solutions and (b) verifications for a given compute budget. Every point corresponds a compute budget. The plots show that as budget scales, the optimal number of solutions and verifications follows a power law, with the number of solutions increasing more rapidly.
Figure 13: Compute-optimal scaling of solutions and verifications in GenRM-Base with Plama--.3-70B-Insttee on MATH.
22

---

Published as a conference paper at COLM 2025
K
Prompts
K.1
Generating Solutions to Math Problems
4-shot Prompt for generating solutions to math problems
Problem:
Find the domain of the expression $1 (5-x))s.)
Solution:
The expressions inside each square root must be non-negative. Therefore, $x-2 Alge 8s, so $x\\ge25, and $5 x Agge 0s, so $x Wle 5$. Also, the denominator cannot be equal to zero, so $5-x >0s, which gives $x<5$. Therefore, the domain of the expression is (12, (.).e((,,)))..iiin Answer: The final answer is $ hope is correct.
Problem:
If S\ddet (mathbf(A) = 2$ and sVedtt \mathbb(() 12, then find $ildet (N\mathbf(A) Mmathhb(BB)..
10 Solution:
We have that $\Ndet (VmathbffAA (A) (1mathb(BB)) = (Wedet Amathbf SAnFinal The final answer is $245. hope it is correct.
A)) (VVdet (lmathbf(B)) (2) (12) (24) (2nnn.i Answer:
12
13 Problem:
Terrell usually lifts two 20-pound weights 12 times. If he 15-pound weights instead, how many times must Terrell lift them order to lift the same total weight?
15
16. Solution:
If Terrell 1ifts two 20 20-pound weights 12 times, he lifts total $2\\edot 12V\edot20=4805 pounds weight. If lifts two 15-pound weights instead for $n$ times, he will lift a total of $2Vicdot15\\cdor n=30n$ pounds of weight. Equating this to 480 pounds,
solve for $n$: (ntee//eiiiiiiiignnnnnnne----e Rightarrow Aqquad n8=480/30=| (1)) Answer: The final answer is $16$. I hope it is correct.
18
19 Problem:
If the system of equations (itiiiosnnnnnleeiiigaagggnnnnao -y8=a, (a03=,,AWn888 Win6y-9 &=b. Anllend(align=))aa has solution $(x, y)s where sx$ and $ys are both nonzero, Anfind s(lfrac(a)(b),, assuming $b$ is nonzero.
21
Solution:
If we multiply the first equation by $-)1frac mls$6yy9xxx-Vffeec33311) $$5nnncce we (~V\frac22)33)) $$AnFinal Answer: The final answer is $-llfrac {2){)).. hope it is correct.
obtain $$since we also know that 56y-sx
24
25 Problem:
26
<Problem>
Solution:
23

---

Published as a conference paper at COLM 2025
K.2 GenRM-Base
Few-shot GenRM-Base prompt for math problems
Problem:
Find the domain of the expression $WVrrac(ssartxx22333 (5-x))s.)
Solution:
The expressions inside each square root must be non-negative. Therefore, $x-2 Alge 8s, so $x\\ge25, and $5 x Agge 0s, so $x Wle 5$. Also, the denominator cannot be equal to zero, so $5-x >0s, which gives $x<5$. Therefore, the domain of the expression is (12, (.).e(l,,)))..iiin Answer: The final answer is $ hope is correct.
Problem:
If S\ddet (mathbf(A) = 2$ and sVedtt \mathbb(()) 12, then find $ildet (N\mathbf(A) MMathhb(BB)..
10 Solution:
We have that $\Ndet (VmathbffAA (A) (1mathb(BB)) = (Wedet Amathbf SAnFinal The final answer is $245. hope it is correct.
A)) (VVdet (lmathbf(B)) (2) (12) (24) (2nnn.i Answer:
12
13 Problem:
Terrell usually lifts two 20-pound weights 12 times. If he 15-pound weights instead, how many times must Terrell lift them order to lift the same total weight?
15
16. Solution:
If Terrell 1ifts two 20 20-pound weights 12 times, he lifts total $2\\edot 12V\edot20=4805 pounds weight. If lifts two 15-pound weights instead for $n$ times, he will lift a total of $2Vicdot15\\cdor n=30n$ pounds of weight. Equating this to 480 pounds,
solve for $n$: hats:///lleiiiiiiin///33------ Rightarrow Aqquad n8=480/30=| (1))
Answer: The final answer is $163. I hope it is correct.
18
19 Problem:
If the system of equations (igiioossnnnnllleeigaaaggnnnnos 448=a, M003=,,AW8888 Win6y-9 &=b. Anllend(align=))aa solution $(x, y)s where sx$ and $ys are both nonzero, Anfind s(lfrac(a)(b),, assuming $b$ is nonzero.
21
Solution:
If we multiply the first equation by $-)1frac mls$6yy9xxx-1ffeec33311) $$5nnncce we (~V\frac22(33)) $$AnFinal Answer: The final answer is $-llfrac {2){)).. hope it is correct.
obtain $$since we also know that 56y-sx
24
25 Problem:
26
<Problem>
Solution:
24

---

Published as a conference paper at COLM 2025
K.3 GenRM-FT
Prompt used for GenRM-FT
Problem: (
Solution:
()
Verification:
K.4
Prompt to Generate Synthetic Training Data
Prompt used to generate synthetic verification rationales for training GenRM
You are math teacher. Grade the Solution, verifying correctness step by step. At the end of the Solution verification, when you give your final grade, write it the form 'Verification: Is the answer correct (Yes/No)? X', where is either Yes or No.
Example
Question:
<Question>
Solution:
<Solution>
10
11 Expected Solution:
<Ground-truth Solution>
13
14
15 Teacher Verification:
<rround-truth verification rationale>
18
Verification: Is the answer correct (Yes/No)?
19
20
22
Example
23
24
Question: <Question>
27 Solution:
28
<Solution>
29
Expected Solution:
31
<Ground-truth Solution>
32
33
Teacher Verification:
<rround-truth verification rationale>
36
37
Verification: Is the answer correct (Yes/No)? No
38
25

---

Published as a conference paper at COLM 2025
10
41
Now, continue grading the next solution step-by-step as follows.
42
Question: (
44
45
Solution: 0
46
47
Expected Solution: ()
19
Teacher Verification:
K.5 GPQA Solution
Prompt used to generate solutions to GPQA problems
Think step-b to solve the following problem. Only include the letter choice
(A, B, or D) your final response. End your answer with Final Answer: The final answer is x. I hope it is correct.'' where X is your final answer.
Problem: <Problem>
Options: <Options>
Answer:
K.6
GPQA Verification
Prompt used to generate verifications to GPQA solutions with GenRM-Base
You are teacher expert in physics, biology, and chemistry. Grade the Solution, verifying correctness step by step. At the end the Solution verification, when you give your final grade write it in the form 'Verification: Is the answer correct (Yes /No)? X', where x is either Yes or No.
Question: 0
Solution: 0
Teacher Verification:
K.7
Thinking Models Solution Prompt
Prompt used to generate solutions from reasoning models
Problem: (A
Mark your solution with Abboxed \ nAnswer:
26

---

Published as a conference paper at COLM 2025
K.8
Thinking Models Verification Prompt
Prompt used generate verifications from reasoning models as GenRM-Base
** **Question:** O
**Student Solution O
Math Solution Grading Instructions
As math teacher, your role is to grade student solutions following these standardized steps:
## Verification Process
10 1. Perform a detailed verification by:
Checking each step sequentially Verifying calculations
12
13
Validating mathematical properties and rules used Examining the logical flow between steps
14
Confirming proper notation and mathematical writing
17
Error Documentation
18
2. For any errors found:
19
Point out the specific location of the error Explain why it's incorrect
21
Show the correct approach
22
Note if the error affected subsequent steps
23
24 ## Assessment Format
3. End your answer in this format:
26 "Verification: the answer correct (Yes/No)? x". Where is Yes or No.
27
#*Verffictionn:
Alright, let 's verify the solution step by step.
Verification Example from GPT-4o
Verification Example from GPT-4o
Problem:
A street has 20 houses on each side, for a total of 40 houses. The addresses on the south side of the street form an arithmetic sequence,
do the addresses On the south side, the addresses the north side of the street. 4, 10, 16, etc., and the north side they are 3, 9, 15, etc. sign painter paints house numbers on a house for $1$15 per digit. If he paints the appropriate house number once on each of these 40 houses, how many dollars does he collect?
Solution:
There are $20$ houses on each side, so the painter will paint
total of $40$ numbers. The house numbers on the south side are $4, 10, 16, 22, 28, 34, 40, 46, 52, 58, 64, 70, 76, 82, 88, 94, 180 106, 112, 118$ and the house numbers on the north side are $3, 9, 15, 21, 27. 33, 39, 45, 51, 57, 63, 69, 75, 81, 87, 93,
99, 105, 111, 1173. Among the house numbers on the south side,
the number of digits in each number is follows:
27

---

Published as a conference paper at COLM 2025
have digit.
10, 22, 28, 34, 46, 52, 58, 64, 70, 76, 82, 88, 94, 108, 106,
112, 118 have 2 digits.
-16, 26, 36, 46, 56, 66, 76, 86, 96 have 2 digits.
58, 68, 78, 88, 98 have 2 digits.
10 64, 74, 84, 94 have 2 digits.
11 100, 110 have digits.
12 Among the house numbers
the north side, the number of digits in each number is as follows: and and 93 and 123 have digit.
13
14 9, 21, 39, 51, 69, 81, 99 have 1 digit.
15 -15, 27. 45, 57, 75, 87, 105 have 2 digits.
16 23, 35, 47, 59, 71, 83, 95, 107 have digits.
25, 49, 61, 73, 85, 97 have 2 digits.
18 29, 41, 53, 65, 17, 89, 101, 113 have digits.
19 43, 55, 67, 79, 91, 183, 115 have digits.
20 -55, 67, 79, 91, 183, 115 have digits.
-19 has 3 digits.
22 Since there
one-digit number, two-digit numbers, and three-digit number among the house numbers on the south side. the total amount of money the painter will collect is cdoc (2Ncott 13)=ot\ 1)= dollars. Similarly, since there are 5 one-digit numbers, two-digit numbers, three- digit number among total amount of money the painter will collect is $(1Ncdot 5) c22cdot 7) (3)cdot (22) dollars. Therefore, the total amount of money the painter will collect is 39+22=61 dollars. 61. hope it is correct.
house numbers on the north side, the
Final Answer: The final
24
25
26
Verification: Let's verify the student 's solution step by step against the expected solution:
28
29
### **Step Problem Understandingxx
32
33 The problem asks for the total amount of money collected by a
sign painter to paint house numbers on 40 houses, divided into
20 houses the north and south sides of the street. Each house has unique address, determined by the arithmetic sequence provided for each side. The painter charges $1 per digit when painting house number.
34
The problem clearly requires to:
36 Identify the numbers on each side of the street.
37 2. Group these numbers by the number of digits (1-digit, 2-digit or 3-digit).
3. Calculate the total cost side of the street, then find the sum both sides.
39
40
The expected solution uses the arithmetic sequence formulas and organizes the numbers by digit count. The student 's solution tries to achieve this goal but deviates significantly in implementation.
41
42
44
**Step Address Lists**
45
46
##*# S*Sth Side:** **
28

---

Published as a conference paper at COLM 2025
17
Numbers belong to the arithmetic sequence \(4, 10, 16, Adots,
1181).
48
Common difference \(d 61).
Using the \(n))-th term formula \(a_n (n-1)d(), the (200 (ttex((t)))) term is:
50
51
a_ (20) = 4 6(28-1) 4 6(19) 4 + 114 118.
52 \]
The numbers confirmed to be \(4, 10, 16, Adots, 118)).
54
55
#*** **rtt :
56
Numbers belong to the arithmetic sequence \(3, 9, 15, Adots,
117)).
Common difference A(d 61).
58
Using the V(n))-th term formula \(a_n a (n-1)d(), the (28 (oo"((tee ( (Ntext(th)))) term is:
59
A[
[_0) = 6(20-1)
6(19) 3 + 114 117.
61
A]
62
The numbers are confirmed to be \33, 9, 15, \dots, 117)).
63
The student constructs these lists correctly in their solution.
**Step 2 is correct.
66
67
### **Step 3: Counting 1-digit, 2-digit, and 3 3-digit Numbers**
70
student attempts to classify the numbers into groups based on digit count. Let's 's verify the classification for each side based on the expected solution:
72
#*## **ututh Side:**
73
d*i-digit numbers**: A(4)) (1 number).
74
**2-digit numbers**: From \(10)) to (941), inclusive.
25
Using the \(n))-th term formula for the first two-digit (N(10V))
and last two-digit (\(941)):
76
77
10 = 4 6(n-1) Nimplies n 2.
78
AJ
A[
6(n-1) Aimplies 16.
81
\]
82
So there are \(16 2 + 1 151) two -igit numbers.
ax3-digit numbers**: From \(101) to \(118), inclusive. There
\(20 16 41) three-digi- numbers.
84
85
Thus:
86
d-iiit:: \(1)) number,
2-digits: N(15)) numbers,
3-digits: M(4) numbers.
89
#*** **North Side:** 90
- -xl-digit numberswx: A(3, 91) (2 numbers). **i-digit numbers**: From \(15)) (991)) inclusive. Using the M(\))-th term formula for the first two-digit (v(15V)) and last two two-digit (1(991)):
93
944
15 = 6(n-1) Aimplies 3.
96
97
98
99 = + 6(n-1) Nimplies 17.
A]
29

---

Published as a conference paper at COLM 2025
there are V(17 **3-digit numbers**: From \(105)) to \(117)) inclusive. There are \(20 17 = 31) three -igitt numbers.
151) two two-digit numbers.
101
Thus:
1-digit: ((2) numbers,
105
2-digits: \(15)) numbers,
3-digits: A(3N) numbers.
The student has errors their count. For example:
109
They incorrectly list certain numbers as having the wrong number of digits.
They double-count mislabel numbers. (For instance, \(41) is listed having multiple digit counts.)
112
**Sp 3 is incorrect** due to significant misclassification
116
#** **Step Cost Calculation**
117
Using the corrected counts:
South :
121
AE
(Cost(Cost) (1 Aedot 1) + (2 Acdot 15) + (3 Acdot 4) = 30
+ 12 43.
\]
124
125
#*## ** Side:** **
\[ \text(Cost) (1 Nedot + (2 Aedot 15) + Vedot 3) 2 30 41.
128
\]
#*## **Total Cost:**
N[
132
(Text(T \J
41 84.
133
The student's calculation is incorrect. They arrive at total of ((61)), which significantly underestimates the total due to their misclassification of numbers.
136
S*Step 4 is incorrect*x due to incorrect cost calculation.
140
#** **Step 5: Final Verification and Grade**
The student correctly identified arithmetic sequences. However, they made significant errors counting the number of digits for many house numbers on both the north and south sides. These errors carried over into their cost calculation, leading to incorrect total.
144
146
*Verification: Is the answer correct (Yes/No)? No**
30


Reviews:
[{"review_id": "GDRATgwf2Z", "reviewer": "Reviewer_hnKb", "summary": "One is presented with two choices when it comes to scaling test-time compute for reasoning: scale a number of solutions and apply majority vote on top, or scale verification compute via generative verifiers. The paper investigates which choice is best suited under different compute budgets and shows that majority vote or self-consistency is preferred in the low-compute regime, but generative verification can dominate in high-compute regime. \n\nThe paper's main contribution are:\n1. compute-matched analysis of SC vs GenRM and some interesting findings related to that. \n2. scaling law for GenRM: showing that it is best to scale number of sampled solution twice as much as scaling number of verifications. \n\nThe authors run experiments on math and scientific reasoning tasks, and the first finding is shown to mostly hold with different model sizes.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Timely and practical problem: LLM inference cost is real, and test-time optimization matters. \nThe authors conduct a multitude of experiments to support their finding that majority vote is better at lower inference budgets, and the results are mostly consistent across settings. \n- The scaling law for GenRM could be useful but unclear whether it will generalize well.", "reasons_to_reject": "- It remains unclear what to do with these results. The authors do not propose a compute-optimal strategy to allocate compute to a given input problem. When the authors say \"SC is better at low compute, GenRM is better at high compute\" I'm inclined to ask what does \"low compute\" mean here? Obviously low compute means different things in under different model sizes, tasks, etc. In other words, the findings do not translate into a practical recipe to scale inference compute with GenRMs. The paper would substantially benefit from such investigation. \n- I'm concerned the results are impacted by the performance of the verifier used. The GenRM verifier used is purely outcome-based, and is trained by the authors. Using LLM-as-a-judge or GenRM base is known to not work well for reasoning. Plus, recent works [1] have shown that process-based verification or GenPRM can work better than GenRM. All this makes me doubt the generalizability of the findings. \n- While the findings may be useful in the meantime, it remains unclear whether they will hold in 1-2 years when GenRMs become generally more efficient and better optimized. \n- While GenRM indeed seems to catch up later, the improvements from GenRM remain very marginal e.g., +1.7% (MATH), +2.1% (GPQA). One risk here is that readers may come to the understanding that SC is almost always better, which is false since SC will only work in domains/tasks where the correct answer will dominate the incorrect answers. In more difficult or unseen domains, or in domains where there is no final answer to begin with (e.g., code generation or theorem proving) SC is not applicable and the paper findings become of much less value.\n\n\n\n[1] https://arxiv.org/abs/2504.16828", "questions_to_authors": "- How good is you GenRM? could you evaluate it on benchmarks such as ProcessBench (even just using outcome labels) and report results? \n- Will your findings translate into a compute-optimal strategy for reasoning with GenRM? If so, how?", "extra_scores": {}}, {"review_id": "UJbJOnRARE", "reviewer": "Reviewer_Kdvi", "summary": "This paper presents an empirical study examining two specific dimensions of test-time scaling in reasoning problems: (i) Sampling multiple solutions, combined with self-consistency voting  (SC) and (ii) Sampling multiple verification chains (with generative reward models/GenRM) for each solution. The authors argue that the computational budget for verification is often ignored in prior work, and that the overall computational budget for both solution generation and verification should be considered. In particular, the authors ask, under a specific computation budget, should one allocate more computation for sampling more solutions or instead sample more verification chains per solution. Through empirical studies on reasoning problems, the authors find that in low computational budget situations, SC leads to better performance, whereas, when more computation is available, scaling GenRM leads to better performance. The authors also present scaling laws for number of solutions and verification chains.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The empirical findings are insightful and provide practical guidance on effective test-time scaling.\n* Hypotheses are verified with comprehensive evaluation across multiple models, benchmarks and compute regimes.", "reasons_to_reject": "* Technical component of the paper is somewhat limited. It would have been interesting to see an approach that unlocks the benefits of GenRM at a lower compute regime, such as by fine-tuning small verifiers. \n* Although the authors provide a brief discussion on thinking models, they have been excluded from the main analysis, citing that controlling the amount of compute in such models is uncontrollable. However, I think they are an important part of the analysis. For example, with extended thinking, the model encapsulates the role of the RM and it would be interesting to see if it can be better than SC for a given compute budget. \n* Although SC seems superior for the tasks considered in this work, its applicability can be limited beyond problems that have a specific, well defined final answer, which makes me question the scope of these conclusions. How are the findings in this paper applicable to problems that have more open-ended responses?", "questions_to_authors": "* In the context of thinking models that encapsulate the role of verification during the process of solution generation, how are these findings/guidelines for allocating computational budget applicable/relevant?", "extra_scores": {}}, {"review_id": "ZZZIIDJF26", "reviewer": "Reviewer_CSkR", "summary": "The paper studies the performance of test-time scaling methods under different computation budgets. It conducts extensive experiments and compares two test-time scaling methods:  Self-Consistency (SC) and Generative Reward Models (GenRM). The results show SC performs better with a lower budget, and GenRM can achieve a higher success rate by generating CoT verification tokens. While the result holds across different model sizes and task difficulties, the authors give an empirical conclusion of the optimal ratio of solutions/verification numbers.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The experiments are solid and sufficient to support the claim. The authors evaluate various models on different tasks and settings to obtain the conclusion.\n\n- The optimal ratio of answer and verification sizes is useful for the community. The authors perform extensive experiments and give the empirical conclusion of balancing the answer and verification size. This result can help further work to obtain better results without further tuning.", "reasons_to_reject": "The conclusions do not provide much new insight or information. The authors only perform empirical studies to inspect the final performance, without further insight or analysis. More further questions need answer. For example, is there an upper performance bound given enough budget, and how much computation is needed to reach it, or some guidance to improve the performance by limiting each verification length?", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper explores tradeoffs between scaling test-time compute by generating more solutions or by spending extra computation validating existing solutions. In a large-scale empirical evaluation, it reports a trade-off between the two modes: for small compute budgets, it's better to draw many samples; for large budgets, it's better to spend more time verifying. There was an extensive discussion, during which many initial reviewer concerns were addressed. While some reviewers still have questions about the extent to which these findings would generalize to other answer scaling or verification scaling paradigms, there seems to be widespread agreement that the current results are interesting, timely, and likely to be useful to the community.
Decision: Accept