Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers
Authors: Shalev Lifshitz
Sheila A. McIlraith
Yilun Du
Venue: COLM 2025 (2025)
Keywords: large language models
test-time compute
verification
scaling
TL;DR: We explore scaling the number of verifier models as a novel test-time scaling dimension for improving language model performance and introduce an algorithm that enables simple scaling along this dimension.

Abstract: By utilizing more computational resources at test-time, large language models (LLMs) can improve without additional training. One common strategy uses *verifiers* to evaluate candidate outputs. In this work, we propose a novel scaling dimension for test-time compute: *scaling the number of verifier models*. We introduce Multi-Agent Verification (MAV) as a test-time compute paradigm that combines multiple verifiers to improve performance. To investigate scaling up the verification compute, we propose to combine multiple Aspect Verifiers (AVs) --- off-the-shelf LLMs prompted to verify different aspects of outputs. AVs are a convenient building block for MAV since they can be easily combined without any additional training. We introduce BoN-MAV as a simple multi-agent verification algorithm that combines best-of-*n* sampling with aspect verifiers, and we show that performance improves as we spend more verification compute at test-time by increasing the number and type of verifiers. Moreover, we demonstrate both weak-to-strong generalization, where combining weak verifiers improves even stronger LLMs, and self-improvement, where the same base model is used to both generate and verify outputs. Our results establish scaling the number and type of verifier models as a promising new dimension for improving language model performance at test time.

Published as a conference paper at COLM 2025
Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers
Shalev Lifshitz', Sheila A. Mcllraith Yilun Du AI,, Uniivrsiity of Toronto, Vector Institute for AI, *Harvard University
Abstract
By utilizing more computational resources at test-time, large language mod- els (LLMs) can improve without additional training. One common strategy uses verifiers to evaluate candidate outputs. In this work, we propose novel scaling dimension for test-time compute: scaling the number of verifier models. We introduce Multi-Agent Verification (MAV) as a test-time com- pute paradigm that combines multiple verifiers to improve performance. To investigate scaling up the verification compute, we propose combine multiple Aspect Verifiers (AVs) - off-the-shelf LLMs prompted to verify different aspects outputs. AVs are a convenient building block for MAV since they can be easily combined without any additional training. We introduce BoN-MAV as a simple multi-agent verification algorithm that combines best-of-1 sampling with aspect verifiers, and we show that per- formance improves as we spend more verification compute at test-time by increasing the number and type of verifiers. Moreover, demonstrate both weak -oa-to generalization, where combining weak verifiers im- proves even stronger LLMs, and self-improvement, where the same base model is used to both generate and verify outputs. Our results establish scaling the number and type of verifier models as promising new dimen- sion for improving language model performance at test time.
1 Introduction
Scaling the size of large language models (LLMs) and their training datasets has driven remarkable progress in artificial intelligence (Brown et al., 2020; Chowdhery et al., 2023; Hoffmann et al., 2022). However, the growing cost of scaling model size and obtaining unseen high-quality pretraining data has sparked interest in methods that improve LLM performance without scaling parameters data. Among these, a promising new direction has emerged: scaling tesl-time compute, where models spend more computational resources during inference-much like humans spend more time thinking through harder problems.
A common strategy for scaling test-time compute is best-of-n sampling (Stiennon et al., 2020; Cobbe et al., 2021; Nakano et al., 2021), where n candidate outputs are sampled from generator LLM and a verifier model scores each candidate output based on its quality or correctness. The highest-scoring output is then selected. Under this strategy, the amount of test-time compute can be scaled up by increasing the number of sampled outputs. However, in this work, we propose new orthogonal scaling dimension: scaling the number of verifier models. We introduce Multi-Agent Verification (MAV), a test-time compute paradigm that combines multiple verifiers to improve performance.
Typically, verifiers are implemented as reward models trained using reinforcement learning from human feedback (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a). However, relying on reward models as verifiers introduces two crucial limitations for multi-agent verification: (1) each reward model has to be trained on expensive curated preference data, and (2) there is straightforward way to combine scores generated by heterogeneous reward models trained on different datasets (they produce uncalibrated scores). These limitations make reward models poorly suited for multi-agent verification and restrict our ability to simply scale up the number and type of verifiers at test-time.
Correspondence to: Shalev Lifshitz (lffshitz.shlle@@@mallll Please find our paper webpage at http://awaaaaaaallllllllgeeeeeee
Yilun Du (ydu@seas.harvarddeedu

---

Published as a conference paper at COLM 2025
BoN-MAVV'
BoN- BoN-MAV@16
MATH Accuracy (%)
66
BoN RM@i
66
BoN-RMGB RM@ 16
cuns 6
59
59
52
52
8 Number of Candidate Outputs (n)
16
0
3 Number of Verifiers (m)
Figure 1: Scaling test-time compute along two dimensions. Left: Increasing the number of candidate outputs (ıt) and comparing three test-time methods: best-of-n with multi-agent verification (BoN- MAV@n), best-of-1 with reward model verification (BoN-RM@n), and self-consistency (cons@m). Right: Increasing the number of verifiers (m) when selecting between n 16 candidate outputs (BoN- MAV@16) surpasses the performance of reward model verification (BoN-RM@16) and self-consistency (cons@16). Candidate outputs sampled from Gemini-1.55 1.5--lashh on MATH (Hendrycks et al., 2021).
To address these limitations and enable scalable multi-agent verification, we propose using Aspect Verifiers (AVs) - off-the-shell LLMs prompted to verify specific aspects of candidate outputs by producing binary True/ False approvals. Like LLM-as-judge (Zheng et al., 2023), but unlike reward models, AVs do not require additional training and their binary outputs can be easily combined across multiple models through simple voting mechanisms. Thus, the number and type of verifiers can be easily scaled up without additional training. We note that these aspect verifiers are just one possible implementation choice for the verifiers in a MAV system, which address limitations of reward model verifiers for multi-agent verification and allow us to scale up the verification compute without additional training.
To investigate scaling multi-agent verification, we introduce BoN-MAV as specific algo- rithm which combines best-of-n sampling with aspect verifiers. This is just one possible implementation of a MAV algorithm which combines traditional best-of-n sampling with multiple verifiers. Given an input, BoN-MAV (1) samples n outputs from a generator LLM, (2) collects binary approvals from set of m aspect verifiers, and (3) selects the output with the most approvals. We investigate scaling test-time compute with this approach along two orthogonal dimensions: the traditional dimension of increasing the number of sampled candidate outputs n, and our novel test-time scaling dimension of increasing the number of verifiers m. We find that using multiple diverse verifiers to select between candidate outputs is an effective strategy, and that performance improves as we use more verifiers.
More specifically, across multiple domains and LLMs, BoN-MAV demonstrates more ef- fective scaling patterns when we increase the number of sampled outputs, compared to best-of-n with reward model verification (Stiennon et al., 2020; Cobbe et al., 2021) and self-consistency (Wang et al., 2022; Lewkowycz et al., 2022). Importantly, we show that when scaling up to 256 sampled outputs, the baseline methods plateau while BoN-MAV continues to improve. Thus, by using more verification compute, MAV can raise the ceiling on test-time performance. We also demonstrate weak-lo-strong generalization (Burns et al., 2023), whereby combining many small aspect verifiers can improve the performance of even stronger generator LLMs, and we show that BoN-MAV enables self-improvement by using the same base LLM for both the generator and set of aspect verifiers. Since BoN-MA is just one simple approach to multi-agent verification, we expect that substantial improvements can be achieved using alternative methods. Overall, our paper makes the following contributions:
(1) We introduce Multi-Agent Verification (MAV) as a new test-time paradigm that com- bines multiple verifiers at test-time, opening novel scaling dimension: scaling the number of verifier models.
(2) For MAV, we propose using Aspect Verifiers (AVs), off-the-shelf LLMs which require no additional training and naturally support combining verification signals from multiple heterogeneous verifiers using voting mechanisms making them easy to scale in number and type.
(3) We demonstrate that BoN-MAV, a simple multi-agent verification algorithm which combines best-of-n with aspect verifiers, improves the performance of various generator
LLMs as we scale up the number and type of aspect verifiers.
2

---

Published as a conference paper at COLM 2025
Question: Exgresa 1. 1.254234... an improper fraction.
Sampled Candidate Output: (INCORRECT) Let + 1.254234...
Correct Anawer: 1377/11
Muttilyy both sides of the equatton by 1000; 1000> - 1234 234234... Sube the flrst equation from the socond equation: 999x 1233 Solve for x 1239/989
Genemated Antwer: 411/333
Augreguter Verificution Score: 2/5
Simplily by dividing numus afor and denaminator S: -11/235
Aupact Vanifier A1
Axpect Vanifier
Aspect Varifier 8S
Axpect Vanifirr de
Aspect Varifier #5
Bass model: Gam m--1. 115--1tash
Slane model: (I pill-mmmm
Biese modal: Gamenn 1. lash
Bsane model: Lill-do-m
Bhase model: Lirrane- laah
Aspect: General Comectnass Streegy: Dorect Approval
Aspuet: Lind Convers one Strmtngg Step Step
Aspect: Log Boundrezs Strengy:: Step-by-Step
Aspaet: Gineral Cerrectness Strinngy S.m Jummer de Solution
Aspact: Math Cernectress
Stiitagy Shep-ity-Sigg
prochen reokes sonveting decimal traction which In freaston there
Let santtat poadssed sckzdon sosp stop
propeiod actutinn taggin
Obky,, leth ththees the sotutionn
lezings equat the reperting
exap dy saap.
commutyy statto We oxtines 1.334234...
comnrtt repersoeeeddn 1.234234...
This ward way start.
check.
ansggiig variall
carpoited luther The grainet
connunyy for an lopis
Stip Bolvve lor 1200/eee
connivs pivater poicty
The connur rlviding fiems
further 411 -0000 di 1000111
check E451 100 be mim indd
furter and be
Stap & Simpllity by chviding the
1.332244 as imprrpoe maards
and 100 di tin. Tree the
ci11/208 This inneee the enni
187111) nor 100z0
hacteen srmpultedd further
occuus While bach 1233 and
Sin the rqpassee sok
to ceciappiiiin farthe
consain an ortr
based arawer 1200
thet the silution ontivdy cornecs
the perpereenn silution opetaiss uno in awrphioonis zto..
APPROVAL TRUE
APPROVAL TRUE
APPROVAL FALSE
APPROVAL
APPROVAL FALSE
Figure 2: Multi-agent verification for single solution. Five different aspect verifiers evaluate an incorrect MATH solution from Gemini-1.5-Pro. Verifiers vary by base model, verification aspect, and strategy. While two verifiers miss the error, the remaining three identify the mistake. This demonstrates how combining diverse verification methods can produce robust signal despite individual failures.
Related Works
Scaling Test-Time Compute. A large body of recent work has explored how increasing test-time compute can improve language modeling performance. One line of research focuses on techniques where single generator LLM produces additional output tokens during inference. These include scratchpads or Chain-of-Thougee prompting (Nye et al., 2021; Wei et al., 2022), self-consistency or majority voting techniques (Wang et al., 2022; Lewkowycz et al., 2022), and various self-reflection methods (e.g. Shinn et al. 2024; Madaan et al. 2024). Other works have explored training LLMs generate special tokens which enhance reasoning ability at test-time (e.g., Goyal et al. 2023; Wang et al. 2023a; Schick et al. 2023). Another line of research focuses on using a verifier model to evaluate the quality or correctness of outputs sampled from generator models (Cobbe et al., 2021; Zheng et al., 2023). Typically, this done through best-of-N sampling (Stiennon et al., 2020), where candidate outputs are generated and the highest-scoring output is selected based on some verifier. This verification can be performed at the outcome-level (Stiennon et al., 2020; Cobbe et al., 2021) or process-level (Lightman et al., 2023; Wang et al., 2024a), and some reward models produce CoT before scoring (Zhang et al., 2024; Mahan et al., 2024). Most similar to our work, Coste et al. (2023) and Eisenstein et al. (2023) have explored using ensembles of homogeneous reward models (identical model initializations trained on the same data but with different random seeds) to mitigate reward model overoptimization (Gao al., 2023). Different from this, we propose a framework for combining multiple heterogeneous verifiers that verify distinct aspects of generation, and investigate scaling both the number and type of verifiers without additional training as novel test-time scaling dimension. We show that spending more verification compute by scaling the number and type of verifiers can substantially improve performance and raise the ceiling on test-time performance.
Multi-Agent Reasoning with Language Models. Recent works have investigated several approaches to multi-agent interaction for improving language model reasoning. Language model debate (e.g,, Du et al. 2023; Chan et al. 2023; Pham et al. 2023; Liang et al. 2023; Subra- maniam et al. 2025; Li et al. 2023; Cohen et al. 2023) and multi-agent discourse (e.g., Chen et al. 2023; Wang et al. 2023b; 2024b) have been studied ways to enhance reasoning, and also as a direction for scalable oversight research (Irving et al., 2018). Some works have explored multi-modal reasoning through agent collaboration (e-g., Zeng et al. 2022; Li et al. 2022a; Ajay et al. 2023; Jiang et al. 2024). Unlike prior work on multi-agent reasoning which focuses on collaborative problem-solving, we introduce multiagent framework where each agent focuses on verifying individual aspects of a solutions.
3

---

Published as a conference paper at COLM 2025
Stor. each denermined the elemental abundancee these [SUFi)] 0.3 des, (Mgrii 0.3 [FeM)] 1-0 [M]HH) Consider the following phonoepherle the Sun: 12 ing10)/faamaiii- - 7.5 12 lngt0mnMggnaj Catoulane the ratlo elltoon wlore photoxpheres of Stor and Ster
Question:
timeimes
respectively Assuming
method, astronomers
Options: [A] -3.9, -12.4, Ha.0, (D) -1.2
Correct
Gutput #1: Faane's can sslve probtem stop ww-.stap........................ the answeria (A)
Output #3: Hann's how to bresk catnn
Ouriut FD: Hore a how to solve the problem step- stleee The unsoonuta (C)
put t4ll Here's how to sclve the prosiam stap -ty sty-steep.... The aroowrr
protiien stop-ity stepl snswer (R)
AV 91:
AV
43
X
AV
AV O2:
AV
X
AV wT::
X
AV N7:
X
AV
X
AV:1:
AUl #7:
X
AV WT:
X
AV
AM
X
AU
AV
MO:
AV
AV
AV W7:
AV 1G:
AV US:
NV N7:
AV AS:
AV
X
AV N7:
AV 65:
X
AV NO:
X
Aggregated Score: 2/9
Astreppited
Aggrojated
Aggrogoted Score:
Figure 3: Illustration of the BoN-MAV algorithm. BoN-MAV combines best-of-n sampling with multi-agent verification: First, candidate outputs are sampled from generator LLM. Then, each output is evaluated by a set of Aspect Verifiers (AVs) that produce binary approvals. Finally, the candidate with the most approvals is selected as the final answer. See Section 3.3 for algorithm details.
3 Multi-Agent Verification
Multi-Agent Verification (MAV) is a test-time compute paradigm where multiple verifiers are combined to evaluate outputs from generator LLM. To implement MAV algorithm, we must address two questions: (1) What type of verifiers can be easily combined and scaled up in number without additional training? (2) How should we aggregate verification signals from multiple verifiers? In this section, we propose answers to these questions and describe one simple implementation of multi-agent verification algorithm called BoN-MAV. We discuss future directions for alternative MAV algorithms in Appendix A.
In Section 3.1, we propose Aspect Verifiers (AVs) as a convenient building block for MAV, since they require no additional training and naturally support combining multiple verifica- tion signals. In Section 3.2, we describe our approach to aggregating signals across multiple AVs. In Section 3.3, we outline the BoN-MAV algorithm, which combines best-of-n sampling with aspect verifiers. Finally, Section 3.4 proposes verifier engineering as method to select relevant verifiers for specific domains or tasks.
3.1 Aspect Verifiers
In the context of test-time computation with LLMs, a verifier typically refers model that evaluates the quality of an output sampled from generator LLM. Here, we ask: What type of verifiers can be easily combined and scaled up in number without additional training?
Prior works have largely focused on using neural reward models as verifiers (Stiennon et al., 2020; Cobbe et al., 2021; Snell et al., 2024). However, these models present key challenges for scaling multi-agent verification. First, each reward model requires training on expensive curated preference data to produce reliable reward scores (Stiennon et al., 2020). Second, while ensembles of homogeneous reward models (identical model initializations trained on the same data but with different random seeds) have been proposed as way to mitigate overoptimization (Coste et al., 2023; Eisenstein et al., 2023; Gao et al., 2023), there is no straightforward way to combine scores from heterogeneous reward models trained on different datasets. This second limitation arises because scores from different reward models are uncalibrated- they operate on different numerical scales based on their training setups. We wish to simply scale up the number and type of verifiers without additional training.
We propose to use off-the-shelf LLMs similar to LLM-as-judge (Zheng et al., 2023) as one possible implementation choice for the verifiers in a MAV system. Specifically, we propose Aspect Verifiers (AVs), off-the-shelf LLMs prompted to evaluate specific aspects of candidate outputs and produce binary True/ / FFasse approvals. Unlike reward models, they require no additional training and their binary approvals can be easily combined through simple voting mechanisms, even when AVs are based on completely different models or training data. Using aspect verifiers, we can easily scale up the number and type of verifiers which may be based on different LLMs, training algorithms, architectures, data, or prompts.
Aspect Verifiers can be configured along three axes: (1) The base LLM (e.g., GPT-4o-mini or 1emini-115--Fasshe (2) The aspects to verify (e.g., mathematical correctness, logical sound-
4

---

Published as a conference paper at COLM 2025
MATH
MMLU-Pro
GPQA (diamond)
HvmanEval
Generator LLM
B-MAV Cons RM
B-MAV Cons RM
B-MAV Cons RM
B-MAV Cons
RM
Gemini-1.5-Flash
66.0
59.0
61.7
66.7
63.3
60.7
42.0
40.0
46.0
80.0
79.0 79.0
Gemini-1.5-Prro
72.7
70.3
71.0
72.3
71.7
69.3
49.0
45.0
49.0
88.0
84.0
88.0
GPT-4o-mini
73.0
74.7
72.3
67.0
63.7
62.7
50.0
48.0
44.0
84.0
87.0
85.0
GPT-4o
76.3
77.3
80.7
75.7
76.3
72.7
59.0
59.0
58.0
92.0
95.0
92.0
Mistral-7B
26.0
22.0
21.7
36.7
25.7
31.0
36.0
32.0
37.0
59.0
46.0
52.0
Llama-3.1-8B
61.7
61.0
54.7
59.3
55.3
51.3
43.0
36.0
41.0
75.0
62.0
64.0
Gemma-2-98
58.7
51.7
55.0
57.7
54.3
54.7
34.0
36.0
38.0
32.0
25.0 51.0
Gemma-2-27B
62.3
55.7
59.3
62.0
58.3
60.0
41.0
40.0
41.0
76.0
66.0 76.0
Table 1: Best-of-n with Multi-Agent Verification (BoN-MAV) across models and domains. Per- formance (accuracy %) comparison of three test-time verification methods using = 16 candidate outputs: the BoN-MAV algorithm (labeled B-MAV in the table), reward model verification (RM), and self-consistency (Cons). Results are shown eight generator LLMs across four domains, with BoN-MAV each domain using the dppain-specific aspect verifier subset Md BoN-MAV outper- forms self-consistency nearly all cases, and generally outperforms RM except on GPQA (diamond) and HumanEval, where BoN-MAV and RM achieve comparable results.
ness, factuality, etc.), and (3) The verification strategy (e.g,, direct approval, going over the output step-by-step, rephrasing, checking edge cases, etc.). Figure 2 illustrates how multiple aspect verifiers can evaluate a single candidate output, and Appendix shows the full list used in this work (and their prompts). Note that we spend very little time engineering the prompts - just enough so that the output is in the desired format.
3.2 Combining Aspect Verifiers
With aspect verifiers as our building block, we ask: How can we effectively aggregale verification signals across multiple AVs? We take the simplest possible approach in our experiments: each binary
approval is single vote, and the aggregated score for a candidate output is the sum of the positive votes from all AVs. That is, the aggregated verification score is the sum of the individual binary scores from each verifier:
1
AggScore(o IM DEM
Σ BinarySccre,,o
(1)
where ol O is the ith candidate output, .M M is the set of aspect verifiers, and BinaryScore,, O + {0,1} maps a candidate output from O to the binary approval produced by verifier v E M for that output. This voting strategy gives equal weight to all verifiers in the final aggregated score, and it proves remarkably effective in our experiments (see Section 4.1). However, future works could investigate more sophisticated aggregation strategies such as grouping verifiers by aspect and then voting across aspects, or having aspect verifiers debate with each other (Du et al., 2023) before producing an approval. We discuss these and other potential directions for future work in Appendix A.
3.3 BoN-MAV
Best-of-n (BoN) sampling is test-time optimization technique (Stiennon al., 2020; Cobbe et al., 2021; Nakano et al., 2021) where nt candidate outputs are sampled from a generator LLM, each candidate is scored by a verifier model, and the highest-scoring output is selected. We introduce BoN-MAV as simple multi-agent verification algorithm that combines best- of-1 sampling with aspect verifiers. It uses the simple aggregation strategy from Equation and consists of three steps: (1) sampling n candidate outputs from a generator LLM, (2) collecting binary approvals from a set of m aspect verifiers, and (3) selecting the output with the most approvals. That is, f = arg maxp<cion (AggScore(o())
where (i)) is the ith candidate output, is the total number of sampled candidate outputs, and is the index of the output with the highest aggregated score (the selected output). Figure 3 illustrates how BoN-MAV can be used to select between set of candidates.
5

---

Published as a conference paper at COLM 2025
MoN-MAVeD
BoN-RM@o
cons
Gemini- 1.5 Prrro Cemini 5.5 F5-Flssh
Pro.U.Pro
GPOA (diamond)
HumanEval
(99)
66
67
19
81
59
63
44
39
78
Accurncy (%)
74
73
53
90
69
70
44
83
67
(%)
26
37
59
Mistrul-7
19
32
50
25
Llama-3.1 3.1-8B
62
60
75
Acmrrcc
68
Number of Candidate Outputs (n)
Figure Scaling the number of candidate outputs. Performance (accuracy %) of test-time methods as we increase the sampled candidate outputs (11) shown four generator LLMSs (Gemini-1.5-llaa Gemini-1.5-Pra Mistral-7B, and Lllma-3.1-88) across all evaluation domains. The BoN-MAV algorithm demonstrates more effective scaling patterns than self-consistency (cons@n) across all domains, and stronger scaling than reward model verification (BoN-RM@n) except GPQA (diamond).
Using BoN-MAV, we can increase test-time computation by sampling more candidate outputs (increasing n) and by querying more verifiers (increasing time computation can be easily parallelized during generation as well as verification. Thus, we can increase both generation compute and verification compute test-time. In addition, BoN-MAV represents just one specific approach to MAV, and more nuanced aggregation algorithms or alternative verifiers could further enhance performance (see Appendix A).
[M(), where test-
3.4 Verifier Engineering
Using aspect verifiers, can create diverse pool of verifiers with different capabilities. However, not all verifiers are equally relevant for every domain. Thus, we propose verifier engineering as a process select subset of verifiers most effective for a particular domain (similar to prompt engineering, where prompts are engineered for specific domains or tasks). We engineer -ppeiin sets of verifiers by first creating diverse initial set M and then selecting the subset Md C M which contains the most relevant verifiers for domain d. Specifically, we select the subset which maximizes the average performance across all generator LLMs evaluated on a validation set.
Experiments
In our experiments, we investigate scaling test-time compute along two orthogonal dimen- sions: the traditional dimension of increasing the number sampled candidate outputs and our novel test-time scaling dimension of increasing the number of verifiers m. We aim to address the following questions: (1) How well does MAV improve performance across diverse domains and various generator LLMs? (2) Can MAV facilitate weak-to-strone generalization and self-improvement? (3) How important is engineering a domain-specific set of verifiers and what are the important design choices? To address these questions, we evaluate the BoN-MAV algorithm described in Section 3 on four domains: MATH dataset (Hendrycks et al., 2021), MMLU-Pro (Wang al., 2024c), GPQA (diamond) (Rein et al., 2023), and HumanEval (Chen et al., 2021) (see Appendix B.1 for details). See Ap- pendix C for additional experiments such using Al-generated verifiers, comparing single verifier which scores rubric of all aspects, and more.

---

Published as a conference paper at COLM 2025
MATH
MMLU-PPo
GPQA (diamond)
HumanEval
Gemini-1 5.-Pro
74
53
000999900000
69
48
86
64
42
Accuracy (9)
77
60
CFT44o
76
0-0-0
94
68
53
91
Accrrny (%)
38
39
Mistrul-7
000000000
19
33
12/9
26
Liamus 3.1-8B
46
Anturacy
0
57
50
48
31
67
14
Number of Veriflers (m)
Figure Scaling the number of verifiers. Performance (accuracy %) of BoN-MAV as increase the number of verifiers (m) up to domain-specific subsets M (detailed in Section 4.1). For each
plot the performance of BoN-MAV averaged across all possible combinations of verifiers drawn from Md, with shading indicating the spread of observed values dark blue shows the 25-75th percentile range (middle 50%) while light blue shows the 5-95th percentile range (90% of outcomes). The leftmost point (m = 0) represents pass@1 accuracy without verification while the rightmost point (m (M uses all verifiers in M
4.1 MAV Enables Scaling Along Two Dimensions
Baselines. We compare Best-of-n sampling with Multi-Agent Verification (BoN-MAV) against two established test-time compute methods: (1) best-of-1 sampling with reward model verification (Stiennon et al., 2020; Cobbe et al., 2021; Nakano et al., 2021), where we use a trained neural reward model as the external verifier to select the highest-scoring candidate output, and (2) self-consistency (Wang et al., 2022; Li et al., 2022b; Thoppilan et al., 2022; Lewkowycz et al., 2022), which selects the most common answer from the set of candidates outputs. For reward model verification, use the current top-performing open-source 8B reward model on RewardBench (Lambert et al., 2024) (see Appendix B.5). We implement the verifier engineering method described in Section 3.4 (see Appendix B.2).
Quantitative Results. We evaluate BoN-MAV across four domains using eight generator LLMs (four closed-source and four open-source). For each model, we sample n = 16 candidate outputs per question and compare between best-of-n with Multi-Agent Verifica- tion (BoN-MAV), best-of-n with reward model verification (BoN-RM), and self-consistency (cons). As shown in Table BoN-MAV outperforms self-consistency in nearly all cases, and outperforms reward model verification on MATH and MMLU-Pro, while achieving comparable results reward verification on GPQA (diamond) and HumanEval.
Qualitative Examples. Figure illustrates how multiple aspect verifiers can be used to evaluate single candidate output. The first aspect verifier uses direct yes/no no approval without step-by-step thinking and incorrectly approves the output while additional aspect verifiers, using the same base model but with more thorough verification strategies, success- fully identify the error. Additional examples are provided in Appendix D (slightly different sets of verifiers than our final domain-specific sets).
Scaling the Number of Candidate Outputs. In Figure 4, we show the scaling patterns for various generator LLMs as we increase the number of sampled candidate outputs (n). Matching the results in Table 1, BoN-MAV demonstrates more effective scaling patterns than self-consistency across domains, and stronger scaling than reward model verification on MATH and MML.UPro while achieving comparable patterns GPQA and HumanFval.
7

---

Published as a conference paper at COLM 2025
70.0
BoN-MAV (ourS)
70.0
DeN-MMVV@ (ours)
67.5
DoN 304gg
67.3
BoN--IMMg
MALL Accureyy (8)
consiinn
63.0
Accuraey
03.0
62.5
82.5
60.00
57.5
57.5
35.0
55.0
52.5
52.5
2
Number of Candidate Outputs (c)
Compute Budget (lotal Querzes)
Figure 6: Scaling to 256 candidate outputs. Comparing test-time verification methods on MATH as we increase the number of outputs sampled from Gemini-1.5-Flash up to 256. Left: Accuracy (%) versus number of sampled outputs nt. BoN-MAV consistently improves with additional samples while reward model verification (BoN-RM) and self-consistency (cons) plateau much earlier. Right: Accuracy (%) versus total compute budget (number of queries to both generator LLM and each verifier).
Scaling the Number of Verifiers. Multi-Agent Verification introduces a powerful new dimension for scaling test-time compute: scaling the number of verifiers. In Figure 5, we show how accuracy tends to improve as we increase the number of verifiers m from zero verifiers up to the full domain-specific subset Md For each value of m E {0,1,2, ..., [M we plot the average accuracy across all possible combinations of m verifiers drawn from Md.. Our results demonstrate that scaling verifier count is promising new dimension for improving model performance at test-time. In most cases, accuracy improves as we add verifiers, with performance gains of up to 10% for large LLMs and up to 20% for small models. Notably, performance gains persist even when strong generator LLMs (Gemini-1.5-Pro GPT-4o) are verified by weaker verifiers (Gemini-1.5-Flash, GPT-4o-mini), supporting our findings in Section 4.2. However, the magnitude and pattern of improvement varies and, in some cases, accuracy initially decreases before improving with additional verifiers. We expect better-engineered verifiers to unlock even stronger scaling patterns.
Scaling Up to 256 Candidate Outputs. We extend our analysis to even larger scales by sampling 256 candidate outputs from Gemini-1.5-Flash on MATH. In Figure 6, we plot accuracy as function of both the number of sampled candidate outputs и (left) and the total compute budget (right). We find that BoN-MAV consistently improves with additional samples, while the baselines plateau early on. Starting at 52.7% base accuracy, the baselines plateau around 61% while BoN-MAV continues to 99%%nearly double the improvement. At low compute budgets, the overhead of querying multiple verifiers with BoN-MAV means we can sample fewer candidate solutions, leading to initially worse performance than the baselines. However, once we have sufficient compute, BoN-MAV significantly outperforms both baseline methods. By spending more verification compute for the same number of candidate outputs, we can raise the performance ceiling at test-time.
4.2 MAV Enables Weak-to-Strong Generalization and Self-Improvement
Weak-to-Strong Generalization. Prior work has shown that weak supervisors can improve the performance of strong pretrained models (Burns et al., 2023). Here, we show that MAV can be used to enhance the performance of strong generator LLMs by combining weaker verifiers. As shown in Table 2, our strongest generators (Gemini-1.5-PPo and GPT-4o) show substantial improvements over their base pass@1 accuracy even though the verifiers are based on weaker models, and Figure 5 shows how their performance changes as use more verifiers. These results suggest that the diverse perspectives of multiple smaller verifiers can collectively produce a signal robust enough improve even state-of-the generators.
Self-Improvement. MAV can also enable models to improve their own performance through self-verification. To demonstrate, we sample candidate outputs from generator LLM (Gemini-1.5-Flaag or GPT-4o-mini) and create multiple aspect verifiers derived from the same LLM. Following the verifier engineering procedure from Section 4.1, we select the best subset of self-verifiers based on validation performance. These self-verification results (Table 2) show in substantial improvements over base pass@1 accuracy across all domains except HulanEval. For instance, GPT-4o-mini shows particularly strong self-improvement on MATH (+7%) and GPQA diamond (+8%).
8

---

Published as a conference paper at COLM 2025
MATH
MMLU-Pro
GPQA (diamond)
HumanEval
Generator LLM
B-MAV passë1
B-MAV pass
B-MAV
passo1
B-MAV pass@]
Gemini-1.5-Pro
72.7
64.7
72.3
68.0
49.0
45.0
88.0
84.0
Wcak-to-Strong
GPT-4o
76.3
68.3
75.7
73.3
59.0
54.0
92.0
94.0
Gemini-1.5-Flash
59.0
52.7
42.0
78.0
79.0
Self-Improvement
GPT-4o-mini
76.0
69.0
65.7
62.3
46.0
38.0
86.0
86.0
Table 2: Weak-to-strong generalization and self-improvement, Performance of BoN-MAV (B-MAV) compared base passø] accuracy. For weak-to-strong generalization (top), use aspect verifiers aspect verifiers
based weaker models than the generator. For self-improvement (bottom), based on the same LLM as the generator. BoN-MAV improves performance in nearly all cases.
Ablation 1: Dvrfier-Enginterii
Ablation 2: Verifier Diversity
MMLU-Pro
GPQA (diamond)
MMLU-Pro
GPQA (diamond)
Generator LLM
Eng
All
Eng
All
Diverse
Same
Diverse
Same
Gemini-1.5-Flash
66.7
65.7
42.0
41.0
66.7
66.3
42.0
39.0
Gemini-1.5-Pro
72.3
70.3
49.0
49.0
72.3
71.0
49.0
55.0
GPT-4o-mini
67.0
65.3
50.0
49.0
67.0
64.7
50.0
42.0
GPT-4o
75.7
75.3
59.0
55.0
75.7
75.0
59.0
58.0
Table 3: Ablation Studies. Left: Performance comparison using engineered domain-specific verifiers (Eng) versus using all verifiers (All). Right: Performance comparison using diverse verifiers from M (Diverse) versus querying the single best-performing verifier multiple times (Same).
4.3 Analysis: Understanding Multi-Agent Verification
To better understand the key design choices that impact MAV, conduct two ablation studies on MMLU-Pro and GPQA (diamond)- two most challenging domains in our evaluation. We investigate: (1) how performance depends on verifier engineering, and (2) whether using diverse verifiers outperforms repeatedly querying the single best verifier.
Effect of Verifier Engineering. In Section 4.1, we introduced verifier engineering as an approach for selecting relevant subset of verifiers M S M for each domain d. Here, we compare our engineered verifier subsets M against simple baseline that uses all available aspect verifiers in M (see Appendix B.3 for full list) without any domain-specific tuning. Table 3 (left) shows that engineering the set of verifiers is more effective strategy. However, Table 5 in the Appendix shows that even the simple strategy of combining all verifiers in M remains competitive with both self-consistency and reward model verification baselines.
Effect of Verifier Diversity. Here, investigate whether using diverse verifiers outper- forms repeatedly querying a single verifier. Specifically, we compare the performance of our diverse domain-specific subsets Md versus making |M questions to the single best-performing verifier v" MM for domain d. As shown in Table 3 (right), using diverse sets of verifiers generally outperforms querying the same verifier multiple times.
Conclusion
We have introduced Multi-Agent Verification (MAV), a test-time compute paradigm that combines multiple verifiers to improve performance. MAV enables test-time scaling along two orthogonal dimensions: (1) the traditional dimension of increasing the number of candidate outputs sampled from generator LLM,, and (2) our novel test-time scaling dimension of increasing the number verifier models evaluating each output. We illustrate the efficacy of MAV at improving test-time performance across multiple domains and generator LLMs, enabling weak-to-strong generalization and self-improvement. We hope that our work opens new direction of research in combining multiple language models for effective reasoning and scaling verification compute, and we discuss additional limitations of our approach and opportunities for future research in Appendix A.

---

Published as a conference paper at COLM 2025
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar preprint ar Xiv:23303.007774 2023.
Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. Advances in Neural Information Processing Systems, 36:22304 20.22304-22325, 2023.
Kenneth J Arrow. Social choice and individual values, volume 12. Yale university press, 2012.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ar preprint ar arXiv:2204005862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from ai feedback. arXiv preprint ar arXiv:2212.00073 2022b.
Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D Procaccia. Handbook of compulaliinnll social choice. Cambridge University Press, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold As- chenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to- strong generalization: Eliciting strong capabilities with weak supervision. arv preprint Xiiv:2312.09390, 2023.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better LLM-based evaluators through multi-agent debate. ar preprint ar Kki:2300.002200 2023.
Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse LLMs. arXio preprint drXit:2309.13007 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. ar Xiv preprint ar XxXiv:2107.0337, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, et al. Training verifiers to solve math word problems. arx preprint ar XXXt:21100.116.. 2021.
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. Lm vs Im: Detecting factual errors via cross examination. aiv preprint arXit:2305.13281 2023.
10

---

Published as a conference paper at COLM 2025
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. ar Xio preprint ar (xX::23310002743, 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeni Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint ar xrXio:2344.06677 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch, Improving factuality and reasoning in language models through multiagent debate. ar Xio preprin! ar arXit:2305.14325 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar arXi::4077.21788 2024.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' D'mour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. Xiv preprint ar XrX::22312.0924,, 2023.
Peter C Fishburn. The theory of social choice. Princeton University Press, 2015.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. Xrviv preprint ar XrXi:22310.02266, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ar preprint ar arXiv:2009.03300 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Xrx preprint wrXiv:2103.03877 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arv preprint ar XrXi:22203.15566 2022.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. ar arv preprint ar orXiv:2210.11610, 2022.
Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. ar preprint 1xXiv:105500099, 2018
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. Xio preprint drXio:2310.06825, 2023.
Bowen Jiang, Yangxinyu Xie, Xiaomeng Wang, Weijie Su, Camillo Jose Taylor, and Tanwi Mallick. Multi-modal and multi-agent systems meet rationality: A survey. In ICML 2024 Workshop on I.LMs and Cognition, 2024.
Jerry S Kelly. Social choice theory: An introduction. Springer Science & Business Media, 2013.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. ar preprint ar Xit::24033.1373 23082443..777... 2024.
11

---

Published as a conference paper at COLM 2025
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural 2022.
Information Processing Systems,
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Shuang Li, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, and Igor Mordatch. Composing ensembles of pre-trained models via iterative consensus. arXiv preprint arXiv:2210.11522 2022a.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code 2022b.
generation with alphacode. Science, 378(6624)
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. ar preprint ar arXiv:2305.19118 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint ar Xie:2335.20050, 2023.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. Xrv preprint 0rXim:2410.18455) 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castri- cato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint ArXiv:2410. 128322, 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question-answering with human feedback. arvie preprint ar xrXi::2112.09332,, 2021.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arviv preprint xxXiv:2112.00114, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744 2022.
Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A Plummer, Zhaoran Wang, and Hongxia Yang. Let models speak ciphers: Multiagent debate through embeddings. arXiv preprint ar arXiv:2310.0627 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. arXiv preprint ar arxiv:2311. 120222, 2023.
Timo Schick, Jane Dwivedi-YK Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: -8551, 2023.
12

---

Published as a conference paper at COLM 2025
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time com- pute optimally can be more effective than scaling model parameters. arXiv preprint WMXiv:2408.03333 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. arxiv preprint ar Xiv:25001.00700 2025.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint ar arXt::2003.05530, 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma Improving open language models at a practical size. ar preprint ar arXiv:2408.0011 2024b.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, al. Lamda: Language models for dialog applications. Xrv preprint ar arXtv:220..08239 2022.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math- shehh-shepeedd Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426-9439, 2024a.
Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of Ilm reasoning: Are multi-agent discussions the key? arXiv preprint 17X1::2402.18272, 2024b.
Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessandro Sordoni. Guiding language model reasoning with planning tokens. aiv preprint arXiv:2310.05707, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. arx preprint ar arXiv:2003111771, 2022.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more ro- bust and challenging multi-task language understanding benchmark. arXiv preprint ar 4rXiv:2406.01574, 2024c.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Un- leashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint ar Kio:2307.05300 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
Le, Denny Zhou, et al. thooghof- prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2455 2022.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratie models: Composing zero-shot multimodal reasoning with language. arxiv preprint ar xrXi::2204.00598 2022.
13

---

Published as a conference paper at COLM 2025
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint (rXiv:2408.15240, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46395- 46623, 2023.
14

---

Published as a conference paper at COLM 2025
A Further Discussion
Multi-Agent Verification (MAV) introduces a promising dimension for scaling test-time compute:
scaling the number of verifiers. In Section 4, we demonstrated that combining multiple verifiers enables more effective evaluation of candidate outputs, facilitates weak- to-strong generalization, and allows for self-improvement. However, our approach has important limitations and there are several opportunities for future work to explore.
First, our investigation is limited to pool of 20 aspect verifiers based on just two base LLMs, and the design of our verifiers is constrained by our ability to come up with diverse verification strategies and relevant aspects. Future work could explore scaling to many more verifiers and try a more systematic exploration of the space of verifiers, potentially using LLMs themselves to generate diverse verification strategies and identify relevant aspects to verify. With better-engineered verifiers and more systematic exploration, we expect to observe stronger scaling patterns.
Second, our aggregation technique described in Section 3.2 uses simple voting mechanism that directly sums the individual binary approvals from each verifier. This approach does not account for the confidence relevance of each verifier, and verifiers do not observe each other's 's decisions or feedback. Future works could explore more sophisticated aggregation methods such
seeiidenc--eiihhtr voting or allowing verifiers to engage in debate (Du et al., 2023) before producing an approval. Moreover, our current approach uses a static engineered set of verifiers MM for all questions in a domain d, even though it may be best to use fewer or different verifiers for specific questions. Future works could investigate dynamically selecting the best set of verifiers for particular problems or adaptively choosing additional verifiers based on the results of the first few verification queries. Additionally, the field of social choice theory (Arrow, 2012; Fishburn, 2015; Kelly, 2013; Brandt et al., 2016) is concerned with procedures for collective decision-making and might offer insights for aggregating the perspectives of diverse verifiers. Although, our setting differs in that we care more about verifier capabilities than preferences.
Next, our implementation of BoN-MAV is limited to only a single generator LLM. Thus, an interesting direction would be to explore sampling from multiple generators in addition to evaluating with multiple verifiers. Since different models may excel at solving different types of problems, this approach could make even better use of the growing ecosystem of LLMs and their diverse capabilities.
while our results show that BoN-MAV can improve language model perfor- mance at test-time, we did not investigate finetuning the generator LLM on the outputs selected by our verifiers. Similar to how prior works have finetuned on outputs selected through self-consistency (Huang et al., 2022) or reward models (Dong et al., 2023), training outputs selected by MAV systems could be explored tor LLM and also each of the LLM-based verifiers. Moreover, an interesting direction for future work is to directly use reinforcement learning to train both the generator and verifier models. That is, generator LLMs can be trained to maximize the scores across multiple verifiers, and the verifiers aspects of responses.
Furthermore,
method to improve the genera-
simultaneously be trained to accurately verify individual
Additionally, we used the strongest open-source reward model that could run on our academic compute, which ranked among the top 10 on RewardBench (Lambert et al., 2024) at the time of writing (outperforming many larger reward models). However, this model is not as powerful as some of the generator LLMs. It would be worthwhile for future work to compare against stronger reward models.
It's also worth mentioning that our work also has a connection to Constitutional AI (CAI) (Bai et al., 2022b), which was one of the first major works demonstrating reinforcement learning from AT feedback (RLAIF) for LLMs, and focused on improving alignment through train-time RL (using the model's own feedback and a set of constitutional principles). Our paper also uses uses LLM-based evaluation to improve performance, but has a different contribution: we propose scaling verification compute by increasing the number and type of verifiers, without any training. So CAI is an RLAIF train-time RL approach for alignment, whereas our work focuses on test-time scaling and introduces an approach where verification compute can be scaled using diverse verifiers without additional training.
15

---

Published as a conference paper at COLM 2025
An underlying thread throughout work and discussion is the vision of growing ecosystem of diverse language models that generate, verify, and learn from each other. Our work on multi-agent verification represents step in this direction, and each of the future directions we have discussed offers a potential avenue for additional progress. We look forward to seeing how the research community advances these ideas.
B Experimental Setup
B.1 Domains
We evaluate our method on the following four domains:
Mathematics. The MATH dataset (Hendrycks al., 2021) consists of competition-level math questions at five difficulty levels. For our experiments, we randomly sample 400 questions from the test set across all five levels: 100 for validation and 300 for testing. General Knowledge & Reasoning. MMLU-Pro (Wang et al., 2024c) is an enhanced version of the popular MMLU benchmark (Hendrycks et al., 2020) which features more challenging, reasoning focised questions and expands the multiple-choie set from four to ten options. As with MATH, we sample 100 questions for validation and 300 for testing.
Lraduate-Level Reasoning. The GPQA dataset (Rein et al., 2023) consists of graduate- level, multiple-choice questions in biology, physics, and chemistry. For our experiments, we utilize GPQA's "diamond" subset - a collection of 198 high-quality and extremely challenging questions. We sample 98 questions for validation and 100 for testing. Coding. HumanEval (Chen et al., 2021) is widely-used benchmark consisting of 164 Python programming questions. We sample 64 questions for validation and 100 for testing.
B.2 Verifier Engineering
For our experiments, we implement the verifier engineering method described in Section 3.4. To create our initial diverse pool M of 20 aspect verifiers, we vary the three key axes that define aspect verifiers:
1. Base model: Gemini-1.5-Flaan
GPT-4o-mini
2. Aspect to verify: Mathematical correctness, logical soundness, factuality, etc.
3. Verification strategy: Direct approval, step-by-step verification, solution rephrasing, edge case checking, etc.
From this pool, we then select domain-specific subsets M C M that maximize average performance across all generator LLMs on the corresponding validation sets. The complete list of verifiers and the dopain-specific subsets are detailed in Table 4. We choose Gemini-1.5. Flash and GPT-4o-mini as the base LLMs for our aspect verifiers since they are cost-effective for large-scale verification and enable us to demonstrate that combining multiple weaker verifiers can improve the performance of even stronger generator LLMs (Section 4.2).
B.3 Aspect Verifier Subsets
Table outlines all 20 aspect verifiers in M and which ones were selected for each domain- specific subset M
B.4
Generator LLMs
We evaluate eight generator LLMs (four closed-source models and four open-source models) and restrict our set of generator models to those released before September 2024. For closed-source models, we use gemini-1.5-flash-e 2024a), well gpt-40-mmmm-222244-7715 and 4pt-44-2024-08-06 (Achiam et al., 2023). For open-source models, we use Mistral-7B-v0.3 (Jiang et al., 2023), Llama-3.1-81 (Dubey et al., 2024), Gemma-2-99 and Gemma-2-277 (Team et al., 2024b).
and gemini-1.5-pro0001 (Team et al.,
16

---

Published as a conference paper at COLM 2025
Base Model
Aspect to Verify
Verification Strategy
MATH
MMLU-Pro GPQA HvaanEval
Math Correctness
Step-by-Step
Logical Soundness
Step-by-Step
Factual Correctness
Step-by-Step
Unit Conversions
Step-by-Step
General Correctness
Direct Approval
GPT-4o-mini
General Correctness
Summarize Solution
General Correctness
Explain Differently
General Correctness
Edge Cases
General Correctness
Common Mistakes
General Correctness
Domain Knowledge
Math Correctness
Step-by-Step
Logical Soundness
Step-by-Step
Factual Correctness
Step-by-Step
Unit Conversions
Step-by-Step
General Correctness
Direct Approval
Gemini-1.55Flasst
General Correctness
Summarize Solution
General Correctness
Explain Differently
General Correctness
Edge Cases
General Correctness
Common Mistakes
General Correctness
Domain Knowledge
Total Verifiers Used
6
14
Table 4: Overview all aspect verifiers in M. Checkmarks (V) indicate which verifiers were selected for each domain-specific subset Md. The table shows all 20 combinations of base models, aspects to verify, and verification strategies that we created (10 per base model). The bottom row shows the number of verifiers [Md for each domain.
B.5 Reward Model Baseline
Our reward model verification baseline (BoN-RM) uses Skywork/syoookk-eewrrd- -3. 1.1--8--..2 2 (Liu et al., 2024), the top scoring open-source 8B reward model on Reward- Bench (Lambert et al., 2024) at the time of writing. This pretrained reward model out- performs numerous larger models including 70B and 340B models, and can be run on academic-scale compute.
B.6 Prompts
For generator LLMs, we use a consistent prompt format across all models while varying the content by domain. Table 12 contains these domain-specific prompts. Note that we spent very little time engineering the prompts - just enough so that the output is in the desired format.
For aspect verifiers, each prompt consists of two components:
1. A domain-dependent system prompt (Table 13) that establishes the verification context (e.g,, mathematical problems, multiple-choice questions, or code implementations)
2. A domain-independenn verification prompt (Table 14 and Table 15) that specifies the aspect to verify and verification strategy
This two-part structure allows us to combine any aspect-strategy verification method with any domain while maintaining consistent evaluation criteria across base models.
17

---

Published as a conference paper at COLM 2025
C
Additional Results
C.1
Using All Verifiers
Table 5 compares BoN-MAV using all 20 aspect verifiers in M (without domain-specific engineering) against self-consistency and reward model verification. Even without engineer- ing domain-specinfe subsets M combining all verifiers remains competitive with baseline methods.
MMLU-Pro
GPQA (diamond)
Generator Model
MAV-All
Cons
RM
pass@1
MAV-AIl
Cons
RM
pass@1
Gemini-1.5-Flash
65.7
63.3
60.7
59.3
41.0
40.0
46.0
42.0
Gemini-1.5-Pro
70.3
71.7
69.3
68.0
49.0
45.0
49.0
45.0
GPT-4o-mini 65.3
63.7
62.7
62.3
49.0
48.0
44.0
38.0
GPT-4o
75.3
76.3
72.7
73.3
55.0
59.0
58.0
54.0
Table Performance (accuracy '%) BoN-MAV with all 20 aspect verifiers (without any tuning, labeled as MAV-all in the table) compared to reward model verification (RM), self-consistency (Cons), and the base pass@1 accuracy of the generator LLM. Using all verifiers without domain-specific tuning remains competitive with reward model verification and self-consistency.
C.2 Using Al-Generated Verifiers
Our aspect verifiers were initially created through a combination of AI assistance and manual prompt engineering, but having to hand-engineer every verifier would limit the generality and scalability of our approach. Here, we conduct an additional experiment using fully Al-generated aspect verifiers. Table 6 shows the performance of Al-generated verifiers compared to our original verifiers, self-consistency, and pass@1 baselines on MMLU-Pro and GPQA (diamond) using Gemini-1.5-Flar We find that using AT-generated verifier prompts can perform even better, demonstrating that MAV can be fully automated. Future work could investigate more effective methods for automating aspect verifier generation and selection.
Specifically, we provided GPT-4o with just 3 example aspect prompts from our original set and asked to generate 10 new prompts with different aspect-strategy combinations. Using these 10 purely Al-generated verifiers (without any manual engineering), we evaluated Gemini-1.5-Flan on MMLU-Pro and GPQA (diamond). For simplicity, we do not perform any verifier engineering for this experiment-ww selection. So we compare to the "MAV-All" results from Table 5. Below is one example of the Al-generated aspects created by GPT-4o:
all 10 AT-generated aspects without
"redundant_steps". "Look for any unnecessary steps that don't contribute to the final result. Think out loud. If the solution includes redundant or misleading steps, reply 'False'. every step is necessary and useful, reply 'True'."
Benchmark
MAV-Automated
MAV-All
Cons
pass@1
MMLU-Pro
68.0
65.7
63.3
59.3
GPQA (diamond)
47.0
41.0
40.0
42.0
Table 6: Al-generated aspect verifiers. Performance (accuracy %) comparison using Gemini-115-Flass the generator LLM with candidate outputs.
18

---

Published as a conference paper at COLM 2025
Generator LLM
Rubric MAV-AIl
Cons
pass@1
Gemini-1.5-Flash
62.0
65.7
63.3
59.3
MMLU-Pro
GPT-4o-mini
64.0
65.3
63.7
62.3
Gemini-1.5-Flash
39.0
41.0
40.0
42.0
GPQA (diamond)
GPT-4o-mini
42.0
49.0
48.0
38.0
Single rubric separate verifiers. Performance (accuracy %) comparison between generating single rubric for all aspects in just one verifier response (Rubric) versus using separate model calls per verifier aspect (MAV-All) with 16 candidate outputs.
Table 7:
C.3 Scoring Rubric of Aspects
In MAV, the verifiers evaluate each aspect in separate LLM call. Here, we try asking an LLM verifier generate a single "rubric" containing the approval scores for each aspect (all in single LLM call). We find that generating single rubric for all aspects performs worse than using a separate model call per aspect (see Table 7). This is likely because having just one response forces the model to think about many different aspects in context, whereas separate model calls allow the LLM verifier to put all of its focus on one aspect at a time. Having multiple LLM verifier calls may have more expressive power, and this could be an interesting direction for future work to explore further.
Specifically, we instruct the LLM "rubric" verifier to think about each aspect in a single response and generate python dictionary mapping aspect names to True/False values (after it has finished thinking about each aspect). We use the exact same 10 aspects as in the BoN-MAV results, but obtain the approvals for all aspects simultaneously in a single response rather than splitting across model calls. As in the BoN-MAV results, we use Gemini-1.5-Flash and GPT-4o-mini the base LLMs for our verifiers, so there are actually 2 model calls: Gemini-1.5-Flasi generates one rubric and GPT-4o-mini generates another rubric. For simplicity, we do not perform any verifier engineering. So we compare the "MAV-All" results in Table 5 (which uses 20 separate verifier calls: 10 aspects for 2 base LLMs).
C.4
Scaling Generator CoT
To compare our approach to a "thinking" or long-Chain-off Chain-of-Thought (long-Col) method which scales generator LLM reasoning tokens without test-time verifiers, we implemented a reasoning baseline where we iteratively prompt the same base models to continuously extend their chain of reasoning until approach on the MATH dataset (approximately 80k tokens per question on average). This comparison is appropriate since both methods scale test-time compute using zero-shot prompting without fine-tuning for mathematical reasoning or using RLVR. Table shows that when controlling for token count, our method performs better than simply scaling reasoning tokens within a single model call.
similar number of tokens our MAV
Model
BoN-MAV (ours) Scaling Reasoning Tokens
pass@1
Gemini-1.5-Flash
66.0
53.0
52.7
GPT-4o-mini
73.0
72.3
69.0
Table Comparison with scaling reasoning tokens on MATH. Performance (accuracy %) comparison between our BoN-MAV approach and a baseline that scales reasoning tokens within a single model call, controlling for similar token counts. Our method outperforms scaling reasoning tokens when given a similar computational budget.
19

---

Published as a conference paper at COLM 2025
C.5 Process Reward Models (PRMs)
For our baselines, we focused on outcome-level reward models and self-consistency since they are most similar to our approach (selecting between complete candidate outputs). However, here we perform preliminary experiment using Process Reward Models (PRMs) that select between candidate responses at each step of the solution. Note that this represents a fundamentally different approach that operates at the step level rather than the response level. Using GPT-4o-mini with Qwen2.5-Math-PRR. and branching factor of 4, we achieved 68.0% accuracy on MATH, which is higher than the pass@1 base accuracy but below our BoN-MAV accuracy of 73.0%. This comparison is not entirely appropriate since PRMs focus on step-wise verification during generation using a single verifier, while our approach focuses selecting between complete responses using multiple verifiers. A more direct comparison would involve using MAV-like approaches to select between intermediate steps using multiple verifiers, which we leave for future work.
C.6 Averaging Different Reward Scores
Here, include an experiment that demonstrates the challenge of combining heteroge- neous reward models that operate on different scales and have been trained on different datasets. Specifically,
average the reward values from three different reward mod- els: Skyyoork/Skkooor--Rewrrrdrr 3.1---m------- 1--8--0.. 2, nicol hnooi/nn-LaLamaa 1-8B-v2 and For each candidate solution, we obtain reward scores from all three models and average them to create combined reward signal for best- of-n selection. Using GPT-4o-mini as the generator LLM with 16 candidate solutions, we achieve 72.7% accuracy on MATH and 63.3% on MMLU-Pro. This is similar to our single reward model baseline (72.3% and 62.7% respectively) but worse than BoN-MAV (73.0% and 67.0% respectively). So our multi-agent verification approach with diverse aspect verifiers outperforms combining heterogeneous reward models in these cases. See Section 3.1 for discussion of the challenges of combining neural reward models trained on different datasets.
allenai/Llama-3. 1-8t-Instrruut-R
C.7 Oracle Baselines
We include oracle baselines for each model across all four benchmarks in Table 9 and Table 10.
MMLU-Pro
GPQA (diamond)
Model
pass@n
B-MAV
Cons
pass@n
B-MAV
Cons
Gemini-1.5-Flash
81.0
66.7
63.3
80.0
42.0
40.0
Gemini-1.5-Pro
85.7
72.3
71.7
86.0
49.0
45.0
GPT-4o-mini
80.3
67.0
63.7
79.0
50.0
48.0
GPT-4o
88.3
75.7
76.3
87.0
59.0
59.0
Mistral-7B
43.3
36.7
25.7
54.0
36.0
32.0
Llama-3.1-8B
87.0
59.3
55.3
90.0
43.0
36.0
Gemma-2-9B
74.0
57.7
54.3
69.0
34.0
36.0
Gemma-2-27B
81.0
62.0
58.3
81.0
41.0
40.0
Table 9: Oracle baseline (pass@n) results for MMLU-Pro and GPQA (diamond). Performance (accuracy %) with π = 16 candidate outputs showing the upper bound performance (pass@n), BoN- MAV performance (from Table 1), and self-consistency (Cons) baseline for MMLU-Pro and GPQA (diamond).
20

---

Published as a conference paper at COLM 2025
MATH
HumanBval
Model
pass@n
B-MAV
Cons
pass@n
B-MAV
Cons
Gemini-L.5-Flash
76.0
66.0
59.0
82.0
80.0
79.0
Gemini-1.5-Pro
85.7
72.7
70.3
92.0
88.0
84.0
GPT-4o-mini
85.3
73.0
74.7
91.0
84.0
87.0
GPT-4o
87.7
76.3
77.3
98.0
92.0
95.0
Mistral-7B
38.3
26.0
22.0
67.0
59.0
46.0
Llama-3.1-8B
78.0
61.7
61.0
87.0
75.0
62.0
Gemma-2-9B
70.0
58.7
51.7
56.0
32.0
25.0
Gemma-2-27B
75.7
62.3
55.7
84.0
76.0
66.0
Table 10: Oracle baseline (pass@n) results for MATH and HumanEval. Performance (accuracy %) with π = 16 candidate outputs showing the upper bound performance (pass@n), BoN-MAV performance (from Table 1), and self-consistency (Cons) baseline for MATH and HumanEval.
D Additional Illustrations and Extra Tables
Figure 7, Figure 8, and Figure provide additional examples of how multiple aspect verifiers evaluate single candidate output. Figure 7 demonstrates verification using multiple strategies with single base model on MATH (Hendrycks et al., 2021). Figure 8 shows verification of a coding solution from HumanEval (Chen al., 2021). Figure illustrates verification of correct solution from GPQA (diamond) (Rein et al., 2023), showing how different base models can assess the same aspect differently. Each figure follows the same format Figure from the main paper.
Table 11 shows the accuracies for MATH when only counting questions with pass@n=1
(where at least one sampled solution is correct).
Model
MAV
Cons
RM
Gemini-1.5-Flash
87.3
77.6
81.6
Gemini-1.5-Pro
87.2
82.1
82.9
GPT-4o-mini
86.3
87.5
84.8
GPT-40
88.2
88.2
92.0
Mistral-7B
67.0
57.4
56.5
Llama-3.1-8B
80.8
78.2
70.1
Gemma-2-9B
82.4
73.8
78.6
Gemma-2-27B
83.3
73.6
78.4
Table 11: Accuracy on MATH for questions where at least one sampled solution is correct. Perfor- mance (accuracy %) when only counting questions where pass@n=1. We compare BoN-MAV (B-MAV), self-consistency (Cons), and reward model (RM) selection with 16 candidate outputs.
21

---

Published as a conference paper at COLM 2025
Sampled Candidate Output: (INCORRECT)
solve problem, we identlly which integers between 200 can be classified "bac" and also chack they are equivaent
Quastion: Let an integer a be bad if thene no way to insert colon ammnn the digits of form time houlve-houur digital clocs), good insertad
Given the range of integors from 100 to 200, the first dight of is fixed at (since we're looking at numbers benwean form tab, (with Oj. 11 twith 1), (witt a
100 200). Tharefore, number
and
subsequent dights.
valid hours that can be formed
solon
a time.
valuss of to S, tais valid hour.
For esempus, 100's is good because we can writs 1:00, but 164 is badl. How meny bad intogars betusean 100 and 200 equivalent to mod Cornect Answer. 10
The minute part b can be from 0 but for the time to be valid, must be between O and Therefore, the bad integers will be:
For 4, 6, 8, 140, 150, 160,
190
Theee integers cannot
from
3. Finding Integers that are Equivolent to mod 4: Next, we need to find which of these bad integers are equivalnnt to 3 mod 4. We can caliulate x mod 4 each bad integer:
Generated Answvrr 0
130 mod 4 140 mod d 150 mod =.2, 180 d 0, mod d 2, 180 mod 4 190 med d integira give remeinder 3iwhen divided
Aggregated Veritication Score: 1/5
Nona
Conclusion: Since there are no bad integers between 100 and 200 that are equiviient to mod 4, final
ansower is O.
Aspect Verlfer
Aspect Verifier #2
Aspect ffeer #3
Aspect Verifier #4
Aspact Verifier AS
Bane model: Gemini-) 1- flanh
Boon model: Gaminnt S-fftehh Aspect: Commernens
S-Piten Aspect: Genera Corisctnesa Sciotegy: Atroct.ooorooa)
Bane model: Giemini-11 5-Flish Aspest: Goneral Correctness Strotogy: Summatic Solution
Base model: teem mini- F-Flishh
Axpwst:: General Cornectness
Axpect: Factual Connectmees
Strotogy: Explain Difffremtly
Strotgyr Stop. by Stop
Strotogy: Stop by Stop
Diey vanik
to sum prpeooat wakrice, the betwean 130 5001 ane casa fidd sakd sina when color is asersiiiars readi.
Orey po ai waluice
eingann betweerr roo and 20 plat lause
Stap The aup the sime
kamatt comeiss Aviid 12 nowr
laut vmnge iscea thes toe mnscc
indked sepeaartdd iecirn with the
dingee
spec fidd sanges for traas and tutall
trawringe
U.Undertanding villd fhime: "he soludion
Step The eeeeerreeee eidor "tacc" prequra
The sot udeen atart correctty son mattenng the zia D--iaur
thetlet
fham ro on varie
unagees li ellanca vesit (ee ised
ti beberveee Orand for remutie..
coreati deeniiieate mhst weil ranga
60 50
dignto betweee and
alre careac. We reed the dig
conmectty
sut numoerr be
sibue tympphort tetegger this
ics an graut
196 io sise aooel aeviingaveee
rfrec treaat bince leainn cu aniture betwore and
Staz aslution comestia
130 200, rimiingo caloruu
The sot stue thee juta tha "heat imcopers
15, 11, on whick pormi wis to timee aelen aui trrcouhh
reldd houes that canbe borned from
The vieeee them Jes
les commett
scenthiing pendirmmgns says
Masiir wur the sauuee eneisingivee gromms vig, 196, 19A, sat, 14th yeu "hua"
Dreai The aolutien parsect to iifier
twa tink thl and 1996.
Taal,
bow Taasel Best in remoernncly
Meddd artcaaaall anaillll sine. Hiseevess
chnoolegg Merenii roveniton Then
san, 1911 1191, 1nd thenen
wnkiiin dlyuce wal.braaanntee... munber snrking
in soneciioo soice vehicri li
hekker to
aatieesden worl for
venaaii coliii Berier
moci actimm trageer
andicia
sak 124, are also caluire
salld
coaw itgge pelddi
now thery bed tingers therrmett
aspeertto de cers The dunt calic rntiuiittenn van comned Shweniens phopeeaee
APPROVAL = FALSE
APPROVAL FALSE
APPROVAL FALSE
APPROVAL TRUE
APPROVAL FALSE
Figure 7: Multi-agent verification for single solution (additional example). An illustration similar to Figure 2. Five different aspect verifiers evaluate an incorrect MATH (Hendrycks et al., 2021) solution sampled from Gemini-1.5-Pro. All verifiers use Gemini-1.5-Flas as the base model but vary their aspects to verify (e-g-, general correctness, mathematical correctness) and verification strategies (e.g. direct approval, step-by-step verification). Four verifiers correctly identify the error, while one verifier using general correctness through summarization incorrectly approves the solution. This demonstrates how diverse verification methods can produce more reliable signals even when using single base model, as multiple verifiers can compensate when another fails.
22

---

Published as a conference paper at COLM 2025
Question:
You hungryy rubbit, and you direaidy have coton ocrtain umber nots but now nood cor more corrona to complese the day's meals. you should return an arrey ot tots mumber at sunen cemmote atter meatas
Correct Answer:
eetinumber, remaining): il[nead remaining): return numbar- + need, remaininggnead]]
the number of corrons loft ofter your mools] i'hhere are not enoughh remsiningcaroos, carross, you will eae maliinng cenrots, but will etill hungry.
return [ number remaining .0]
10, 101-1111.00 (11.0)
con22, 11, 9) - [7.0]
Sampled Candidate Output: (INCORRECT) estinumber, remaining): natan min]number remaining, need]
Varubbles:
@rnumee jntenger
the number carmss that you hrace
return (eaten, remaining (eaten number]]
Peiced intogir
you
Adgregated Verification Score: 1/3
Svemaiiing
number of remoiiing
stook
Aspact
Aspect Verifier Verifteerifier +3
Aspect
Base modall: GPT-4c-mini
model: Gemini-1 1-3-lish
Base model: Gamini-1. S-Fash
Aspectt Lngical Soundness
Aspect:
Aapect: Math Cor correctness Strategy: Shep-hy-Step
Strategy: Step- Sturt
Strategy: Edga Caass
Latk proposed solution
handloa cogrrozaua
con cornaclly.
Ohay, lot cnalpas the code soup stap.
colculate how many comota robbit will have caton ofter the moal.
Calculating
The line minjumber: remaiiing, nswd) intwndded to
autreasion semaiing: (wannn numberd does cncertl
The coltutation of remaining carmre ino nact, This
carrzte nattnn rrtakin
Thix line calcu Linee the rons numberea
minyummer samtiinng need):
+I wes, number semairing gowe the totsl cammte the nichtt could
roflett rumber remuniing campos outing, Inatond,
pattenialyy oity nut camote aloendy saben plus the reme ming onsec.
The mide function used ensure that rotal caton does anceez mawes.
aoster then numbect cstenia kowu than aqual the wena ring camms shoule aimply be matning ng.
joom ringt and tocal number camola
reoded reodsltooodi (roods This oo vest becoured thee sabiit snt than meeds more
correct because ruboit nends camots than eat, onlyy eat what
The logie thee not hansla case vctern
wore
Calculating
The return statemere return numniiinng-
tctal enten should rumber memaitt ring
semeining shoule be
2. returm [eatur, nama ninng toaten
rumbbor)) This calta remainingg rond jaaneng
numberj]
Edge Cases)
now many carncts are cutt after the meat.
The exprossion zona ning (psten mumbrr moonn coloulate
Coss usn0, 1.0ahaauld ratum (0, eot).
siwaady eseen crummed from nital somainnng curoot (eomanningg ningl. trutao concec bocauser cocurttll the
howeree this aepressionia incoreet. The correct cslcutation
should eimple reraiining (ewmen number'' only watten is promter
1)should rotum (otata only
than caten loss or equal number, the remaining
should not change,
nos caton ony aduitional
Cara-d 1i0 mitmm!! (15, Jonnwer)
Cara ward, Riahaald rntum (15, (ooon
Caead wontl insencalimmemm (onneer))
4. trrat Catculatior:
Slshuctenctmm o)
the rabait the remaining camona, it aamouii be hungry. which meons the total waten should be number remraiinng and the soma ning should b.
Care 1000, 1000, sy (1000
ift rabut cato loss than nocds, the romoining should be calzusittdd correctl browd en new mony actually
jatwady axtstied no merainngg.
Conclusion:
The propoced solution hes ont outniing remei rig canats and coos comtott handle cose
analysts coloulation remaining stlll needs
reeraiiing
mibtit iemasiinng
TTerefore, solation dozs handla all curecc
The Theiefore, propoaed selution incomect.
and soundaay cardeiiddoo derrddiomm conseity.
APPROVAL FALSE
APPROVAL FALSE
APPROVAL -TRUE
Figure 8: Multi-agent verification for single solution (additional example). An illustration similar to Figure 2. Here, three different aspect verifiers evaluate incorrect HvmanEval (Chen et al., 2021) solution sampled from Gemini-1.5-Flash Two verifiers correctly identify the error through careful analysis, while one verifier incorrectly approves the solution.
23

---

Published as a conference paper at COLM 2025
Question: You deetiited a raw fgand-acspter par in aepreazed neum! crest and were deestend develppenttal line the rintrrttion
Sampled Candidate Output: (CORRECT) lerr''s saep sralyde:
ght apprahiicss
The copurimerral setup: The reacarciiee ber and try ng emages the intrrcction secptor
the ligand The aignn i colled collst two sigha helces. You cretn tic arooec los Cin vector with the ligend ORF at the 5' prettoon hussd ntherry the recector OSF or the of
erracclllter helces, three beta shosts that caast canding
pour Plood to JCPP under the contro, of spoo to promotoro and rogi tugulattor
luaed sOPP the C-temmruss The lgand is unier the C3A promatss and enchwery
enspin lndf ind
ligard the
2. Creiie detwon The vector stlire Cre-tox system dhool gene oapro The
control Red 2. meara that enpressto both gereo blocked aniil Gra recombe noo la proseett.
JusP litt
sorrt thene aus enop in the io stop- in0313 casseme You had aleo rateed
artioodles against the oreeesse prosere and row use thom sout the construct exaresoior on the Westembali
thei calle This cores in introcted scttaans Eim recrnttr anw spach oally in reurr)
S0110
The
plecevi trannfoction to maay
cullars.
vigns conetruct
model homorgguus
aroundd You cress to SOX Cr-
the ligind
hyrricygous miko; after the oftign igg is born you vicusiio Bremunder the itofuuorssent
seziamn mast siatly ressen
confocti tooosiipp You not obse a greon sight, Which the anoowrs bolow
Lack algnati Deephs the sebun, the resesrcham didn oeeeeve
ng
A.sayyyingg aptionss Weite
(M) mhe peeptor-ear constructio in stuck in the desgg. I01 the unhaneer for the ligand und receptor oxpresaiinna mitsinng.
ligand the raceptor are in paraciitn intationship.
deting neatty callet innnt recuusaitty plevent the eun pposcomn on Fialiin
(o) the receptor and cOFP and not the fromm
whee should mech the cell suffica
comutract was
tramamericcas somaan the lacaptir,
Correct Answer: (D)
expresaiion
layele lntkedt
Resegpoi and sbann out comee This the most theyy exptaratioon the
Generated Answwer: (D)
Uraroaatted rocilyy reaulling sack graoon sightl.
RGG
Aggregated Verification Score: 2/3
The snower to
Aspest Veriffor 81
Aspect Verifier #2
Aspect Verifler #3
Reaw model:: GP 1--o-min
Busse model: Gamanic S-Fllshh
Bose model: Gamiin-1 1- Fleshh
Aspect: Soundness
Axpect: Logical Soundmies Strategy: Stop Shop
Aspect:
General Correctneess
Stratugy::
Strategy: Explain Differentiy
Let's anelyoo the proporeed asiution step by ship
Okew, ln's go nough the sotution step by stop.
Okay let's break down the solution in citte went
exprrmsoral
description exporiennta
Step The expetmernt arrup accucriity describadd The reseanchers are trying to understand the inseraction of
setup la accurate The rensa chhars are indire trylig to anaiosa the
The epp espar alms to study the insrectionn of
interaction a rece-peeccporr pair neural prest calla uating
now regregeeennn spepptor paw
Igar Igard-rcca recaptor painin maurs creat culls. The
elemerns. This step la legicalllyy sound.
liai receptoor grees, eoch euchi to
Cre-len symamm The eeplanation the Cre-isn symarm to
fuo uscare tn Cheryy and aGIT
exprainect toshs
cossettee to contrel tho rxpresainn of the agand andi receptor wet armouuli and. Thas step ds segioally soured.
ekoe laaiP en anotoctt22 stop toxozti cesscttes
connec pano peppoasion penourin thott both ganoo
designad block expressioo on undil recembtiran mass
are neo wose only necrst enta whare
legically
Creasing with soorio milec The explanation reganding the
cross with S2k10 ii ricic accas soddellllddedd recovnte
Stop Tike easssing soaeno Ore mice ins 1 of
The prokiem thar the researt donnt was
anettheirwwian correctiy stated. ta soop legically sourd.
geer faovee www.ictiiinggi....
remoiing stop cassurten and
buth figigge recnppoo This step is soundd
Lack rigrall ohereeern eiggene siguat
Step The grean la atzeerved remutt, and
andienting wdli enc mecilannii nuttoc soveectty nees the stage for analycing potartial veasers the back empprnaaion 1h ahie
ta probbaa the reasarchera ane ingg esplaats This
"AA Parserra ratatonaaa ge a insievanti the
ehop Augisa
atdrr sxgrastion garastine relani namp whether the recrpppee is tesuree.
lagicaly
Step Si The amabea a ch options to whene chings gut
tricky.
E. Recsprar EFP stuckin Golgi
relaturratis
designod
walat craita could unlt. posa cuplain the
Dition AIA possrinne retarin tip weor zistt prosest
with embrans
which alooud
wilhh supressioe This ansiasss regioally
empress for of the wowp--aagee recepponn Thials valld
dtnch sa the manterinn, act the Beiig
states th sooaliiiiy itat propoeeee
Seceptor
deslgn
Cpllton The necupraz beinng stuck in the Gingila
engnamaaan busi the
complendly prewent
the comatruett This rasoniingia la naasomatiia but coocp not fully
poor iall fur lens by plans the trnn es soone domaiir.
exprematert The woctor usea a sihangi simtee
pour sommeshor log cully sourd.
"pttion ClA entancer could lnndita low
wit rovviintiee
smancss lont but mocoonn by corppea provrtii Thra
C. Recestor and uCFF our frmme This the
-. Mourr coheresc The proporeed addution sugacatt to
most Boaty unparattens the wesprar and
coundiien repeeepesiie compiiantt nisunn exprestiion on. this maroning ir eoundi se.
This wiveerthengg twnn doown the
recupicr and uGIP are out of hrame the sCPP would not
fooms the xdFP promivson wiin i' nandiaasss ansd
ackoondadgeu the poterttll for low ropromaton with ruting out
tranalated conecdy mesud ing the tack of grean sigrall
property lascing to the lack grean fluo vecemro.
Ib Resnewv and efFP of tha The rmpperaeeekvee
Howwent ulion crans movr thet ths enveriiiie
condiuueesttte that mie authe mone lnnly reprovatiion thel
ensiaration Thas where the logic factera
incuedionn prewaippenaaaag aww.tattra sumulttion the e2in)) miscnnerninngg opiially sount.
Siree stazain propeouti urs togiceilyzuund and
leedlo coffuent constaaionn fenl can chade mo sostication.
APPROVAL TRUE
APPROVAL - FALSE
APPROVAL - TRUE
Figure 9: Multi-agent verification for a single solution (additional example). An illustration similar to Figure 2. Here, three different aspect verifiers evaluate a correct GPQA (diamond) (Rein et al., 2023) solution sampled from Gemma-2-27B. The verifiers vary in their base models, aspects to verify, and verification strategies. Notice that Gemini-1.5-Flast incorrectly rejects the solution when evaluating logical soundness but correctly approves it when prompted to explain the solution differently, Mean- while, GPT-4o-mini correctly approves the solution when evaluating logical soundness. Different base models can produce different evaluations of the same aspect.
24

---

Published as a conference paper at COLM 2025
Domain
Generator Prompt
MATH
You are a helpful assistant skilled in math problem-soloing. Alwaay end your solution with the final numerical answer enclosed in LaTeX [boxed{] notation. If there is no solution, reply with an empty boxedd Please solve the following math problem step by step: < Question Provide your detailed solution below:
MMLU-Pro
Answer the following multiple choice question. Think step by step before answering, and then output the answer in the format of "The answer is (X)" at the end, where X is the LETTER of the correct answer.
QUESTION: < Question >
Think step by step, then end with EXACTLY "The answee is (X)", where the LETTER of the correct answer. Do not include the answer lext itself, only the letter.
GPQA (diamond)
Same as MMLU-Pro.
HumanEval
Read the following function signature and docstring, and fully imple- ment the function described. Your response should only contain the code for this function.
Function Signature and Docstring
Table 12:
Generator Prompts. Generator prompts by domain. Each domain uses one consistent prompt across all generator LLMs.
Domain
Aspect Verifier System Prompt
MATH
You are critical verifier tasked with ecaluating mathematical problem- solving. You will be presented with question and a proposed solution. Your job is to carefully go over and analyze the solution. Follow the instructins.
MMLU-Pro
You are a critical verifier tasked with evaluating multiple-choice question- answering. You will be presented with question, the multiple-choice options, and a proposed solution. Your job is to carefully go over and analyze the solution. Follow the instructions.
GPQA (diamond)
Same as MMLU-Pro.
HumanEval
You are critical verifier tasked with evaluating code implementatiinn.. You will be presented with a prompt and code implementation. Your job is to carefully go over and analyze the code. Follow the instructions.
Table 13: Aspect Verifier System Prompts. System prompts for aspect verifiers. These provide domain-specific context for the verification instructions in Table 14 and Table 15.
25

---

Published as a conference paper at COLM 2025
Aspect to Verify
Verification Strategy
Aspect Verifier Prompt
Math Correctness
Step-by-Step
QUESTION: <Question>>
PROPOSED SOLUTION: <Solution>>
INSTRUCTIONS: Go over each step in the proposed solu- tion and check whether it is mathematically correct. Think out load. If you reach a step that is incorrect, stop and reply 'FINAL VERIFICATION ANSWER: False'. If you get to the end of all the steps and each step was correct, reply 'FINAL VERIFICATION ANSWER: True'.
Logical Soundness
Step-by-Step
QUESTION: <Question>>
PROPOSED SOLUTION: <Solution
INSTRUCTIONS: Go over each step in the proposed so- lution and check whether it is logically sound. Think out load. If you reach a step that is not logically sound, stop and reply 'FINAL VERIFICATION ANSWER: False'. If you get to the end of all the steps and each slep was logi- cally sound, reply 'INAL VERIFICATION ANSWER: True'.
Factual Correctness
Step-by-Step
QUESTION: < <Question>>
PROPOSED SOLUTION: <Solution>>
INSTRUCTIONS: Go over each step in the proposed solution and check whether the facts presented are cor- rect. Think out load. If you reach step with incorrect facts, stop and reply 'FINAL VERIFICATION ANSWER: False'. you get to the end of all the steps and each step had correct facts, reply 'FINAL VERIFICATION AN- SWER: True'.
Unit Conversions
Step-by-Step
QUESTION: < Quuestion>
PROPOSED SOLUTION: <Solution>>
INSTRUCTIONS: Check if the units are handled cor-
rectly in each step of the solution. Think out loud. If you find any issues with the units, stop and reply 'FİNAL VERIFICATION ANSWER: False' If all units are han- dled correctly, reply 'FINAL VERIFICATION ANSWER: True'.
Table 14: Aspect Verifier Prompts (Part 1). Aspect verifier prompts for each aspect-strategy combina- tion. These prompts follow the system prompts in Table 13.
26

---

Published as a conference paper at COLM 2025
Aspect to Verify
Verification Strategy
Aspect Verifier Prompt
General Correctness
Direct Approval
QUESTION: <Question>>
PROPOSED SOLUTION; <Solution>
Is this solution correct for the given question? Respond with ONLY 'FINAL VERIFICA- TION ANSWER: True' or ONLY 'FINAL VERIFICA- TION ANSWER: False'. Do not provide any explanation or additional lext.
INSTRUCTIONS:
General Correctness
Summarize Solution
QUESTION: < Question>
PROPOSED SOLUTION: <Solution>
INSTRUCTIONS: Summarize the solution in your own words, explore anything you think may be incorrect. Think out load. If you find something that's incorrect, stop and reply 'FINAL. VERIFICATION ANSWER: False', If you' 've gone over the solution and everything seems cor- rect, reply 'FINAL VERIFICATION ANSWER: True'.
General Correctness
Explain Differently
QUESTION: < <Quetiion>
PROPOSED SOLUTION: <Solution>>
Explain the solution in a different way than it was presented. Try find any flaws in the solution. Think out load. If you find something that's incorrect, stop and reply 'FINAL VERIFICATION ANSWER: False'. If you' ve gone over the solution and everything seems correct, reply 'FINAL VERIFICATION ANSWER: True'.
INSTRUCTIONS:
General Correctness
Edge Cases
QUESTION: < Question
PROPOSED SOLUTION: <Solution>>
INSTRUCTIONS: Check if the solution handles edge cases and boundary conditions, test extreme values spe- cial cases. Think out loud. If any boundary conditions or edge cases fail, stop and reply 'INALL VÉRIFICATION False'. If all boundary conditions and edge cases are handled correctly, reply 'FINAL VERIFICA- TION ANSWER: True'.
ANSWER:
General Correctness
Common Mistakes
QUESTION: < <uuesion>>
PROPOSED SOLUTION: <Solution>>
Check if the solution has any common mistakes, calculation errors, or misconceptions that typi- cally found in this type of problem. Think ouf loud. If you find any common mislakes, stop and reply 'FINAL VERI- FICATION ANSWER: False'. If no common mistakes are found, reply 'FIALL VERIFICATION ANSWER: True'.
INSTRUCTIONS:
General Correctness
Domain Knowledge
QUESTION: < Question
PROPOSED SOLUTION: <Solution>>
INSTRUCTIONS: Check if the solution correctly applies relevant domain-knowledge established theories, and stan- dard practices for this type of problem. Think out loud. faay domain knowledge is misapplied or violated, stop and reply 'FINAL VERIFICATION ANSWER: False', If all domain-specific knowledge is correctly applied, reply 'FINAL VERIFICATION ANSWER: True'.
Table 15: Aspect Verifier Prompts (Part 2). Aspect verifier prompts for each aspect-strategy combina- tion. These prompts follow the system prompts in Table 13.
27


Reviews:
[{"review_id": "SyPaBpUp4Q", "reviewer": "Reviewer_6Jrp", "summary": "They propose applying additional test-time compute to verification by implementing multiple \"aspect verifiers\", which verifier different aspects of an output (e.g. logical soundness, general correctness, etc...). These aspect verifiers are produced via different prompts to an LLM judge model. They then aggregate the different judge outputs by averaging their scores. They then run best of N against this multi-aspect verifier (MAV). To obtain the set of aspects for a given task, they perform \"aspect engineering\", which involves selecting different random subsets of aspect verifiers from a large pool and then choosing the set which performs the best for a given task. They show that their approach enables effective test-time scaling not only in terms of the number of sampled outputs for BoN, but also by scaling the number of aspect verifiers. They conduct experiments on MATH, MMLU-Pro, GPQA, and HumanEval and find that 1) MAV outperforms baselines like standard verifier and majority voting with BoN scaling on all tasks except for GPQA; 2) in most cases, increasing the number of aspects improves performance; 3) using multiple aspect verifiers with small models can effectively verify larger model outputs; and 4) finally the conduct some analysis of how the specific set of aspects used impacts performance, finding that the diversity of the different aspects is important for maintaining high performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* They conduct fairly thorough experiments with multiple LLMs and multiple evaluations to understand the benefits of applying multiple aspect verifiers to improve verification capabilities\n* Their method is very simple and would be straightforward to reproduce\n* The paper is clearly written, and everything is easy to follow and understand\n* The problem of improving verifiers with additional test-time compute is a very interesting research direction and they show strong results using a simple approach", "reasons_to_reject": "* They method requires hand engineering multiple aspects for a given, limiting it's generality and the ability to scale the number of verifiers.\n* Moreover, having the different aspects be done in separate LLM calls seems unnecessary. Why can't you just provide the LLM the full rubric and have it output a score for each item in one generation?\n* It is unclear whether you perform the \"aspect engineering\" on the test set or on some validation set. If it is on the test-set, I would be concerned about the validity of their findings. If it is on the validation set, then they should make this clear in the paper.\n* Is the MMLU and GPQA done with a chain of thought? Without a chain of thought, it would be pretty hard to verify I feel.", "questions_to_authors": "See the reasons to reject above.", "extra_scores": {}}, {"review_id": "gNTCoqYAaO", "reviewer": "Reviewer_D7wR", "summary": "The paper presents a novel approach to reasoning with language models by introducing BoN-MAV (Best-of-N Multi-Agent Verification), which integrates best-of-n sampling with multiple verifiers. A verifier is an LLM prompted to verify some aspect of the generator's response. This work describes a new way of scaling test-time compute by scaling the number of verifiers (as compared to scaling the number of candidate outputs). The paper evaluates several different generator LLMs and several sets of verifier LLMs, and proposes a method for selecting which verifiers to use for a particular domain. The paper evaluates math, coding, general knowledge & reasoning, and graduate-level reasoning.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors present strong experimental evidence that BoN-MaV works: they evaluate several generator models, as well as several sets of different verifiers, and show its performance over self-consistency. The authors also show that their technique scales as the number of verifiers increases.\n\nThe paper is also well-written and presented.", "reasons_to_reject": "The paper as a whole is quite strong, but it is missing a few more interesting baselines that weaken the authors' argument slightly.\n* While the authors test many different LLMs for the generations and many different LLMs for the verifiers, the authors only test 1 8B RM model. There appears to be a stronger reward model released around the same time as the model they chose to use was released (they chose [Skywork/Skywork-Reward-Llama-3.1-8B-v0.2](https://huggingface.co/Skywork/Skywork-Reward-Llama-3.1-8B-v0.2) while [Skywork/Skywork-Reward-Gemma-2-27B-v0.2](https://huggingface.co/Skywork/Skywork-Reward-Gemma-2-27B-v0.2) was available.). Given that many of their verifiers were state-of-the-art closed-source LLMs, choosing not to use as powerful an open-weight reward model feels like it's not a fair comparison of methods.\n* Another strong area for comparison would have been reward models designed for reasoning models, like process reward models (PRMs) or fine-grained reward models. PRMs are cited and mentioned in the paper, but none were chosen for comparison. Many PRMs are made specifically for verifying different components of a model's outputs, and, unlike general-purpose reward models that grade the overarching output of a model, will more likely grade the correctness of a model.\n* Lastly, no effort to evaluate heterogeneous reward models (the authors provide reasons why it would be difficult) was made. Even something as simple as finding an average threshold to use to convert all reward values to 0 or 1 and average would have been helpful.\n\nAgain, these weaknesses do not take away much from the paper, but it would be substantially better if the authors had shown that BoN-MaV was much stronger than these baselines mentioned.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JtmRl5aR7S", "reviewer": "Reviewer_G6Pq", "summary": "This paper proposes a new test time computation method that exploits multiple verifiers. The proposed approach uses off-the-shelf LLMs as aspect verifiers to evaluate an LLM's output from multiple different aspects. Multiple verifiers evaluate each output of an LLM with a binary score,  and the proposed method uses the average score to select the best one. The proposed method is easier to use since it does not need additional verifier training. The paper compares the proposed method with the self-consistency method and a best-of-n method using a reward score. The experimental results show that the proposed method outperforms baselines in MATH and MMLU-pro benchmarks, and the multi-agent verification can achieve weak-to-strong generalization.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clearly written and easy to read.\n2. The idea of using multiple verifiers is simple but interesting. \n3. The paper evaluates the proposed method with multiple benchmarks and using multiple LLMs for the generator and verifier.", "reasons_to_reject": "I agree that the proposed method is convenient since we can use off-the-shelf LLMs as verifiers. However, it is still unclear whether multi-agent verification is an effective approach because off-the-shelf verifiers (GPT-4o-mini and Gemini-1.5-Flash) are generally more powerful than the 8B reward model used in RM. Therefore, it is possible that the proposed method's good performance comes from the power of large LLMs, not from the use of multiple agents.", "questions_to_authors": "1. Table 1 shows that the proposed method's gain over baseline methods tends to be small when generator LLMs are powerful. For example, GPT-4o shows the best performance among others in all the tasks, but B-MAV does not outperform Cons and RM for GPT-4o. Can you explain why the proposed method does not help much for strong generator LLMs?\n\n2. A simple baseline method is to ask a single agent LLM to evaluate a sentence from multiple aspects rather than using multiple agents. Have you tried this?", "extra_scores": {}}, {"review_id": "OScEh9I1Hk", "reviewer": "Reviewer_tRY4", "summary": "This paper proposes to improve LLMs outputs at test time by generating n answers with the LLM and selecting one answer (Best of n) using the binary approval rate of several smaller models judging the answers that have various task specific prompting.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well presented and well structured.\nThe experimental results are interesting and show an improvement over other BoN selection methods (reward model, self consistency) as well as better scaling with the number of candidate outputs.", "reasons_to_reject": "The method is similar to the one used in Constitutional AI (https://arxiv.org/pdf/2212.08073) that also uses various prompting of verifier LLMs to select the best answers. There are some differences (the direct usage here vs. RL training there, the averaging of the verifiers answers here instead of random selection of verifiers there). But such approaches should be discussed as similar to yours.\n\nThe results lack the comparison with an oracle baseline that would select the real best of n (pass@n). It would give a better idea of the ceiling performance of such BoN approaches.\nThere could be simpler performance evaluation of the approach by comparing the rates at which one of the correct answers (when it exists) among the n generated were selected by different methods. This would help decorrelate the base model performance and the performance of the proposed approach.\n\nThere is no comparison with the efficiency of compute usage with other test time compute approaches. This method is probably more compute intensive compared to reflection models but it can be parallelized which could lead to a better latency. Is that the case?\n\nThe method seem to rely heavily on hand-crafted domain specific verifier prompts and the figure 5 seems to suggest that there is a large variance with the verifier selection. This limits the generality and scalability of the approach.\n\nPractical usage of the approach seem to be as a selection mechanism for RL training but this is left as future work.", "questions_to_authors": "l.254 it is said that \"all possible combinations of m verifiers drown from Md\" are used. Does this means that you produced 16 generations overall, ran all verifiers and mixed the verifier results in all possible ways on the same generations and the same verifiers output? This saves compute (or API calls) but hides some part of the variability of the results and would help understand what the percentile ranges from figure 5.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies, in a generate-verify style inference method, the relationship between the number and type of automatic verifiers and overall performance. Among other results, authors find that this method allows for smaller verifiers. Overall, I think this is a nice contribution to COLM that can be broadly summarized as pushing in a direction of research on scaling behavior influenced by verifier design/quality. Authors include additional experimental results in the rebuttal following reviewer suggestions that should be included in any camera-ready version of the work.
Decision: Accept