Weaver: Shrinking the Generation-Verification Gap by Scaling Compute for Verification
Authors: Jon Saad-Falcon
E. Kelly Buchanan
Mayee F Chen
Tzu-Heng Huang
Brendan McLaughlin
Tanvir Bhathal
Shang Zhu
Ben Athiwaratkun
Frederic Sala
Scott Linderman
Azalia Mirhoseini
Christopher Re
Venue: NeurIPS 2025 poster ()
Keywords: test-time compute
repeated sampling
weak supervision
weak verification
Primary area: deep_learning
TL;DR: We introduce Weaver, a framework that combines multiple weak verifiers to effectively select responses in repeated sampling, achieving frontier model accuracy without supervised fine-tuning, while reducing verification costs by 99.97%.

Abstract: Verifiers can improve language model (LM) capabilities by providing feedback or selecting the best response from a pool of generated candidates. Currently, high-quality verifiers are either unscalable (e.g., humans) or limited in utility (e.g., tools like Lean for formal proofs). While LM judges and reward models have become broadly useful as general-purpose verifiers, a significant performance gap remains between them and oracle verifiers. To help close this gap, we introduce Weaver, a framework for designing a strong verifier by combining multiple weak, imperfect verifiers. First we find that weighted ensembles of verifiers, which typically require learning from labeled data, significantly outperform unweighted combinations due to differences in the verifiers. To reduce the dependency on labeled data, Weaver leverages weak supervision to estimate each verifier’s accuracy and combines their outputs into a unified score that better reflects true response quality. However, directly applying weak supervision algorithms poses several challenges, including inconsistent verifier output formats and handling low-quality verifiers. Weaver addresses these challenges by using dataset statistics to normalize outputs and filter specific verifiers. We study the effectiveness of Weaver in repeated sampling settings, where a model generates multiple candidate responses at test time and a verifier is used to select the correct one. Our evaluations demonstrate that Weaver significantly improves the pass@1 performance across several reasoning and math tasks, achieving o3-mini level accuracy with Llama 3.3 70B Instruct (a much cheaper non-reasoning model) as the generator, and an ensemble of smaller judge and reward models as the verifiers (86.2% average). This gain mirrors the jump achieved between GPT-4o and o3-mini (69.0% vs. 86.7%), which required extensive finetuning and post-training interventions. To make Weaver more efficient, we train a compact 400M cross-encoder using Weaver's combined output scores. This distilled model retains 98.7% of Weaver's full accuracy while reducing verification compute by up to 99.97%.

WEAVER: Shrinking the Generation-Verificattio Gap with Weak Verifiers
Jon Saad-Falcon' E. Kelly Buchanan"
Mayee F. Chen
Tzu-Heng Huang Brendan McLaughlin'
Tanvir Bhathal
Shang Zhu Ben Athiwaratkun Frederie Sala
Scott Linderman' 1 Azalia Mirhoseini'
Christopher Ré
Stanford University
Together AI Equal contribution
SUniversity of Wisconsin-Madison
[jonsaadfalcon, kelly buchanan, mfchen) @staneord.de...m
Abstract
Verifiers improve language model (LM) capabilities by providing feedback lecting the best response from pool of generated candidates. Currently, high-quality verifiers are either unscalable (e.g.. humans) or limited in utility (c.g.. tools like Lean for formal proofs). While LM judges and reward models have become broadly useful as general-purposs verifiers, a significant performance gap remains between them and oracle verifiers. To help close this gap, we introduce WEAVER, framework for designing a strong verifier by combining multiple weak, imperfect verifiers. First we find that weighted ensembles of verifiers, which typically require learning from labeled data, significantly outperform unweighted combinations due to differences in the verifiers. To reduce the dependency on labeled data, WEAVER leverages weak supervision to estimate each verifier's accuracy and combines their outputs into a unified score that better reflects true response quality. However, directly applying weak supervision algorithms poses several challenges, including inconsistent ver- ifier output formats and handling low fow-quality verifiers. WEAVER addresses these challenges by using dataset statistics to normalize outputs and filter specific verifiers. We study the effectiveness of WEAVER in repeated sampling settings, where model generates multiple candidate responses at test time and a verifier is used to select the correct Our evaluations demonstrate that WEAVER significantly improves the pass@1 performance across several reasoning and math tasks, achieving 03-mini- level accuracy with Llama 3.3 70B Instruct much cheaper non-reasoning model) as the generator, and an ensemble smaller judge and reward models as the verifiers (86.2% average). This gain mirrors the jump achieved between GPT-4o and o3- 3-min (69.0% vs. 86.7%), which required extensive finetuning and post-tras post-training interven- tions. To make WEAVER more efficient, we train a compact 400M cross-encoder using WEAVER'S 's combined output scores. This distilled model retains 98. % of WEAVER'S full accuracy while reducing verification compute by up to 99.97%.
Introduction
core challenge in deploying language models (LMs) is verification: determining the quality or correctness of model's response. This problem arises across various components of the LM pipeline,
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

including dataset curation, model alignment, and informed inference-tine decision beaiiign-makingg.. A verifier a function that scores responses. When combined with repeated sampling-generatinng multiple candidate responses from LM-a strong verifier can be used to select correct candidate response, significantly enhancing model capability on tasks such as math, code, and reasoning [5, 55, 70]. However, without a perfect verifier, a generation-verifcatiin gap emerges [72]: a LM can generate a correct response, but we fail to identify it.
In some settings, we have access to oracle verifiers that can perfectly identify correct responses, A prominent example Lean, a formal theorem prover that can be used for problems such MiniF2F [94]. However, this is often a limited setup, as not all mathematical proofs can be processed by Lean. In contrast, more generally applicable verifiers such as LMs prompted as judges [13] and reward models [39, 45, 68] can be broadly applied to tasks like scientific reasoning, coding, and instruction -orllwiin-foloowing [32, 40, 62]. However, these weak verifters produce noisy, inconsistent scores, often exhibit poor calibration, and suffer from high false positive rates [73]. We ask: to what extent can we leverage weak verifiers to improve accuracy in the repeated sampling regime?
We explore scaling verification, specifically how to combine multiple weak verifiers to improve response selection for repeated sampling. As new pre-trained models become available, the pool of weak verifiers continues to expand and offer diverse, complementary sources of signal that could improve response selection they can be aggregated effectively. Recent work has explored scaling verification through techniques such as verfiieciinnntion or averaging LM judge scores [9. 41, 93]. However, it remains unclear how to effectively combine multiple weak verifiers. We observe three key challenges towards ensembling weak verifiers:
1. Naively aggregating weak verifiers is insufficient for reliable verification. Weak verifiers such LM-based judges or reward models produce noisy, biased, and poorly calibrated scores, leading to inconsistent performance. [13, 39, 73]. While using naive unweighted average of verifier scores straightforward, implicitly assumes uniform verifier quality, causing low-quality verifiers to dominate and degrade the overall accuracy [21, 76, 88]. Moreover, while previous work has hypothesized that more sophisticated weighted ensembles should perform better, this claim has not been studied [41].
2. Effective ensembling with limited labeled data is challenging. More sophisticated ensembling techniques typically learn verifier weights from labeled data, but such data is expensive and difficult to obtain. Weak Supervision (WS), a family of statistical techniques developed for data labeling, offers a potential solution through algorithms that aggregate multiple weak signals- such as crowd- worker annotations and
heuristics- -hille while only requiring a small amount of labeled data [23,57, 57,58,, 58). In traditional WS, practitioners can design and shape each weak signal ensure suf- ficient quality (i.e., iteratively tweaking program-bar bbsed heuristics), and guarantees of WS hinge on a baseline level of quality. Our weak signals, however, are fixed tre-trained language model verifiers, which have wildly varying accuracy especially when applied to can emit incompatible outputs (logits, binary scores, Likert scores) [39] that we cannot easily tweak. Due to these conditions, WS algorithms may not perform well when directly applied to verification.
tasks and
Verification is expensive to deploy at inference. Verification can dominate inference- time costs [45, 68], since each verifier must process both the problem and its candidate response(s) [42], often evaluating intermediate steps [42] and multiple solution paths [70]. In fact, achieving modest gains over unverified generation can require 10x to 128x the inference compute per query [9, 41, 69, 93].
In this work, we introduce WEAVER, a framework for aggregating weak verifiers without supervised finetuning on ground truth labels (Figure 1). First, we demonstrate that we have access to large corpus of labeled training data (e.g.. 50,000 query-response pairs), we can learn weighted ensembles that can outperform naive averaging by up to 1 1..% points. This is because of wide variability in verifier accuracy. However, in many wooldworld scenarios, data. Second, to reduce the dependency on labeled data, we adapt Weak Supervision to the verification setting by addressing challenges around inconsistent outputs and low-accuracy verifiers. WEAVER filters out uninformative verifiers, normalizes verifier scores, and builds a latent variable model over these scores and the unknown true labels to estimate the verifier accuracies to be used as weights [26, 58).
do not have access to such quantities of labeled
Empirically, WEAVER improves over unweighted averaging of verifier scores by 17.1% 1% and repeated sampling with majority voting by 13.5% (Table 1). WEAVER allows us to improve over Pass 1 by 17.9% for 8B models and 14.5% for 70B models across reasoning and mathematics tasks (Tables and 18). This mirrors the performance jump from GPT-4o to 03-mini (73.9% vs. 88.2%) but without
2

---

Instruction
LM Generation
GPQA Diamond
GPQA Diamond*
90
Selection Accuracy (59
75
Judges
P True
False
0.0
0.5
Weak Supervision Estimated Opgimal Weighting of Verifiers
Success Rate
80
.3.min
65
70
59.9% Cempute
0.7
60
98 of Accuracy Galns Preserved
60
-5
50
LM Ceneration Verdict
Plat Janak
10"
10'
10t
Accept Raject
Number of Repeated Generations
Verification Computer Budget (ExaFlops)
Pwogg (Gnccee
Hacority Vaiing Hull-Aqert Vatlication
Weower Datilled t0 481
Hil (70.88
Weaver Methodology
Weever (70 oz;
Set-ieitication (70.05)
1a. Frneertee
Gel-bbeiication
Majoriyy voting 10 del
Paroto Region
Figure 1: WEAVER Framework: We propose WEAVER, a framework combining multiple weak verifiers to effectively scale repeated sampling without parameter finetuning on ground truth labels (left). W WEAVER significantly outperforms majority voting and shrinks model's generaation-vrriiccn gup by 14.5%, on average, for GPQA Diamond and other datasets (Table 1) (middle). By distilling WEAVER from an ensemble of 70B verifiers to single 400M cross-encoder, we can preserve 98.2% of the accuracy gains of WEAVER while reducing inference compute cost by 99.97% (right).
parameter tuning post-training steps. We also study how WEAVER scales along different axes of test-time compute: generation, verifiers, model size, and inference budget (Section 5.2). We find that even we increase the number of generations, many standard verification baselines (e.g. majority voting) quickly plateau (Figure 3). Naive ensembling saturates more slowly, but its gains are limited by high compute costs, and sensitivity to the choice and number of verifiers. We show that WEAVER outperforms alternative verification approaches by 13.5% while improving over Pass@1 by 18.3% as we scale from 2 to 2 generations (Figure 3).
Finally, as an extension of WEAVER, we train a 400M parameter cross-encoder verifier using WEAVER'S selected responses mitigate the compute costs of calling multiple weak verifiers each response. We demonstrate that using distilled WEAVER cross-encoder as a verifier retains 98.7% of the accuracy gains from the learned verifier ensemble while reducing compute costs by three orders of magnitude- - saving 99.97% inference FLOPS while still capturing an effective verification strategy (Section 6). Overall, our findings highlight that more reliable, scalable verification is possible even in the absence of ground-truth labels peeels-pavinggge the way for improved data filtering, model alignment, and inference-time decision-make maaiin..
Related Work
LM Judges and Reward Models: Both LM judges and reward models are promising approaches for evaluating language model outputs, but their high false positive rates limit their reliability [73]. LM judges can evaluate outputs without additional training [24, 47, 80), using approaches from simple prompting to chain-ol -iiiin-of-hhe of-thought reasoning [47] and specialized fine-tuning [64, 75]. However, they face poor generalization across contexts [22, 59, 64] and systematic biases in position and self ceff-preference [8, 37, 54, 96]. Similarly, while reward models have become central to model alignment [4, 14, 43], they struggle with noisy training signals from low inter-annotator agreement [3, 19, 53, 78] and learned biases favoring attributes like response length [18, 38, 67]. Recent work has improved individual verifier reliability through better data collection and chain-of-ther of-thoughtt reasoning [83, 92], yet fundamental challenges persist [7, 20J. WEAVER advances beyond these approaches by combining multiple verification signals with adaptive weighting, thus leveraging the complementary strengths of weak verifiers while suppressing noise and reducing false positives.
Weak Supervision: WEAVER builds upon statistical techniques from weak supervision, which emerged as a framework for programmatically generating training labels by aggregating multiple weak sources [56, 58). While initial work focused on classification tasks [23, 57], recent advances
have expanded to handle multi-task settings [66] and structured prediction [77]. WEAVER applies this framework to answer verification, treating binary imperfect verification signals (e.g. reward models and LM judges) weak supervision voters that classify candidate solutions as correct incorrect.
3

---

This novel application combines predictions by converting these diverse signals into binary verdicts, enabling WEAVER to learn better verification strategies from weak but complementary verifiers.
Verification as another compute axis and aggregation: Recent work has explored verification as new scaling axis [9, 41,1,44 46, 69, 73, 93]. However this work limits their analysis one verifier, and instead scale how many times to verify [93]. Approaches that do leverage multiple verifiers often rely on substantial amounts of labeled data for aggregation or creating specialized verifiers [36, 41J. With WEAVER, we show that it is possible to combine verifiers without ground truth labels, even when they are not specialized. Other work has focused on combining multiple verifiers for post-training the base model using RLHF [21, 81, 84]. We note that this is direction of future work (Section 7).
3 Preliminaries
First, we define the problem of how select among repeated samples. We then define verifiers and key evaluation metrics, including the generation verification gap.
cnneration-verfiiaatoo
Problem Definition Let q Q be input text query, and let R~ be a corresponding
response sampled from language model M with non-zero temperature. For a given query-response 10 such that y(g.r) ) is the correctness label of for q. We are given an unlabeled test dataset Diest = { ((Krrs) r where r, = (ray K consists of K repeatedly sampled responses from M for each qı. We do not have access to true labels yij :- y(garra)) for any i.j.
pair (g.r), we define y x R
For each (qt, rr)) E Diest, our goal is to select a correct response j* E [K] that satisfies y/j- 1. We can broadly describe this selection rule using a scoring function Q x R - R, namely j ==arraaaaaa (q..T)))
Using veritiers A verifier, either reward model or an LM prompted a judge, can be expressed as scoring function on query-response pairs V: Q x R -> IR For reward models, the verifier score continuous, while for LM judges, the verifier score is typically discrete (for our setup, we use [0.1] and {0.1}, respectively). We assume that we have access to multiple verifiers V= We apply each of the m verifiers to each (g.rr,)). ,. for total of nm scores on Diest with Sijk: : Un (gyr We aim to use V to construct good verification strategy f.
Evaluation metries The Pass @ K metric is the probability that there exists correct response among K generated responses: Pass @ K = - 1(3j e [K] = 1). This metric is independent of the verification strategy, and depends on the choice of M, K, and the task dataset. The success rate of a verification strategy is E Mej.. where j = argmax, rgg. Success rate is dependent on the verification strategy and bounded by Pass @ K, and equality obtained with oracle verification (i.e., f f* can always select correct j as long as exists).
We define the generation-verrjicatoo gap as Pass@K K Success Rate, A large positive gap indicates that although correct answers are generated, the verification strategy fails to select them consistently. We aim to close this gap and will use it to evaluate verification strategies.
4 WEAVER: A Framework for Weak Verifier Aggregation
In Section 4. we demonstrate that naively averaging multiple verifier scores to select responses significantly underperforms weighted ensembles; however, common methods for computing weights require labeled data [65, 89]. We introduce WEAVER (Section 4.2), method for weighted aggregation of verifier scores with minimal data that draws inspiration from Weak Supervision.
4.1 How to aggregate multiple verifiers: weighted vs unweighted ensembles
A straightforward approach for using multiple verifiers is naive ensemble-seleeeee the response with the highest average verifier score: f(g,riy) : >>= Sijk- This approach [41] does not consider the relative accuracy of verifiers. However, observed that there is significant variation in the success rates of individual verifiers spanning a range of up to 37.5%-.uggeeeee -ooggsstgg that naive ensembles could be suboptimal (Table 14).
An alternative to use weighted ensemble. One approach to use labeled dataset to identify and use the perpefforrmng verifier, effectively assigning weight of 0 to discarded verifiers. Other strategies

---

include using Logistic Regression or Naive Bayes classifier, where the scoring function Fig. the probability Pr( = -1 (..,.... which are fit using labeled data and can be either modeled as logistic function or factorized using Bayes' rule and independence assumptions, respectively.
In Figure 2, compare naive ensemble with weighted ensembles for several tasks, using Llama 3.3 70B Instruct to generate responses and using collection of 33 7B-72B reward models and LM judges verifiers. We that using weighted ensemble can achieve up to 1 1.2 points higher success rate than the naive ensemble. However, all weighted ensembles shown are "oracle" methods: they are computed using Viy for all E [n].j- j [K]. although in practice these labels are unknown for Diest In fact, when we instead use 0.01n labeled samples, accuracy drops by 20.1% 1 on average (Table 15).
This raises the question of how best construct weighted ensembles with limited labeled data.
Improvement over
Naive Ensemble (%)
Oracle Unweighted Ensembles
Top-1 Verifiers
Top-5 Veriflers
Top-10 Verifiers
Supervised Weighted Ensembles
Naive Bayes
Logistic Regression
MATH500
GPQA Diamond
MMLU
MMLU Pro
Figure 2: Weighted Verifier Ensembles Outperform Naive Verifier Ensemble: By using oracle data to keep the best verifiers (i.e. top- K verifier ensembles) or learn aggregation weights for verifiers (i.c. supervised weighted ensembles), we can improve beyond naive combinations the verifiers available by 3.6% and 7.8%, average, respectively.
4.2 WEAVER: weighted ensembling of verifier scores with minimal labeled data
We first describe the WS method we use WEAVER to construct weighted ensemble over binary verifier scores. Because verifiers often produce scores in inconsistent formats and exhibit low accuracies accuraciss-ccalllngges not typically encountered in traditional WS WS-we introduce a binarization and verifier discarding strategy in Appendix B.2 to discard low-quality verifiers and ensure that only sufficiently reliable binary scores are used as input to the WS method.
4.2.1 Weak Supervision Algorithm
In Weak Supervision, the input is unlabeled dataset, where each entry has multiple binary "votes" on the true label. Applied to our setting, each entry is a query-response pair, forming a dataset of size nK, and verifier scores Sizk binarized into votes &iyk E {0,1) for all i.j.k. Our goal is to predict the probability that a response is correct Pr(y, = 1|sig) Simm) ) for all i.j.
WS model We can view all Vij samples of unknown random variable Y and each &yyk across i as samples of a random variable S.. WS then defines latent variable graphical model over the random binary vector (Y,SI....Sm.. where Y is latent while Sq....,,, are observable. While existing WS methods assume various models, one common assumption is that S; S, for each S,,Sy. That is, S, and S, conditionally independent given Y; intuitively, each verifier captures independent aspects of the correctness of the response. Under this assumption, we can write the posterior as the following, for some given binary verifier scores {81 5m))
m P
Pr(Y = 1|S1 = S1...., =
(I)
Pr(Si181,...55..
The weighted ensemble score for each query-response pair can be written in terms of: 1) Pr(S, 5m = km), which we compute the law of total probability as Ey Pr(Y=
> Pr(S, 3, =1y) using the conditional independence assumption; 2) Pr(Y =), which can be estimated from Ddwy and Pr(S; x = 1). or equivalently Pr(S;=1p 1|y== = which is the verifier's "accuracy parameter" tths cannot computed directly since we do not have access to Y. Next, discuss how to estimate accuracy parameters, Pr(S, 1|Y 1), without labels.
5

---

WS parameter estimation We outline parameter estimation technique first introduced in [56].
- Pr(.. -1 ( Sateo------- Pr -
Due to the assumption that S, S,LSj the following equation equatiion holds:
- -))r(( -0)
-Pr(S 1 =
(2)
Note that Sr) can computed from the known verifier scores, and Pr(Y = 1) is estimated from Ddey, Then, equation 2 a quadratic equation over the accuracy parameters, We can write this equation for every pair S, and for every pair of values {0,1}* ]2 they can take. Furthermore, we can
write another type of equation over the accuracy parameters:
Pr(S, = 105,=1)-Prr(5 Pr.S. Y 1) Pr(s = -))r-))
This is consistency property that holds regardless of the conditional independence assumption, and we can write this equation for each of the m S,'s. ss Because we know that the accuracy parameters should follow equations 2 and 3, we can construct objective function that aims to minimize the e
difference between the left and right hand sides of these equations. We write this efficiently in
matrix notation. Let P E IR 2x2 be a diagonal matrix with diagonal [Pr(Y= = 0) Pr(Y = 1)]- Define ju E 2m2x22 to be the matrix of accuracy parameters, and define O E ER 222mm22 2m to be a matrix over the joint probabilities of pairs of Sy; more formally:
A2i- 122-1:24
Pr(S, = 0015==0--0000 Pe(S, =o(s,=0
Px(S: 1 o) Pt(S, Pr(S,-1[Y-1) Oz 021-1226,11-:22 1:21
,
Pr(S 00)
ViE [m] 1)
Oz 1:1-1-212j-::333
Pi 0 =0 Ps(S; -=(S=005;=11
Pr(s;=1,5; 00 Pr(S,=1.S,=1)
[
Let off-diag denote the elements of a matrix that lie outside its 2 block diagonal. Then, to estimate j that satisfies both equations 2 and 3, we have the following objective:
minimize,, O (pPP)")ontaaa Jirr || [ 712
We optimize 5 using gradient descent to estimate the verifier accuracy parameters. These estimates then used in Eq. (I) to select the response with the highest estimated posterior. We provide full derivation in Sec. B.1.
full
5 Results
In section 5.1, we provide empirical results on WEAVER's performance compared to other approaches for selecting responses in repeated sampling. section 5.2, we study how WEAVER'S 's performance scales along several axes: the number of responses, model size, verifier counts, and inference compute.
Datasets, Verifiers, and Baselines Our reward models range in size from 8B to 72B, are all open- source, and are obtained from RewardBench [39], popular evaluation tool for reward models. We prompt open-source language models from Chatbot Arena [13] to serve as judges. Unless specified, we use Llama 3.3 70B Instruct to generate responses and use all 33 reward models and judges. We evaluate on
MATHHO00 MATH500, GPQA Diamond, MMLU College, and MMLU Pro. See Sec. C. for more details.
We compare WEAVER against verifier-free free baselines as well as standard verification strategies. First Sample, also known as Pass only uses the first response and does not scale test-time compute or verification. Majority Voting involves repeated sampling but not verification, picking the most common final answer from the responses [5, 10, 70]. We compare against the highest scoring reward model and a naive ensemble of the top-10 10 reward models on RewardBench. We also evaluate two recently proposed methods that scale verification but do not use multiple verifier models weighted ensembles: Self- Verification [93] and Multi-Agent Verification [41]. Lastly, we report the oracle Pass@ K rate, which establishes an upper bound for the success rate of these verification strategies.
5.1 WEAVER Shrinks the Gap with Frontier LMs
In Table 1, we evaluate WEAVER along with baseline verification methods, the first sample performance of frontier LMs, and the Pass@ 100 metric. We use LlaMA 3.3 70B Instruct to generate K = 100 responses per query. We find that WEAVER'S weighted ensembling of multiple verifiers allows us to majoriite outperform majority vote by 14% and come within 5.7% of the Pass@ 100 oracle metric. Furthermore,
respoor
WEAVER rivals the performance of frontier reasoning models coming within 0.5% of OpenAl's minm [52]- (venn even though we use a non-reasoning model for generation.

---

Table WEAVER Outperforms Baseline Verification Methods and Shrinks Gap with Frontier LMs.
Datasets
MATH 500
GPQA Diamond
MMLU College
MMLU
Methodology
Generations (K)
Pro
Average
First Sample
1
78.0%
42.9%
82.6%
69.9%
68.4%
Majority Voting
100
83.0%
74.4%
Towdins
Hi Highest Scoring RM on RewardRench [39
100
78.2%
49.7%
86.0%
77.0%
72.7%
Naive Ensemblle Tup-
RewardBench [39]
Self Verification [93]
100
78.1%
43.1%
82.0%
69.5%
66.9%
Multi-Agent Verification [41]
100
81.3%
47.8%
84.1%
72.6%
71.6%
WEAVER
100
66.4%
94.9%
90.2%
86.2%
Fionter Appreactis
GPT-40 [51]
1
77.4%
35.
87.1%
75.4%
69.0%
Claude 3.7 Sonnel [2]
69.21%
48.0%
86.1%
78.1%
70.4%
Llama Maverick [48]
87.6%
68.9%
91.1%
81.0%
82.2%
03-mini [52]
94.4%
74.0%
92.2%
86.0%
86.7%
Oracle Verification (Pass
100
98.6%
81 81.0%
96.0%K
92.0%%
91.9%
5.2 WEAVER Improves Compute-Accuracy Trade-Off for Scaling
By proposing to combine multiple weak verifiers instead of one, we introduce yet another axis for test-time scaling. this section, we study how well scaling verification with WEAVER interacts with common previously studied axes for verification, summarized in Table 2.
Table Scaling Dimensions for Generation and Verification Models
Scaling Dimension
Base Model
Verifier Type
Visuals
Sample Count: More
Temperature-based sampling
Majority Vote, Weak Verifier,
Figure 3
Generations
Top-K, WEAVER
Model Size: Larger Models
Llama 70B
RM-8B -> RM-70B
Table 3 Figure
Verifier Count: More Models
Llama 8B/70B
RMs and LM Judges
Inference Compute: More
Temp- based sampling
Weak Verifiers WEAVER
Figure 5
FLOPs for Gea./Ver.
(1) Scaling Candidate Generations: we study the performance of verification methods as we increase the number of repeated samples in Fig. 3. Based on prior work [4, 12], the number of responses increases, we are more likely to see a correct response (i.e. Pass K increases), and hence more likely select correct response given a good verification strategy. However, differences in verification translate into different scaling rates. We evaluate the performance of WEAVER and baselines for K 2 20 comparing to mi-m and Pass K well. Across all tasks, WEAVER yields the most substantial gains when scaling the number of generations. WEAVER consistently narrows the generation- verrificaiinn gap with the oracle upper bound (Pass @ K) while alternative verification strategies plateau after few generations. The effect is particularly pronounced on difficult tasks like GPQA. We detail the scaling trends observed in Fig. 3 in C.3.
(2) Scaling Model Sizes: In Table 3, we study how WEAVER applied on smaller models (both verifiers and for generating responses) can allow us to match the performance larger models, enabling weak-to-strong verification. We consider an 8B responses along with 8B verifiers-ant and compare this 70B setting (LlaMA 3.3 70B Instruct, 8B-72B verifiers) as well as 03-mini. We see that WEAVER applied at the 8B scale comes within 2.2% of the majority vote baseline the 70B scale, and WEAVER at 70B comes within 0.5% o3-mini, demonstrating weak-to-strong verification phenomenon.
LlaMA 3.1 8B to generate
Two axes for scaling verification are (1) the number of verifiers used or (2) the number of scores sampled from each verifier. Fig. 4 shows how the performance changes as we ensemble more verifiers (up to 15) using naive ensembling and WEAVER. We can see that aggregating verifiers continuously improves performance over using the top- verifier by up to 8.5%. However, these gains diminish as continue to add models to the ensemble. These results reflect the classic ensemble bias-varian tradeoff: initial gains arise from variance reduction as independent verifier errors averaged out, but performance plateaus as additional verifiers contribute redundant
(3) Scaling Verifier Count:
7

---

GPQA Diamond
MATH 500
MMLU Pro
90
95
Success Rate (%)
80
95
333mm
90
85
90
+15.8%
+11.2%
Gain
Gain
+27.8%
Gain
75
30-
80
Sample
70
pitt Sample
40
2:
2
21
2
75
20
2
22
21
2
2
65
2
27
2
26
Number of Repeated Generations
Repeated Generations
Repeated Generations
PassgK (Dracla)
Weaver
Naive Ensemble
Majority Vating
Mult--Agett Varification
Self-Verificaton
Figure 3: Scaling Generations Boosts Performance with WEAVER: The generation- verification gap shrinks when increasing and leveraging WEAVER, outperforming alternative verification methods by an average 18.3%.
Table 3: WEAVER Reduces Gap between Model Classes: 8B and 70B, 70B and Frontier LM
Datasets
Generator Model
Verifier Model
Aggregation Strategy
Average
GPQA Diamond
MMLU College
MMLU
MATH
Pro
N/A 8B and below
Majority Vote WEAVER
69.0%
30.5%
72.7%
56.4%
57.2%
Llama 3.1 8B Instruct
80.0%
47.1%
85.7%
67.2%
70.0%
A WEAVER
+11.0%
+16.6%
+13.0%
+10.2%
+12.8%
N/A 72B and below
Majority Vote WEAVER
83.0%
47.4 47.4%
84.1%
74,4%
72.2%
Llama 3.3 Instruct
93.4%
66.4%
94.9%
90.2%
86.2%
W. WEAVER
+10.4%
+9.0%
+10.8%
+15.8%
+ +14.0%
03-mini
N/A
First Sample
94.4%
74.0%
92.2%
86.0%
86.7%
information due to correlated biases systematically difficult examples. [1]. We also include results on scaling the number of scores per verifier in Appendix C.5. We find that scaling the number of verifiers yields better performance than sampling multiple scores from the same verifier (i.e., via prompt tuning or temperature variation). However, this method is complementary and could be used conjunction with scaling the verifier count.
GPQA Diamond
MATH 500
MMLU Pro
75
100
95
Success Rate (96)
20
Accuracy Gains
+8.6% Accuracy Gains
+7.4% Accuracy
Ensomble
Ensemble
Ensemble
55
la
85
80
Figure 4: WEAVER Benefits from Adding Top-K Verifiers.
(4) Scaling Time Compute: We study how performance scales in the total compute used for both verification and repeated generations. Figure 5 shows the relationship between inference- time compute and success rate for different generation-verte verification systems. For each method, we scale the number of generations exponentially from to 100 and plot the required inference compute for generation and verification together versus the success rate. Note that Fig. differs from Fig. since Majority Voting requires () verification inference calls while WEAVER requires 30+ calls for the weak verifiers. We find that WEAVER achieves the highest maximum success rate; notably, majority voting plateaus at around 2 to 2 ExaFLOPs per query while WEAVER continues scaling until 512

---

ExaFLOPs. However, the additional compute required for WEAVER can be prohibitive. We explore how to reduce this computational burden while retaining WEAVER'S performance in the next section.
GPQA Diamond
MATH 500
MMLU Pro
97.3%
65
97. 37%% Compute Saved
92.5
97.3% Compute Saved
Success Rate (%)
80.0
60
Accuracy Gains
Accuracy
55-
Galins
85.0
80
Gains
50-
62.5
75-
80.0
11.5
$0
2
2'
2' 24
21
20
75.0
2
2'
2'
2
21
2'
27
2
2'
24
2
21
Inference Compute per Query (ExaFLOPs)
Inference Compute per Query (ExaFLOPs)
Inference Compute per Query (ExaFLOPs)
Weaver Distilled
Weaver
Naive Ensemble
Majority Voting
Figure 5: WEAVER Improves the Accuracy-Compute Performance Trade-Off'.
WEAVER Distillation: Improving Verification Efficiency at Inference
We explore distillation strategies for fine-tuning smaller LM as task-specific verifier. In particular, we train cross- encoders;: the input is concatenated query-response pair, while the output is WEAVER's pseudolabel generated from Weak Supervision, namely Pr(yaj (..5))......immm (see Section 4). Cross- encoders are used for a variety of NLP tasks requiring comparison between two textual inputs, including search result reranking [50], question-answer matching [25], semantic textual similarity assessment [60], natural language inference [16], and response quality evaluation in dialogue systems [31]. For the model, we selected ModemBERT--Lan (396M) [85]. For more details, please see Appendix C.6.
Figure 6, shows the performance of WEAVER on the Llama-70B generations against the cross- encoder on GPQA Diamond. Across tasks, we find that the distilled cross encsder is able to cap- ture 98.2% of the performance of WEAVER. Fur- thermore, while it would cost 35.35 exaFLOOSs to
GPQA Diamond
Selection Accuracy (%)
70:
99.97% Compute Saved
65
98.2% of Gains Preserved
run all the verifiers needed for preparing and eval-
uating WEAVER. while running 400M cross- encoder costs 1.01 exaFLOPs for evaluating 100
55
samples, this more than three orders of magni- tude in compute cost reduction, saving 99.97%
50
of the FLOPs originally required for running the 70B verifiers. We also outperform majority voting by 23.2% while only incurring 0.57% increased inference cost over only generating the responses. We see similar results for additional datasets in Figure 14 (Sec. C.6).
40
10a
10'
104
Verification Compute Budget (ExaFLOPs)
Weaver: Distilled (0.4B)
MAV (70.0B)
Weaver (70.0B)
Self-Verification (70.08)
Majority Voting (0.08)
Pareto Region
These results suggest that, through distillation, we can capture the combined strengths of the weak verifiers used for WEAVER, and deploy gen- eralizable and lightweight coos-enccderss that use only a fraction of the parameters used for gen- eration. This reduces our hardware constraints considerably; rather than utilizing an 8-GPU node per 70B verifier (i.e. Nvidia H200s with 80B memory), only require single A100 GPU with 32GB of memory for our cross-encoder. Our economical approach has the added benefit of leaving the original generation model frozen (i.e. without any parameter tuning), allowing preserve the original model's distribution at inference [87, 91].
Figure 6: Distilling WEAVER into a 400M Cross- Encoder Almost Entirely Captures the Perfor- mance of WEAVER, Yielding 99.97% % Compute Savings. *We train/evaluate 80:20 split.

---

7 Conclusion
WEAVER offers flexible and scalable framework for aggregating weak verifiers, achieving strong performance across diverse reasoning tasks while remaining computationally efficient. Our results show that combining many weak verifier can further close the generation verification gap with fraction of the compute. Looking forward, several directions stand to extend WEAVER 's impact. First, designing specialized verifiers can further enhance task-speeific accuracy. Second, WEAVER can be leveraged for RLHF for better supervision signals. Multimodal extensions of WEAVER represent another exciting frontier. Finally, the compute efficiency of WEAVER opens the door to improved on-device verification.
Acknowledgements
We thank Simran Arora, Daniel Biderman, Bradley Brown, Owen Dugan, Ryan Ehrlich, Neel Guha, Simon Guo, Jordan Juravsky, Jerry Liu, Avanika Narayan, Benjamin Spector, and Benjamin Viggiano for their constructive feedback during the composition of the paper. We would also like to thank our collaborators at the Stanford Artificial Intelligence Laboratory (SAIL) and TogetherAI.
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize); NSF under Nos. CCF2247015 (Haddware-Aware) CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML): US DEVCOM ARL under Nos. W911NF-23-244 84 (Long-context) and W911NFF21122-025 (Interactive Human-Al Teaming); ONR under No. 0000142312633 (Deep Signal Processing); Stanford HAI under No. 247183; Google DeepMind: Google Research; Google Cloud; NXP; Xilinx; LETI-CE; Intel; IBM; Microsoft; NEC: Toshiba; TSMC; ARM; Hitachi;
BASF; Accenture; Ericsson; Qualcomm; Analog Devices; Salesforce; Total; the HAI-GCP Cloud Credits for Research program; the Stanford Data Science Initiative (SDSI); members of the Stanford DAWN project: Meta, Google, and VMWare; and members of the Stanford SEAMS project: IBM and Felicis. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions recommendations expressed this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
References
[1] Taiga Abe, E. Kelly Buchanan, Geoff Pleiss, and John Patrick Cunningham. Pathologies of predictive diversity in deep ensembles. Transactions on Machine Learning Research, 2024. Featured Certification.
[2] Anthropic. Claude 3.7 sonnet and claude code, February 2025. Announcement blog post, 5 min read.
[3] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A General Language Assistant as Laboratory for Alignment. ArXiy Preprint arXii:212X2
2021.
[4] Ralph Allan Bradley and Milton E Terry Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4):324-345, 1952.
[5] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024.
[6] Zhe Cao, Tao Qin, Tie- Yan Liu, Ming- Tsai, and Hang Li. Learning rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pages 129 100,136,2220 2007.
[7] Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMss 2024.
[8] Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the Judge? A Study Judgement Bias. ArXiv Preprint 10666 2024.
10

---

[9] Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, and Serean Ô Ank. Sets: Leveraging self verificaiin and self-correction for improved test-time scaling. arXiv preprint arXiv:2501. 2025.
[10] Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, lon Stoica, Matei Zaharia, and James Zou, Are more Ilm calls all you need? towards scaling laws of compound inference systems. arXiv preprint ar 2r03 024119, 2024.
[11] Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, lon Stoica, Matei Zaharia, and James Zou. Are more Ilm calls all you need? towards scaling laws of compound inference systems, 2024.
[12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Vosse William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
[13] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024.
[14] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.
Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems, 30, 2017.
[15] Ganqu Cui, Lifan Yuan, Zefan Wang. Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint 2510:2502.01455 2025.
[16] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
[17] Nicolai Dorka. Quantile regression for distributional reward models in rlhf. arXiv preprint htt::2409.101644 2024.
[18] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. ArXiv Preprint 24X1v:2404.04475, 2024.
[19] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. Advances in Neural Information Processing Systems, 36, 2024.
[20] Jacob Eisenstein, Jonathan Berant, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexan- der Nicholas D' Amour, Krishnamurthy Dj Dvijotham, Katherine A Heller, Stephen Robert Pfohl, and Deepak Ramachandran. Reward Model Underspecification in Language Model Alignment. In NeurlPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023.
[21] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not climinate reward hacking. arXiv preprint arXiv:2312.09244 2023.
[22] Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. RAGAs: Automated Evaluation of Retrieval Augmented Generation. ArXiv Preprint 2XXx:23309.121177 2023.

---

[23] Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods, In International conference on machine learning, pages 1280-3291. PMLR, 2020.
[24] Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate as You Desire. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), 2023,
[25] Siddhant Garg, Thuy Vu, and Alessandro Moschitti. Tanda: Transfer and adapt pre-trained transformer models for answer sentence selection, 2019.
[26] Alastair R Hall. Generalized method of moments. A companion theoretical econometrics, pages 230-255, 2003.
[27] HazyResearch. metal. https://https://giiee http://///tttpp//tttttaat//aaaap http://hww../.o/taa////aaaaaaaaa
2018.
[28] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
[29] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute- large language models. arXiv preprint arXiv:2203. 2022.
[30] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
[31] Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Trans- former architectures and pre-training strategies for fast and accurate multi-sentence scoring, 2020.
[32] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024.
[33] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[34] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq. Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.037.. 2023.
[35] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[36] Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of Ilm outputs. arXiv preprint arXiv:2407.13692 2024.
[37] Dahyun Koo, Yejin Choi, and Eunsol Choi. Cognitive Biases in Large Language Models as Evaluators. ArXiv Preprint 23
2023.
[38] Nathan Lambert and Roberto Calandra. The aligmment ceiling: Objective mismatch reinforce- ment learning from human feedback. ArXiv Preprint 1...::2311.00111111 2023.
[39] Nathan Lambert, Valentina Pyatkin. Jacob Morrison, LJ Miranda. Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. RewardBench: Evaluating Reward Models for Language Modeling, 2024.
[40] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Guirajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follovin models. https: // //// githhb.... cow/tatsu 1ab/ aalpaca_eval,220 2023.
12

---

[41] Shalev Lifshitz, Sheila A. Mcllraith, and Yilun Du. Multi-agent verification: Scaling test- time compute with multiple verifiers, 2025.
[42] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023.
[43] Chris Yuhao Liu and Liang Zeng. Skywork Reward Model Series. https://huggingtee co/ co/Sky Skywork, September 2024.
[44] Chris Yuhao Liu, Liang Zeng. Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-roward. Bag of tricks for reward modeling llms. arXiv preprint arXiv:2410. 18855, 2024.
[45] Runze Liu, Junqi Gao. Jian Zhao, Kaiyan Zhang. Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can lb Ilm surpass 405b IIm? rethinking compute-optimal test-time scaling, 2025.
[46] Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang. Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can 1b llm surpass 405b Ilm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06701, 2025.
[47] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. ArXiv Preprint 220.ii:2333..66643 2023.
[48] Meta. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, 2025. Accessed: 2025-05-12.
[49] Xiaoyu Tan Minghao Yang, Chao Qu. anf-orm-llama3.1-20 2024.
[50] Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert, 2020.
[51] OpenAI. GPT-4 Technical Report. ArXiv Preprint arXiv.2303.08777 2023.
[52] OpenAL. Openai 03-mini system card. Technical report, OpenAI, January 2025. Publication.
[53] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. 35:27730-27744, 2022..
[54] Qian Pan, Zahra Ashktorab, Michael Desmond, Martín Santillán Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. Human-Centered Design Recommendations for LLM- as-a-judge. In Proceedings of the 1st Hluman-Centeree Large Language Modeling Workshop, pages 16-29, 2024.
[55] Isha Puri, Shivchander Sudalairaj. Guangxuan Xu, Kai Xu, and Akash Srivastava. A probabilistic inference approach to inference-time scaling of Ilms using particle-basee monte carlo methods, 2025.
[56] Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: rapid training data creation with weak supervision. The VLDB Journal, 29(2):709-730, 2020.
[57] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4763-4771,,019
[58] Alexander Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 2016.
[59] Selvan Sunitha Ravi, Bartosz Mielczarek, Anand Kannappan, Douwe Kiela, and Rebecca Qian. Lynx: An Open Source Hallucination Evaluation Model, 2024.
[60] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Conference Empirical Methods in Natural Language Processing, 2019.
13

---

[61] Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 11 2020.
[62]
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman, GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
[63] Saharon Rosset, Ji Zhu, and Trevor Hastie. Margin maximizing loss functions. Advances in neural information processing systems, 16, 2003.
[64] Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An Auto- mated Evaluation Framework for Retrieval-Augmentte Generation Systems. ArXiv Preprint arXiv:2311.0942 1.09776, 2023.
[65]
Robert E Schapire. Explaining adaboost. In Empirical inference: festschrift in honor of vladimir N. Vapnik, pages 37-52. Springer, 2013.
[66] Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Roberts, and Frederic Sala. Universal- izing weak supervision. arXiv preprint 11.Xiv22112.00665 2021.
[67] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. ArXiv Preprint arXiv:2310.0377 2023.
[68] Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai- Chang, Mareus Rohrbach, and Anna Rohrbach. When to solve, when to verify: Compute-optimal problem solving and generative verification for Ilm reasoning. 2025.
[69] Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Chang, Marcus Rohrbach, and Anna Rohrbach. When to solve, when verify: Compute-optimal problem solving and generative verification for Ilm reasoning, 2025.
[70] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024.
[71] Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. Prmbench: A fine-grained and challenging benchmark for process-level reward models, 2025.
[72] Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvement capabilities of large language models, 2025.
[73] Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling flaws: The limits of IIm resampling with imperfect verifiers, 2024.
[74] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether thfin-of- can solve them, 2022.
[75] Liyan Tang, Philippe Laban, and Greg Durrett. MiniCheck: Efficient Fact- Checking of LLMs on Grounding Documents, 2024.
[76] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating Ilm generations with panel of diverse models, 2024.
[77] Harit Vishwakarma and Frederic Sala. Lifting weak supervision to structured prediction. Advances in Neural Information Processing Systems,
[78] Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of RL.F in Large Language Models Part II: Reward Modeling, 2024.
14

---

[79]
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferecces via multi-objective reward modeling and mixture-of -epeerrss.........e.ee ArXiv Preprint 4xXi:22406.12845
[80] Jiaan Wang, Yunlong Liang, Fandong Meng. Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is ChatGPT a good NLG Evaluator? A Preliminary Study, ArXiv Preprint arXiv:2303.044: 00.440444444.. 2023.
[81] Junlin Wang, Roy Xie, Shang Zhu, Jue Wang, Ben Athiwaratkun, Bhuwan Dhingra, Shuai- wen Leon Song. Ce Zhang, and James Zou. Improving model alignment through collective intelligence of open-source llms, 2025.
[82] Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406001574, 2024.
[83] Zhilin Wang, Yi Dong. Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egent, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. HelpSteer: Multi-Attribute Helpfulness Dataset for SteerL.,, 2023.
[84] Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D' A'mour, Sanmi Koyejo, and Victor Veitch. Transforming and combining rewards for aligning large language models. arXiv preprint arXiv:2402.00742, 2024.
[85] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and lacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024.
[86] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le. and Denny Zhou Chain-of- -f-thuhhh prompting elicits reasoning in large language models, 2023.
[87] Peter West and Christopher Potts. Base models beat aligned models at randomness and creativity, 2025.
[88] Tengyu Xu, Eryk Helenowski, Karthik Abinav Sankararaman, Di Jin, Kaiyan Peng, Eric Han, Shaoliang Nie, Chen Zhu, Hejia Zhang, Wenxuan Zhou, Zhouhao Zeng, Yun He, Karishma Mandyam, Arya Talabzadeh, Madian Khabsa, Gabriel Cohen, Yuandong Tian, Hao Ma, Sinong Wang, and Han Fang. The perfect blend: Redefining rihf with mixture of judges, 2024.
[89] LU Ying et al. Decision tree methods: applications for classification and prediction. Shanghai archives of psychiatry, 27(2): 2015.
[90] Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.01981. 2024.
[91] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025.
[92] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative Verifiers: Reward Modeling Next-Token Prediction, 2024.
[93] Eric Zhao, Pranjal Awasthi, and Sreenivas Gollapudi. Sample, scrutinize and scale: Effective inference-time search by scaling verification. arXiv preprint arXiv:2502.01839) 2025.
[94] Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: a cross-system benchmark for formal olympiad-level mathematics, 2022.
[95] Lianmin Zheng, Wei-Lin Ln Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan I.i, Dacheng L.,, Eric. P Xing. Hao Zhang, Joseph F. Gonzalez, and Ion Stoica. Judging alm-ae-a-judge with mt-bench and chatbot arena, 2023.
[96] Lianmin Zheng, Dacheng Xu, Jiajun Dong, Andy Zeng. Shuo Xie, Eric P Xing, and Percy Liang. Evaluation Biases for Large Language Models. ArXiv Preprint arXiv:2305. 17926, 2023.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We ensure our claims in the abstract and introduction accurately reflect our paper's 8 contributions and scope. We clearly state our approach using weak supervision, the evaluation context, and the specific performance improvements achieved with WEAVER.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations in Section 7, addressing specialized verifier development, dataset distribution challenges, and concrete directions for improvement like extending WEAVER to RLHF and multi-modal verification.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide theoretical foundations for our weak supervision approach in Sec- tions 4.2.1 and Appendix B.1, including assumptions and complete proofs. Our mathematical formulations for the Discrete Weak Supervision Model include all necessary equations and derivations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We thoroughly disclose information needed to reproduce our main experimental results, specifying datasets, models, verification techniques, and evaluation metrics. We provide implementation details of WEAVER Appendix B.4 along with our code repository.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
17

---

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of aosed-source models, it may be that access to the model is limited in some way (e-g., to registered users), but it should possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Ys]
Justification: In our supplementary materials, we provide code repository for our WEAVER implementation along with access all datasets used in our experiments. Furthermore, our methodology is also described in sufficient detail for reproducibility in the paper in Sections and 4.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
httpe//ggiidee//oeeeeeeeee soiooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. wttpcccpppppppeeeeeeceecccccceet httgggudiss/////
/ttpg/dd///ooeeuuuiiieeeeee /Codeeuumissssseseee ssisnsolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We clearly specify all training and test details necessary to understand our results in Section 4 and Appendix C. We include details about models, datasets, parameter settings, and evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
18

---

Justification: For each of our tables and figures, we perform 5- runs and report the average of the runs. We report the standard deviation of these results in Sec. C. We also report statistical significance through trend analysis across multiple datasets with power-law fitting in Sec. C, including R² values demonstrate the quality of fit and reliability of our scaling trends.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We include the GPUS, workers, memory, and storage requirements needed for running the experiments in Appendix D.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: t/tturi/nneurpps http://rrrrr..cpppiiil/eeeeeeeee
Answer: [Yes]
Justification: Our research conforms to the NeurIPS Code of Ethics, focusing on improving AI efficiency through better verification methods. Our work aims improve Al capabilities responsibly without introducing harmful applications.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
19

---

The authors should make sure to preserve anonymity (e.g., if there is a special considera- tion due laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss broader impacts in Section highlighting potential positive impacts like improving language model reliability through better verification. Our rescarch doesn't 't introduce significant negative socictal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper doesn't release high-risk data or models that would require safeguards. Our focus is on verification methodology rather than artifacts with misuse potential.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
20

---

Answer: [Yes]
Justification: We properly cite and credit the creators of all assets used, including datasets and language models listed in Table 13. We respect the licenses of all models and datasets referenced our work in Sec. C.I.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e-g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Ye]]
We release our WEAVER implementation code and the configuration files used in supplementary materials. We provide documentation for these assets alongside code repository.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper doesn't 't involve crowdsourcing research with human subjects. We use existing datasets and computational experiments without human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine. but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
21

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Since our paper doesn' 't involve human subjects research, no IRB approval was required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration not required.
Answer: [NA]
Justification: The core method development in our research does not involve LLMs as important components. We evaluate language models but didn't use them in developing our research methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: (https://neur)) //terr//n.u..... wewpurripp..cccofffeeeeeeeeceeee cc/Coonferencces/200//111 for what should or should not be described.
22

---

A Table of Contents
1. WEAVER Methodology (Appendix B)
(a) Discrete Weak Supervision Model with Known Difficulty (Appendix B.1)
(b) Practical Considerations for WEAVER (Appendix B.2)
(c) Implementation Decisions for WEAVER (Appendix B.2.2)
2. Experiments: (Appendix C)
(a) Models and Datasets (Appendix C.1)
(b) Verification Baselines (Appendix C.2)
(c) Scaling Trends of WEAVER (Appendix C.3)
(d) Scaling Candidate Generations (Appendix C.4)
(e) Scaling Verifier Count (Appendix C.5)
(f) Weaver Distillation (Appendix C.6)
(g) Individual Verifier Optimization (Appendix C.7)
3. Miscellaneous (Appendix D)
(a) Compute Requirements (Appendix D.1)
23

---

B WEAVER Methodology
B.1
Weak Supervision Model
We can construct a data generating model over response correctness y and the binary verifier outputs s. The model is defined as:
Response Correctness:
Vij ~ (er
(ernoull(ww..):: (emoouli(1-wxoo)
if vie [e]]
Verifier Score:
Styk Mij
1f 00
where:
x is the probability that a response is correct.
Wk,1 1 is the true positive rate (TPR) of verifier and wk, is the true negative rate (TNR), which we refer to as the verifier's s accuracy parameters.
Here, each verifier & emits binary score fijk {0,1}, which is assumed to be noisy indicator of whether response Vig is correct. The likelihood of the verifier's s binary output X, E {0,1] is:
Wk.1
if Vıo and kijk 1,
Wk,1
if yku and ktyk: O,
= Y=yi))=
Wh.0s
if yiu = and 8ijA = O,
Wk
if yiy = and &iyk = 1.
We are interested in estimating the correctness of a response Vij E {0.1} based on assessments from multiple verifiers 8; = (1....5) Applying Bayes' Rule, we get:
Pr(S Styl Miy Pry =1
Pr(yi=1155 =
(6)
Pr(s.))
where Pr(3:))
-
Eq. (6) requires evaluating the full conditional likelihood:
Pr(S šig Miy) Pr((S = "......... Sijm 1yt),
which is joint distribution over binary random variables. Since each verifier S₁ E {0,1} is binary, then there are 2n possible verifier output configurations for 5 (0,1)". This results in 2m free parameters per class label to construct the distribution Pr(S= Sig Viy)-
Conditional Independence Assumption To avoid this exponential blowup, we can assume that the verifiers provide conditionally independent outputs:
p(S P(s
which reduces the number of parameters from 0(2 to (m) and enables efficient inference, under the assumption that each verifier provides unique information about the correctness of response.
Then, Eq. (6) simplifies to a Naive Bayes-style estimator:
Pr(yy = = = Pr(Si 8.g1..... ......,,,, Sm =:jm 1 =1
Pr(S = 5iz) =5))
Pr(m-= = Pr ( = Bisk [1-=
Eve(o.)
B Byy (y)=
The parameters in Eq. (7) include:
The prior probability of correctness T P =1)
The veriffer-speeiffe conditional likelihoods P(Sk yk))
24

---

B.1.1 Parameter Estimation
Supervised Setting When ground-truth labels Vij are available, parameter estimation reduces to computing empirical frequencies. We can estimate the prior as:
N
4
For each verifier k we could estimate:
1x E (.. [B0s 11-11=111-1466 -11-1(5==11
En, 1i1e =1}
Σi, [Bsy =0} Eie =0}
0
Weak Supervised Setting When few labeled By are available, we can use to estimate x, but we still need to estimate the verifier accuracy parameters , compute | Pr(SA B.jk Bij 1). Instead of using labeled data, we estimate accuracy parameters using moment matching. In particular, match observable second moments of verifier outputs to the model-implin moments under conditional independence assumptions, based on an approach from [27].
Pairwise Statisties. For each pair of verifiers ,2 and binary outputs a.be {0,1}. express the joint probability of their outputs using the marginalization rule and the conditional independence assumption:
Pr(S,, = =0,
(8)
=Pr(Sk, =
where the conditional distributions for verifier are:
Wk.1+ a=1, Pr(8:=aly=0)
1 Wk.u, a= 1,
Pr(Sa=alyp 1 Wk11
a=0,
Wk,0 a=0.
Marginal Statistics. Similarly, each verifier's marginal distribution can be written as: =
(9)
Note that this equation holds true regardless of the conditional independence assumption.
Estimation method
Construct the second order moment matrix O D (2m) ((mm) where:
0 2 111-1:2,21-1:22
Pr(S;
Px(S:=1) ViE [m]
2: 1:21, 1:2)
0r Pr(8; 1r(8;=1.5,
Pr(s,
Vi [m)
(10)
Construct the conditional probability matrix μ (2V) where each row encodes:
a 122+a.bb = a (y)
Wk..
- Wk..1
Wky.
WK.. Wkmm 0
Wkmml
Label prior matrix P 2x2 is diagonal matrix:
Pr(ji 00)
P
0
Pr(m-=1)
25

---

Then, equation is = on the entries off of the 2 x block diagonal, and equation 9 equivalent (0) pPK Therefore, we optimize the following loss to compute p..
minimize,| Munimizezul(Offtddiae - )ofPP)")off-inee" "aap="""))oanee====dda(((({)---0 [[]]dda((()))ppp"""]
By solving via gradient-descent, we obtain estimates of the verifier accuracy parameters (wK,),Wk.))
B.1.2 Inference: computing response correctness probabilities
Once the aceuracy parameters {wr,1 1 Wk.0 are estimated and P(yy = 1) computed from small labeled development dataset, we can compute posterior correctness probabilities for each response:
Pr(vi-= = 1)
II
Pr(S. = 8ighlun =
k=1
m
3x))) PP = 00)
Pr(S = 3ıynby =0)
k=1
Normalizing these, we have a full posterior ([yy=11|5- 3,,)) which provides a score with which we can select response for each query.
B.2 Adapting Weak Supervision to the Verification Setting
Given a of verifiers, we describe two modifications to the verifiers that allow apply weak supervision to the verification setting. First, we discuss the motivation for how we binarize verifier outputs and filter out low-quality verifiers. These two steps before invoking Weak Supervision methods are critical, as WS assumes that verifiers are binary and are of sufficiently high signal. Second, we will describe our proposed method for binzarization and filtering, along with results.
B.2.1 Motivation
Binarization The weak supervision algorithm described in the prior section requires binary verifier outputs. This is suitable for verifiers such as LM judges, which produce {0,1) } outputs, but verifiers that are reward models often produce continuous scores different scales. fact, naive thresholding can result in low-signal verifiers (e.g., outputting all 1s). How do we convert reward models' continuous verifier scores into binary scores?
Filtering out low-quality verifiers Beyond directly impacting the quality of an ensemble, low- quality verifiers reduce the effectiveness of Weak Supervision algorithms. How do we discard verifiers that have low signal?
Skewed marginals: Consider a dataset where Pr(y= 1) 0.5 and we have a verifier with Pr(Sk- = 1) 0.999. A skewed verifier with extreme marginal (e.g., from naive thresholding for binarization) and near-constant outputs adds little information to the ensemble. It primarily increases noise in the objective in Eq. and should thus be discarded. Yet, not at all verifiers that have extreme marginals add little signal; for instance, if instead Pr(y= 1) 1)5s0,, 0.99, a skewed verifier could be highly accurate.
Therefore, the definition of a low-quality verifier depends on the distribution of correct responses.
Breaking symmetry in the WS objective: common assumption of Weak Supervision is that majority of the verifiers have better-than-randor accuracy [23]. Otherwise, there is possibility that
the WS algorithm can yield non-unique solutions; the terms in Eq. 5 are the joint probabilities over pairs of verifiers as well as their marginals, which do not uniquely determine if verifier satisfies Wk,1 Wkko0 >0 0.5 or not. Therefore, it is critical to remove as many low-accuracy verifiers as possible ensure that the optimization procedure converges a unique solution.
B.2.2 Adaptation method
We now describe our proposed method for binarization and filtering of verifiers, after which the Weak Supervision algorithm described in Appendix B.
26

---

Table 4: Ablation of Development Set used for WEAVER Class Balance Estimation: As our reward model threshold, we set it to the default of the 0.5.
Approach Model Size
Set
Benchmarks
Size
MATH500 GPQA Diamond MMLU MMLU Average
Naive Threshold
WEAVER
70B
88.1%
52.0%
92.4%
83,5%
79.0%
(0.5 Threshold)
WEAVER
70B
90.4%
67.1%
91.1%
87.0%
84.5%
WEAVER
70B
92.4%
72.7%
93.5%
90.4%
87.5%
WEAVER
70B
20%
92.4%
72.7%
93.5%
90.4%
87.5%
WEAVER
70B
100%
92.4%
72.7%
93.5%
90.4%
87.5%
Table Ablation Verifier Selection Strategies for WEAVER: Comparison of different strategies for dropping faulty verifiers by dataset using each verifier's ' marginal probability.
Model Size
Benchmarks
Approach
Verifier Selection
MATH500 GPQA Diamond MMLU MMLU Pro Average
WEAVER
70B
No Dropped Verifiers
90.4%
52.0%
91.1%
84.2%
79.4%
Low Marginals Dropped (Mostly Negative Verifiers)
WEAVER
70B
93.4%
60.6%
91.7%
91.0%
84.2%
High Marginals Dropped (Mostly Positive Verifiers)
WEAVER
70B
83.4%
69.7%
87.9%
78.4%
79.9%
Extreme Marginals Dropped (Mostly Positive or Negative)
WEAVER
70B
90.8%
72.7% %
92.4%
85.0%
85.2 %
Table 6: Ablation of Adaptive Threshold Dev Set Size for WEAVER.
Model Adaptive Threshold
Benchmarks
Approach
Size
Dev Set Size
MATH500 GPQA Diamond MMLU MMLU Pro Average
WEAVER
70B
0.01
92.4%
72.7%
93.5%
90.4%
87.5%
WEAVER
0.05
92.4%
72.7%
93.5%
90.4%
87.5%
WEAVER
70B
92.4%
72.7%
93.5%
90.4%
87.5%
WEAVER
70B
1.0
92.4%
72.7%
93.5%
90.4%
87.5%
1. Binarization: We small amount of labeled samples (which already using compute Pr(y = 1)) to determine a threshold for converting continuous verifier outputs to binary outputs. With only to 10 labeled queries from benchmark development sets (which < 1% of the evaluation set), we can estimate thresholds that bolster performance by averages of 8.44 for the 70B models when compared binary splitting along the median score of the dataset (Table 4).
2. Filtering out low-quality verifiers: To mitigate the impact of low-quality verifiers, we prune verifiers with extreme marginal behavior, depending on the class balance. For datasets with estimated class balance between 20% and 80%, we filter out verifiers with positive rates outside this range. If dataset has fewer than 20% positive samples overall, we remove verifiers that predict positives more than 80% of the time. Conversely. for datasets with more than 80% positives, drop verifiers that predict positives less than 20% of the time. We find that verifier pruning leads to 12.5% performance improvement for the 70B model setting (Table 5).
27

---

Table Performance Comparison Between Continuous and Discrete Logistic Regression: For supervised fine-tuning on the verifier scores, the continuous model consistently outperforms the discrete variant across all datasets by avoiding the lossy conversion floats to binary votes required for the discrete variant.
Continuous vs. Discrete Logistic Regression Performance (%)
Dataset
Method
MATH 500
GPQA
MMLU College
MMLU Pro
BBH
Discrete LR
93.1
74.3
87.5
87.1
90.1
Continuous LR
97.2
78.1
90.4
92.0
96.5
Improvement
+4.1
43.8
+2.9
+4.9
+6.4
Table 8: Number of Unique Extracted Answers vs. Positive:Negative Sample Ratio per Query Correlations for Llama 3.1 Instruct Models
Llama 3.1 8B Instruct
Llama 3.1 70B Instruct
Correlation Metries
Correlation Metrics
Metric Type
Metric Type
Dataset
Dataset
Pearson
Spearman Kendall's Tau
Pcarson
Spearman Kendall's Tau
MATH 500
-0.676
-0.745
-0.565
MATH 500
-0.631
-0.842
-0.709
GPQA
-0.312
-0.117
-0,096
GPQA
-0,148
-0,093
-0.089
MMLU College
-0.595
-0.700
-0.591
MMLU College
-0.551
-0.862
-0.769
MMLU Pro
-0.590
-0.555
-0.425
MMLU Pro
-0.446
-0.693
-0.585
BBH
-0.365
-0.386
-0.300
BBH
-0.268
-0.594
-0.474
Table 9: Performance of WEAVER with Different Clusters Counts
Clusters for WEAVER Dataset
Cluster Count
Dataset
4
5
MATH 500
93.4
87.6
83.8
82.8
81.2
GPQA
66.4
66.4
66.4
66.4
66.4
MMLU College
94.9
91.7
90.1
89.6
89.8
MMLU Pro
88.4
90.2
87.1
84.6
79.8
Average
85.8
84.0
81.9
80.9
79.3
C Experiments
C.1 Models and Datasets
We evaluate our models with several benchmarks for instruction-following reasoning, mathematies, and coding: MATH500 [28], GPQA [62], MMLU [28], MMLU Pro [82], and BBH [74]. We provide an overview each dataset in Table 10. For MMLU, we selected the college-level questions for evaluation: biology, chemistry, physics, mathematics, computer science, and medicine. For MMLU Pro, we take random sample of 500 queries out of the 12K queries available. For BBH, take four tasks from the dataset of 6K queries available: Penguins in a Table, Causal Judgement, Logical Deduction (Five Objects), and Tracking Shuffled Objects (Five Objects).
Benchmarks:
Models: We evaluate candidate generations using range of weak verifiers minffesrsmmodelss with imperfect better-than-raanom accuracy. Our verification system includes two primary classes of weak verifiers: Reward Models and LM Judges.
28

---

A reward model (RM) is trained language model that assigns scalar score to candidate responses based on how well they align with human preferences [39, 71]. Given query and candidate response, the RM outputs a value Vij E [0.1] representing the estimated quality of candidate j according to criteria such as correctness, helpfulness, and safety.
Reward Models:
Examples of reward models include those from the RewardBench leaderboard [39], such as INF-ORM [49], QRM Gemma [17], and Skywork Reward [44]. We also include process reward models (PRMs), which score the reasoning process itself- emphassiinn step-by- step logic and coherence rather than just the final answer [15, 90].
For our study, we selected the top-20 reward models from RewardBench and the top-20 process reward models from Process Reward Bench [71] at both 8B and 70B parameter scales. We exclude any RM or PRM that fails to provide a positive learning signal-i.e. those whose rankings perform hetter than random selection benchmark train sets (Appendix C.I). The diverse training objectives and datasets used for these reward models introduce systematic biases that affect their verification capabilities [39, 71], with different loss functions--inlludigg Bradley-Tery loss for pairwise preferences [4], margin loss for fixed score differences [63], and pairwise ranking loss for relative ordering [6].
Previous work has noted that is nontrivial combine the outputs of reward models and judges as they provide logits and binary decision rules [76, 88]. Instead, we find that we can normalize all RM scores to the range [0,1] using robust percentiles: the bottom 5th percentile is mapped to 0 and the top 95th percentile to 1. For models that provide multiple scoring dimensions (e.g., ArmoRM (79]), we use only their primary output. For study, selected the top-20 reward models from RewardBench [39].
LM Judges: An LM judge is a language model used to assess the correctness of a candidate response by generating binary verdict: Vij E (0,1). where I indicates that the response is judged correct. These models typically apply shaii-oof-thoughe (CoT) reasoning to arrive at their decisions [86]. Each LM judge takes query and response input and outputs single binary verdict.
We well-known chat models from ChatBotArena [13] as LM judges, which known for their general-purpose reasoning capabilities. To ensure consistency and determinism, we greedy decoding (temperature T =0) when generating, judgments.
junderetnngjjdgmmetts
Table 10: Benchmark Overview: Evaluation configurations for AlpacaEvval 2.0, Arena-Hard-Auto, AIMO, MATH500, GPQA, MMLU, MMLU Pro, and Big-Bende Hard (BBH).
Benchmark
Dataset Size
Scoring Type Metric
License
MATHS00
500
Ground Truth
Pass@1 I
Apache 2.0
GPQA
646
Ground Truth
Pass@1 1 CCBY 4.0
MMLU College
719
Ground Truth
Pass@1
MIT
MMLU Pro
500
Ground Truth Pass@
MIT
29

---

RewardBench
ChatBot Arena
250
200
-20
Roward Model
of
200
atoed
stoua
2014
Mbc ator
Dec 2tdd
Nl 2020
o2 2014
o2 raga
hoz
Age
of
Mon
Quarters
Figure 7: Growth available, the need for better selection and utilization strategies for these models test-time continues to grow.
RMs and LMs: As more and more RMs and LM judges become
Table 11: Distribution of Generation Accuracy across Datasets for Llama 3.1 8B Instruct Model
Llama 3.1 SB Instruct
Distribution of PositiveeNeeative Generations
Percentage of Total Dataset
Dataset
Ratio
0.0-0.1
0.1-0.2
0.2-0.3
0.3-0.4
0.4-0.5 0.5-0.6 0.6-0.7 0.7-0.8
0.8-0.9
0,9-1.0
AIMO
72.2%
7.8%
3.3%
7.8%
2.2%
3.3%
2.2%
0.0%
1.1%
0.0%
10.3%
MATH
12.2%
11.4%
11.6%
8.2%
6.4%
8.0%
8.4%
6.60%
11.2%
15.0%
49.9%
GPQA
28.5%
21.2%
16.1%
7.4%
7.3%
5.3%
3.7%
3.3%
2.9%
2.6%
28.3%
MMLU College
7.8%
7.6%
7.1%
6.5%
6.7%
5.3%
7.2%
6.3%
8.6%
22.9%
64.1%
MMLU Pro
22.4%
9.8%
10.6%
5.4%
6.4%
7.2%
4.0%
7.6%
7.6%
15.2%
46.6%
BBH
3.2%
6.7%
10.3%
8.3%
10.3%
14.8%
11.6%
10.7%
10.4%
12.1%
56.9%
Table 12: Distribution of Generation Accuracy across Datasets for Llama 3.1 70B Instruct Model
Llama 3.1 70B Instruct Distribution of Positive//eeaativ Generations
Percentage of Total Dataset
P/N
Dataset
Ratio
0.0-0.1
0.1-0.2 0.2-0.3
0.3-0.4
0.4-0.5
0.5-0.6 0.6-0.7
0.7-0.8
0.8-0.9
0.9-1.0
MATH 500
7.0%
4.2%
3.8%
2.0%
2.2%
3.8%
4.2%
4.0%
7.8%
61.0%
78.0%
GPQA
36.8%
5.6%
5.1%
3.7%
6.2%
4.3%
4.5%
4.8%
8.0%
20.9%
42.9%
MMLU College
8.1%
3.2%
1.8%
2.2%
1.5%
1.7%
1.8%
3.2%
2.4%
74.1%
82.6%
MMLU Pro
16.4%
4.2%
1.8%
3.4%
3.0%
3.0%
3.2%
4.8%
6.0%
54.2%
69.9%
30

---

Table 14: WEAVER Verifier Accuracies and Covariances
Model
Benchmarks
Approach
Size
MATH500 GPQA MMLU Pro Average
Verifier Accuracy Range
70B
32.3%
43.6%
36.7%
37.5%
Verifier Covariance
70B
0.0284
0.0341
0.0342
0.0322
Verifier Accuracy Range
8B
25.6%
31.4%
32.1%
29.7%
Verifier Covariance
8B
0.0199
0.0337
0.0240
0.0259
Table 13: Models Tested for WEAVER.
Parameter Count
Model
Source Code
License
Loss Function
100ma-3.--708-Insmmme
Open-Source
70B
Ilama 3.1 Community
Cross- Fntmpy Loss
Llama-3. 1-40SB-lnsttruct
Open-Source
405B
Llama 3.1 Community
Cross-Eatropy Loss
Llama-3.3-708 3amma-333770--Instttee
Open-Source
Liama. Community
Cross-Eatropy
Meta- Llama 3.1 1055B Instract quantized
Open Source
405B
Llama 3.1 Community
Cross: Entrupy Loss
DeepSeck
Chat
Open-Source
DeepSeek License
Cross. Pooss-Ftropp
Lammac
Open-Souree
70B
DeepSesk License
Cross-Eatropy Loss
DeepSerkQwen128
Open-Source Source
32B
DeepSeck License
Cross: Entropy Loss
DbepSeckLJama88
SoureS
DeepSeck License
Cross-Entropy Loss
DoepSockQwen7B
Opea-Source
7B
DeepSock Liecuse
Cross-Eatropy Loss
LM Judges
Qwen2 72B Instruct
Open-Source
Tongyi Qianwen
Entropy
Qwen2.5 72B Instruct
Open Source
72B
Tongyi Qianwen
Cross Entropy Loss
Qwen/Qwen2.5-722
Open-Source
Tongyi Qianwen
Cross- Entropy
QwQ-32B
Opea-Souree
32B
Apache 2.0
Cross-Eatropy Loss
Qwen 1.5 110B Chat
Open-Source
110B
Tongyi Qianwen
Cross Fntrop Loss
Qwent 72B Chat
Open- Pen--Souree
72B
Tongyi Qianwen
Cooss-Entropy Loss
Qwen-2.5 5-78-nnstructt Qwen 2.5 BB-nstrrt
Opea-Source
7B
Tongyi Qianwen
Cross- Cooss-Eatropy Loss
Open Source
Tongyi Qianwen
Entrupy
Mixtral 8x22B vO.
Open Source
176B
Apache 2.0
Cross-Fatrry Fntropy Loss
Mixtral-8x
Open-Souree
Apache
Cross-Entropy Loss
WizardLM 8x22B
Opea-Souree
176B
Apache 2.0
Cross-Eatrop Loss
W-zardL
Open-Source
176B
Apache 2.0
Cross-Entropy
Cross Entropy Loss
dbrx-instruct
Opea- Peca-Souree
132B
Databricks Open Model
Cross-Eatropy
SkyT1
Open Source
328
Apache 2.0
Cross Cross-batropy Loss
Open-Souree
8B
MIT
Pairwise Ranking Loss
GRM-Llamad 2-38-22--Brrr- 1i-rewardmmode--
Open Source
31
Apache 2.0
Pairwise Ranking Loss
GRM Gemmuz 2B rewardmodel ft
Open Source
2B
Apache 2.0
Pairwise Ranking Loss
RMs (801 and helow))
Skywork Rew.rd--llama.......
1-82-v0.2
Open-Source
8B
Skywork License
Pairwise Ranking Loss
QRM-Llama3,
Open-Souree
MIT
Quantile Regression Loss
M.M.-1aaMa.31-88
Open- Source
8R
Skywork I License
Uncertainty-Ae Aware I Loss
GPM-Llama-3.1-83
Open- Ppen-Souree
MIT
Pairwise Ranking Loss
Opea- Sour-Soree
8B
Llama 3.1 Community
Pairwise Ranking Loss
AmoRM-LLama.3 --8--0000
Open Source
8B
Llamu Community
Pairwise Ranking Loss
Qwen2.5-Math PRM-7B
Open Source
7B
Tongyi Qianwen
Cross Entropy Loss
EurusPMM-Stngy
Open-Source
7B
Apache 2.0 Apache
Cross-Entrops Loss
BurosPRM-Stage2
Open-Souree
Cross-Eatropy
internlm2--b-rewward
Open-Source Source
7B
Apache 2.0
Pairwise Ranking Loss
Decialon-Tres-Rewaddee Thw----adddad--aa---33......
Opea- Peca-Souree
Skywork License
Decision Troe Loss
RMS (2713 (771)-2233 7283)
Skywork-Reward Reward
QRM--Gmma-2227B
Gemma-2-27B--00
Open-Souree
Open-Source
27B
Skywork License
MIT Custom License
Quantile Regression Loss
Pairwise Ranking Loss
AN--ORM-llama3 1 1-70B
Opea-Source Source
70B
Binary Cross Entropy Loss
Qwen2.5
Open Source
Tongyi Qianwen
Cross. Entrupy Loss
Open- Source
72B
Tongyi Qianwen
Cross. Cross-Fattroy Fasropy Loss
intemlin2-20b-reeatt
Open-Souree
208
Apache 2.0
Pairwise Ranking Loss
Decision: Tree- Reward Gemma-2-277
Open Source
27B
Skywork License
Pairwise Ranking
C.2 Verification Baselines
C.2.1
Verifier-Free Approaches
First Sample (Pass@ 1): This baseline uses only first generated response without any verification or selection mechanism. It represents the standard approach where models generate single response and provides lower bound for performance comparison. This method does not scale test-time compute or employ verification.
31

---

Majority Voting: A verifier-fre freeer--feeeeee approach that generates multiple candidate responses and selects the most frequent final answer across all responses [5, 11, 70J. This method leverages repeated sampling but does not use verification models to assess response quality. Instead, it relies on the assumption that correct answers will appear more frequently than incorrect ones across multiple generations.
C.2.2 Alternative Verification Strategies
Naive Unweighted Aggregation: We consider three oracle configurations using the top-1, top-5, and top-10 verifiers (ranked by their their agreement with ground-truth labels). Across all datasets, these oracle ensembles substantially outperform baselines. On average, the best-performing unweighted ensembles exceed first-sample performance by 20.3% and outperform majority voting by 15.0% (see Figure 2). For more difficult benchmarks such as GPQA and MMLU Pro, the top-5 and top- 10 ensembles consistently outperform top- 1, suggesting that verifier diversity is especially beneficial on challenging examples. However, these oracle ensembles rely access to ground truth to rank verifiers, limiting their use in practice and motivating the need for learned, unsupervised weighting.
Naive Bayes: We implement Naive Bayes classifier that models the probability of response correctness given verifier scores: P(yo = ((j,))mm...))mmm P(sy) ...m My, = =())))=)))))))))) Under the conditional
independence assumption, this factorizes as P(s:j1, Stjm Biy 1) = I P(sijk Vij 1). We estimate the parameters using labeled data from the development set. This approach provides probabilistic framework for aggregating verifier outputs but requires labeled data for parameter estimation.
Logistic Regression: We train a logistic regression classifier where the input features are the verifier scores (s.j1...,,ijmm and the output the correctness of each response: Р(и,- = =[ssi) = sij + +b), where a is the sigmoid function. The weights w and bias b are learned using labeled training data. This supervised approach can capture more complex relationships between verifier outputs than naive averaging but requires substantial labeled data for effective training.
Multi-Agent Verification (MAV) [41]: This approach combines multiple "Aspect Verifiers" (AVs)- off-the-shelf LLMS prompted to verify specific aspects of candidate outputs through binary True/False approvals. Unlike reward models, AV require additional training and can be easily combined through voting mechanisms. The MAV framework uses BoN-MAV (Best-of-N with Multi-Agent Verification), which: (1) samples candidate outputs from generator L.M,, (2) collects binary approvals from multiple aspect verifiers that vary across three dimensions (base LLM, aspect to verify, and verification strategy), and (3) selects the output with the most approvals. In our implementation, we use Llama 3.3 70B Instruct as the judge model rather than Gemini 1.5 Flash/Pro as used in the original paper.
Self- Verification [93]: This method implements a sophisticated sampling- based search approach where models verify their own responses through detailed natural language analysis. The approach goes beyond simple self-critique by using structured verification prompts that: (1) rewrite candidate responses in rigorous mathematical theorem-lemma-prun through step-by-step analysis, and (3) compare responses to localize potential mistakes. The method leverages two key principles: comparing across responses provides signals about error locations (since models struggle with error recall but can identify errors when given their locations), and different output styles are optimal for different tasks offain--f-thuughh for generation, rigorous mathematical format for verification). This approach differs from naive self gerfiieiiiiattin by using structured, multi-step verification protocols rather than simple correctness judgments.
format, (2) systematically scan for errors
32

---

Table 15: Logistic Regression and Naive Bayes Performances across Datasets and Dev Set Sizes
Model
Dev Set Size
Dataset
Approach
Size
0.01
0.05
0.2
0.5
1.0
Logistic Regression
70B
70.5%
74.7%
81.4%
93.1%
97.2%
MATH-500
Naive Bayes
70B
67. 4%
78.1%
85.0%
89.2% 92.2%
Logistic Regression
70B
55.9%
59.4% 69.8%
71.4% 72.9%
GPQA Diamond
Naive Bayes
70B
47.2% 49.2% 57.6% 62. % 64.3%
MMLU Logistic Regression
70B
72.1%
1%
81.0% 84.6% 86.0% 92.0%
Naive Bayes
70B
60.2% 73.1% 73.1% 78.6% 78.6%
C.3 Scaling Trends of WEAVER
Scaling laws describe how performance metrics such accuracy, sample efficiency compute cost change as we scale controllable resources, i.e. the number of trials K, model capacity. [33] showed that, for axed-parameter Transformer language models, the cross-entropy loss decreases in both model size and data. This framework has since been extended to explore optimal tradeoff's between model and data scaling [30], well inference-time scaling with multiple samples [5, 12].
power-law
First, we establish the power law scaling of the Pass @ rate. Assume the i-th problem has an unknown "difficulty" p. [0,1], the probability that one response is correct. With K independent samples, the chance we get at least one correct response is
a(p;,K=11((0 (1-)))) x1 expl K) for small pt
Define the indicator variable:
1, the i-th query is solved least once (with probability q.),
X;
0, otherwise,
and let the total number of solved problems be Y --= X,
The expected coverage (Pass@ K) is the expected fraction of problems solved after trying K times per problem:
Pass@K:=
1 N
Σ0 (1-(1-p))
To model population-leve variation in problem difficulty, assume each problem's 's correctness probability p; is drawn from a Beta distribution: pt ~ B). This captures the idea that some problems are easier (high p.) while others are harder (low pt). with the overall distribution controlled by the shape parameters a, 3. Then, the fraction problem that can be solved in K attempts follows,
Pass@K Ep-Beutaa 1)
B(a, B+K)
-1-Ep-Rem 3y [1 - 11
(11)
B(a, B)
by the definition of the Beta function B(-.)) Taking logarithm:
B(a, (1 (K)K
B(a, B+K) B(a, 3))
logPass @ Kolog
B(a,3)
by log(1 -z) z when æ is small, which holds for large K. Then, expressing the Beta function in terms of the Gamma function leads to:
K(8+K)T(a+3) r(0)r(a+++K)
logPass@K<
33

---

For large K, we can apply Stirling's approximation of the Gamma function logI(x) glog(2x) + logz:
(x) = rlogæ 2
logl log[-logPen Passss |g
B+K a+ -++3+K K
2 -
= =ologg +ogg
when we retain the leading term. turn, the log of the expected coverage follows power law in K, scaling as:
logPass @ k= - logk +log()==
(12)
Verifier Success Modeling Now suppose we pass the K candidates through scoring model ("verifier") which selects the top-scoring answer. The verification process succeeds if (i) at least one correct answer was generated and (ii) the verifier ranks a correct answer highest.
Selection@ 1 ((K)== P[top-scoring response is correct)
(13)
Assume the verifier assigns scores such that correct responses drawn from score distribution f1 and the incorrect responses from distribution fo- Let 5( {s, : Mo =1} and 5 = {s, y/j =0}
denote the scores of correct and incorrect responses, respectively. Then a query is successfully verified if:
Selection@ @ P maxx(1) maxx(
Our goal is to compute the probability that the maximum of c draws from fi exceeds the maximum of K draws from fo-
To model the correctness of responses, assume each query i has a latent correctness probability H) ~ Beta(a,,3), reflecting query-specifie difficulty. Given Pi, each of the K responses sampled independently as:
Vij ~ (p... j=l,..,K
This implies the number of correct responses follows a Binomial distribution:
K
C
Mi ~ Binomiaam
assuming (1) conditional independence of responses given pt. (2) identical correctness probabilities within query, and (3) fixed number of responses K.
Because the correctness probability p varies across queries, the dataset-level Selection@ curve requires marginalizing over p:
Selection @ (KK = Bp-BBua(a,3)) [Selection @ 1(K [p)]
Combined with the need to model max comparisons over verifier scores, renders the exact calculation of Selection@ @ analytically intractable.
To enable tractable, smooth modeling of Selection @ we introduce the following parametric form:
Selection @ 1(K) (exc))(xxp((<K<" (1- (1-(1-7)*3
(14)
The coverage term exp(-CK-9) approximates the probability that least one correct
response is generated.
The verification term 1-(1-x)** approximates the chance that the top-scoring response is correct, given that least one correct response exists. The parameter 7 controls whether veri- fier performance improves sublinearly or superlinearly with K. The parameter represents the effective per-response probability that a correct response is successfully selected by the verifier, conditioned on the response being correct and included in the candidate set.
34

---

To obtain practical scaling trends, we fit parametric models in Eq. (14) to the empirical averages computed from 5 independent runs for each value of K, across each dataset and verification strategy. Specifically, we use the L--BFGS-B B algorithm to optimize smooth approximation following [29]. To ensure numerical stability and robustness to outliers or heavy-tailed noise the observed selection accuracies, we minimize the Huber loss between the predicted values and the empirical means. The Huber loss behaves quadratically for small residuals and linearly for large ones, making it less sensitive to outliers than mean squared error (MSE) while maintaining smooth differentiability for gradient-based optimization. It is defined as,
2 Ia(r)= = (8(1)) otherwise
where d'> is a tunable threshold that controls the transition between the two regimes, We search over 8E 00000...05.0.1,...5....
to select the value that yields the best fit.
Additionally, we introduce floor and ceiling parameters to bound the predicted values and model saturation behavior. The floor accounts for the irreducible failure rate even at high K, while the ceiling models the upper bound on achievable performance (e.g., due to imperfect verifiers or ambiguous problems). The final fitted form is:
(1-((-77) (1-00(-()))111((-7))))
Selection @1
I (K) Rz floor - (ceil (ceil-floor)-ee (ooor)-epp((-5) exp(-cK-n) ()
(15)
We can use an unbiased estimator to evaluate best-of-k selection accuracy when a fixed verifier is used rank responses, as described in [69]. However, in the case of WEAVER, the development set constitutes 1% of the data and is itself selected based on the value of K. In turn, the ranking of responses no longer independent of K, introducing bias into the ofst-of-k estimate a result, we instead rely on Monte Carlo estimates to approximate best-of-k performance, sampling k responses multiple times and computing the average accuracy of the top-ranked output under the K K-dependent verifier. We use an unbiased estimator for coverage, as described in [12].
Fig. 8 and Fig. 9 along with Table 16 illustrate how the different verification strategies scale with the number of generations and the fit to the parametric form in Eq. (14), Each method exhibits characteristic scaling behavior that aligns with Eq. (14). WEAVER demonstrates improved performance over naive ensembles and majority voting. The fitted parameters in Table 16 quantitatively capture these trends across datasets, providing evidence that the parametric from in Eq. (15) closely model empirical outcomes. Fig. 10 and Fig. 11 along with Table 17 illustrate the predictive performance of the parametric form in Eq. (15), showing that models fit on subsets of K can extrapolate to unseen values of K.
Table 16: Fitted parameters for Scaling Trends in Fig. 8 and Fig. 9.
Datsett
Approech
Expatice
Iix
ccll
12
R2
MSE E1
GPGA-22 Dhansnni ((18)
Parik
p-Booc Booe) capt- C-K
0.5000
69419
0.7013
0.0000
0.5000
GPQA-22 Dianceni (70B)
3-Bax Bose) capl C-8 Cl
0,3058
0:675
0.7320
0.3250
0.5053
00994
0.000
0.1000
GPQA-22 Dinneni
Ma
3-1ax (ose)-eapt C-8 cId
044233
0,410
0.0499
6.121
1.009
08634
0.0000
GPQA-2 Dirrrel
Nave Eroenbie
3-Bacc Beatt--etsettCaa CK c10 (1-23
0/671
0.003
0.424
0.5000
02500
MayerytiK
Pav8
0.000
69825
0.300
0.000
0.500
MM
63145
13551
1,000
0.2764
0.5000
0.0000
Mijety18K
Molacttolll-aaao Nooe -eoe----------------- (10-KK((0-00 (10-)1-))))
06955
0.738
1206
0151
1.0055
09151
00000
Neww Fraambe
S- (call- feo----dooe--------- -wpp:---p-----0000 (100-----00000-7>>>>
0.8311
1.551
1200
05651
09941
01000
0.500
Mbll.--Collee-22111 (T16)
Parig
g-dooe (usi done -apl (-8-))
0.5044
05734
0301
0392
00952
0.000
0.0000
Mha.ii-Callee-222 (118)
Wonter
Mooee (catl- Rooe -xpp-c-K- 11a-------00------000 (10-(1->))*
08234
0947
42153
0.00
00997
01033
MWb.u--Eillge-222 (188)
MayesylNK
Booe (cat) Rone -xpl-C-K (10-(1-7))
03412
01084
12881
0385
1003
05912
01330
MblLu--Cologe-2211 (118)
Nave Emaenbie
Bose (cad- Bone -xpl-C-K (1+-(2=))"
01233
09226
53365
5200
0581
6.000
05915
03330
GPQA-17 Diarninl (5R)
Pace K
Moo (oee--tee---oee the -xpt-c-K--
0356
09536
1017
X
06966
0.000
0.0000
GPgA-y7-10 Dil.n-na (1R)
Woner
Hoo Mow -apt-c----10 (10(1-=))))
0366:
0/119
1000
01851
04445
0.00
R000
GBgA-y7-120 Diariiat (if))
Majeryl8K
Mooo- How (100-----00001--)0""
0.333
13311
00000
2000
05909
R0000
GPOA-22 1004-22.00 (iarnon) (8B)
Nave Emsnnii
Baa (uel) thee eapt (-8-CILL (1 (1-v)**)
0.339
03333
1200
03871
0.500
D5406
0,001
00000
MATH 300-000 -2 (53)
Pav8K
g-lact (oetl Booe -apt-ppp <-K C)
0.0099
1,000
17782
X
09976
0.001
0.0000
MXTH -01-22 (88)
Waner
(ente Bore -apf-c-K (1(--(1-))"
14100
024400
1000
039133
00214
03000
MXTH 300-221000 (38)
Maseay 16K
Baoe (exn. Booe -apf-c-K (1(--(1-2)
02015
51181
1011
50986
25000
09954
03000
MATH 301-22 (38)
Nave Enscalie
b-oae (uat) Booe) capl C-K cttL (1 =
0.507
0/3566
12084
30000
02990
0.5000
00756
0,001
0,00000
Packk
3- Moor+tcal---oo -aapil---ee---pp--000
0399
1030
1991
(171))
X
89901
010000
Waner
3- -ooo+ Cei (10 (10-(1=)))
3860
621
89885
01000
MMLLL--Paae (88)
MayesylKK
2 s 0at
04429
6MN
02297
19284
01001
Mh-.1-PPwr2 (88)
Naw Ersemble
g- -.leettoll-Buee--epppp -epp----t (11)))
84771
0,301
1589
1600
61711
85181
89866
01000
10000
Parek
g-dac eapi
04316
60914
0968
1972
X
69994
0.000
0L5000
Wezer
2- capl (00
0,626
03494
1346
3400
63346
0.000
80358
0.000
0.250
MMLLU-CHlE Call--lll---2111
Mas
1-llaci/sil cas (-8-0M
0.796
1925
03130
6/576
25000
00349
0.0000
0.000
(2
Nave Ereendie
g-llact/eill Booe (-tttt---CC0----0000 (1-)))))))
0.005
1429
3,0000
0
0.500
09945
0.0000
35

---

GPQA-v2-Diamond (708)
MATH-500-22 (708)
100
Success Rate (%)
Success Rate (%)
90
60
80
Passøk PL(Exp) (Rit)
40
70
Weaver Bernuuli (Rit)
Majority Bernouli
NalveEns Bernuuli (Fit)
60
2
2
2
27
21
2
2
27
k (Number of Generations)
(Number Generations)
MMLU-Pro-v2 (706)
MMLU-College-22 (709)
100
100
Success Rate (%)
90
Success Rate (%)
90
80
85
70
75
60
70
2
2
2
27
2
2
2
27
k (Number of Generations)
(Number Generations)
Figure WEAVER Scaling trend fit for 70B models
uPiamon- Diamond (70B)
MATH 500 (708)
100
Success Rate (%)
80
Success Rate (%)
90
60
80
Pass@ik PL(Exp) (Fit)
40
70
Weaver Bernouli (FRt
Majority Vote Bernouli (Fit)
60
2
2
2
21
2
k (Number of Generations)
k (Number of Generations)
MMLU-Prr-v2 (70B)
MMLU-College-22 (708)
100
100
95
Success Rate (%)
90
Success Rate (%))
80
85
80
70
75
60
70
21
23
2
27
21
2
2
k (Number of Generations)
k (Number of Generations)
Figure 10: WEAVER Scaling trend predicted for 70B models
36

---

APPA-VV-Diamoos (88)
MATH-500-22 (88)
100
90
80
Success Rate (%)
80
Success Rate
60
70
40
Pass@k PL(Exp) (Fit)
50
Weaver Bernouli (Ri)
20
40
Majority Bernoulii
NaiveEns Bermoulli (Rt)
2
2
2
27
21
2
2
27
k (Number of Generations)
(Number Generations)
MMLU-Pro-v2 (8B)
MMLU-College--2 (8B)
100
100
Success Rate (%)
Success Rate (%)
90
80
80
60
60
40
50
2
2
2
27
2
2
2
27
k (Number of Generations)
(Number Generations)
Figure 9: WEAVER Scaling trend fit for 8B models.
GPQA-v2-Diamond Diamond (8B)
MATH-500-V2 (8B)
100
100
90
Success Rate (%)
80
Success Rate (%)
60
70
60
40
Pass@k PL(Exp)
50
Weaver Bernoulli (Fit)
20
Majonity Vote Bemnulli (Fit)
NaiveEns
21
2
2
21
2
k (Number of Generations)
k (Number of Generations)
MMLU-Pro-v2 (BB)
MMLU-College-2 (8B)
100
100
Success Rate (%)
Success Rate (%))
80
80
70
60
40
50
21
23
2
27
21
2
k (Number of Generations)
k (Number of Generations)
Figure 11: WEAVER Scaling trend predicted for 8B models
37

---

Table 17: Fitted parameters for Scaling Trends with 90% of data in Fig. 10 and Fig. 11.
Dower
Apponara
Siuaatio
ce1
62.0
rPgA-ch Dernned
Woase
0-070
0.67%
16777
D-3163
LRQA-12-baaaad (708)
3"Boxi lenk Boor)ann <A 23--7--
66530
09075
00064
1.041
0.500
00000
0.000
0.0929
EPgA <7 Rieeaad (300)
0.08M
06061
03211
1805
0.500
0.000
0.0000
0.0600
(245)
0/69
0990
XX1TH-90-22/11)
0.599
0.000
KXTH 27022/700
D-Booetoed -aaetorr--aeeecee
0.727
XXL.EPPu-22000 (706
0000
09985
0.00
0350
00000
0.000
0.009
Wezver
B4912
09148
1257
3900
1200
02000
10984
20800
0.000
0.330
MecaageS
2 -oxoe-leck 3-Boe-ieed-BBoee-aaae -aore--sp----C- CII (3---)--))))
B6623
6750
30000
1900
11211
10005
10081
20000
0.000
0.000
ANn (-P
3- Thoe-tor--Borr-rrpp (anp---myyc----0---111-10-
0960
15.5
1000
0000
00000
Mjoayieg
08:15
1080
1301
1001
68234
6006
1500
1000
0306
0.000
10000
00030
00000
0.083
GPQAxx2 Gianeed (88)
RoskK
3- - -.
0,225
6960
0630
10000
00000
0000
0.350
oPgA-22 Staeand (tt)
Wezner
3= Baati----or---------- (00pp---c--000--23---00000
B3316
0/60
2009
2007
7000
03083
0.889
(16)
Mieert
0.001
XLATH (00))
haeK
03986
1000
1200
02566
00005
0.009
Wasna
lork Bcor)-capC-CA -00 (3---/0-----
B4t0)
62411
1004
0231
0515
0.000
0.002
SEATI 0800
Meeayyes
B.961
1981
115m
2240
1940)
01000
00630
5XTH--90-22085-
Nawe Eascdree
tol Bcse) sapl < <."" (11-11-==
0.900
1808
1000
0,000
1000
1001
05565
0.000
3-Bas leed (000------000--3-2222200
0.977
1000
0550
0.063
1900
AMELL- AMLL--haeGG
B4777
6:337
13010
3300
51013
0.000
0.009
SOELL-CCRRg-nE (88)
hack
D0M1
1/C%
8906
X
(0000
08006
0.500
Weowe
06178
09687
11:33
1000
0.305
6,000
0000
03300
XXELL--OAap- (88)
3-doo- fend Bcori sapt 1aa-pp----0------1100011 (X-0001-13-2/4")
BA244
67186
55766
69281
0.308
1,2277
19726
00000
0.0066
0.5939
XONLL-OOkgpo2200 (B)
2 2oo Borr sap. C. (00--11-//")
B/074
0/911
1.250
0.006
03060
2,4001
0.0000
0.0029
38

---

Table 18: WEAVER with 8B Models Exceeds Majority Voting and Naive Ensemble across All Datasets: Candidate are generated with Llama 3.1 8B Instruct while the weak verifiers are 8B parameters smaller in size.
Datasets
MATH 500
MMLU
MMLU
Methodology
Generations (K)
GPQA
College
Average
First Sample
28.3%
64.1%
46.6%
47.2%
Majority Voting
100
69.0%
30.5%
72.7%
56.4%
57.2%
Insetines
Top- SoppRRnked RM from RewardBench [39]
100
73.8%
25.4%
70.1%
53.4%
55.7%
Top-10 RM Ensemble from RewardBench [39]
100
70.2%
22.1%
73.9%
49.4%
53.9%
Multi-Agent Verification (41]
65.4%
31.4%
70,5%
55.2%
55.6%
Self- Verification [93]
100
71.4%
32.2%
70.4%
53.0%
56.8%
WEAVER
80.0%
47.1%
85.7%
67.25
70.0%
GPT-4o-mini
76.8%
38.45 49.4%
82.2%
61.8%
64.8%
Claude 3.5 Haiku
70.0%
36.4%
75.9%
65.2%
61.9%
Oracle Verifier (Pass 100)
100
99.2%
95.2%
98.5%
96.8%
97.4%
C.4
Scaling Candidate Generations
MATH500
GPQA Diamond
MMLU Pro
20.3
27.5
False Positives Predicted
20
22.5
Naive Ensemble
30.0
Weaver Distiled
5.0
0.0
Total Models Verifier Ensemble
Total Models Verifiar Ensemble
Total Models in Verifier Ensemble
Figure 12: False Positive Rates across Verification Systems
GPQA Diamond
MATH500
MMLU Pro
90
90-
Success Rate (%)
80
70
60
50
Gain
60
60
40
Fist Samale
Sample
29
23
22
2
20
Number of Repeated Generations
Number of Repeated Generations
Number of Repeated Generations
Mejority1@e
MulliAgent
SelfVerification
Figure 13: WEAVER Scaling 8B Generations and Models
C.5 Scaling Verifier Count
In Table 19, we include results of scaling verifier scores. We note that for reward models (RMs), which are typically deterministic [39, 71]. multiple scores must be obtained by varying the prompt; for LM Judges, we can vary either the prompt or the sampling temperature to generate diverse outputs from the same model (Table 19). We find that for both types of weak verifiers, RMs and L.M judges, scaling the number of models yields better performance than sampling multiple evaluations from the same model via prompt tuning or temperature variation. However, we note that these approaches are complementary.
39

---

Table 19: Ensembling with Multiple Verifiers Outperforms Increased Sampling with Single Verifier: Candidate responses are generated with Llama 3.3 70B Instruct while the weak verifiers range size from 8B to 72B parameters. For details on prompting, please see Appendix C.7.
Benchmarks
Methodology
MATH500
GPQA
MMLU Pro
First Sample
78.0%
42.9%
69.9%
Majority Voting
47.4%
74.4%
Best Reward Model
94.4%
58.4%
81.8%
Score)
Best Reward Model (5 Scores, 5 Prompts)
93.2%%
55.3%
82.5%
Top-5 Must Accurale
Reward Models
95.4%
64.1%
87.3%
Bost LM Judge (1 Score)
90.2%
61.1%
79.5%
Bost Judge (5 Scroes, 5 Prompts)
S8.15
57.2%
Top-5 Most Accurate Judges
93.4%
65.2%
85,4%
Accuracy Heatmap: MMLU-Pro (70B)
Natve Encemble
VSS
Discrete
VS
Continun JOU
0.90
35
SF2
6:2%
3:77
1.6
145
2.
322
0.54
0.10
0.L
3.5
E.07
28
Verthtee
STC
6:24
X77
0:1
3.00
5.0
575
6.24
3.5
6.54
341
0.0
87
6:2
3.5
CM
6.02
20
345
6.24
3.50
0.02
3.55
340
6:24
341
6.54
381
0.00
6:7
0.9
D4T
3.6
6M
6.71
22
6.2
X79
0.0
366
4:2
345
6.34
6:44
6:55
045
6.72
379
0.71
62
0.00
6:7
3.66
0.0
6.67
100
100
Number Reposted Generations
Number Repeated Generations
Number Repested Generations
Figure 14: WEAVER Performance Improvements from Scaling Generations and Verifiers: In- creased candidate generations and weak verifiers available generally improves performance.
When breaking down weak verifiers into RMs or LM Judges, individually, we find that additional LMs leads average gains of 5.4% and 6. 1%, respectively (Table 19). In contrast, sampling additional scores from single RM LM judge yields only 0.8% and 1.1% gains on average. These results suggest that leveraging the complementary strengths of multiple verifiers can be more effective than eliciting multiple judgments from a single verifier. See. C.7 provides additional details on the verifier prompting. Finally, Figure 14 illustrates the tradeoff of scaling the number of verifiers versus increasing the number of scores from a single verifier, showing that scaling verifiers is helpful when the coverage increases as we increase sample count.
In Fig. 14 illustrates how the number verifiers and repeated generations interact to influence success rate. We observe that increasing the number of generations tends to be more effective than increasing the number of verifiers alone but only when paired with the right verification strategy. For example, naive ensembling of verifiers plateaus in performance even more generations are added, whereas WEAVER continues to improve with both axes. This highlights that generation diversity is stronger driver of performance than verifier count alone, and that weak supervision methods like WEAVER are essential to fully leverage this diversity. We illustrate the verification generation tradeoff for additional datasets in Appendix C.3.
40

---

Model
GPQA Diamond
MATHS50
MMLU Pro
Average
Weaver with Lanna Verifier
59.4%
91.1%
87.4%
79.3%
Naive Ensemble
52.1%
88. 9%
84.1%
75.0%
Weaver Liama Verifiers Improvement %
+2.2%
+3.3%
Table 20: Performance Comparison of WEAVER with Llama Verifiers and Naive Ensemble with All Verifiers
Method
GPQA Diamond
MATH500
MMLU Pro
Dev Set Size
Average
Oracle Frontier Models
Oracle Verification (pass 100)
88.9
99.8
98.4
95.7
03-mini
78.3
96.2
85.5
86.7
GPT-4o
59.1
74.6
73.3
69.0
Unsupervised Baselines
Best-of-N (random)
48.7
82.8
70.6
0%
67.4
Majority Voting
51.2
88.2
79.4
0%
72.9
Seff-Verification
52.3
86.5
78.1
0%
72.3
Naive Ensemble (mean)
54.5
90.6
84.3
0%
76.5
Supervised Baselines 1.0x Dev Set (100% labeled)
Decision Tree
64.5
87.1
86.3
100%
79.3
AdaBoost
67.1
90.2
84.1
100%
80.5
Logistic Regression (LI))
73.1
92.5
90.2
100%
85.3
Logistic Regression (L2)
78.1
97.2
92.0
100%
89.1
Supervised Baselines 0.01x Dev Ser (1% labeled)
Decision Tree
52.1
69.4
71.0
1 16
64.2
AdaBoost
53.4
72.1
71.4
1%
65.6
Logistic Regression (1.2)
55.9
70.5
72.1
1%
66.2
Logistic Regression (L.I)*
54.1
72.4
73.5
1%
66.7
XGBoost
57.5
75.1
76.3
1%
69.6
Weaver (Ours)
71.1
93.4
90.2
1 %
84.9
Weaver-Distilled (400M)
66.4
91.1
87.8
1%
81.8
Table 21: Comparison of Verification Methods across Reasoning Benchmarks. WEAVER achieves strong performance with minimal labeled data (0.01x dev set) compared supervised methods that require 100x more labels.
AAdaBosst: esstimatorss (25, 75, 100, 1500, learning_rate (0.1, 1.0), max__epph (1,2), 2), selected via cross-validation.
'LI- Reg-lagulr == 0.01, selected cross- validatiii.
XGBooss: default parameters with num rounds=10, objective-binarylloistii.
C.6
WEAVER Distillation
RMM
Instruttion
Generation.
Instruction
RM,
Weaver Distilled LM
Weaver Distilled LM
Generation,
IM Generation
LM Judge
Ceneration,
Generation,
LM Judce,
Inference
Training
Figure 15: Overview of WEAVER Distillation (Section 6)
41

---

For the loss function in WEAVER distillation, we utilized crosspentropy loss with Adam [35]. Our classification architecture comprises a single linear classification layer with 0.1 dropout applied the input, which consists of the final hidden state from the [CT.S] token. Regarding learning dynamics, we implemented linear warmup and linear decay via the Sentence-Transform Transformers library [61], employing learning rate of 5e-6 and training batch size of 64 across all experimental setups.
Weaver: Distilled (0.48)
Weaver (70.08)
Majority Voting 10.08)
MAV (70.08)
Sel--Veriiiattio (70.08)
Pareto Region
MATH500
GPQA Diamond
75
Selection Accuracy (%)
92.5
Computo Savedd
90.0
99.97% Compute Saved
65
98,2%
87.5
97.4% Preserved
60
Gains Preserved
85.0
55
62.5
80.0
45
77.5
10
10
10
40
10a
10a
10a
MMLU
MMLU Pro
Selection Accuracy (%6
90
99,57% Compute Saved
92.5
99.97% Compute Saved
85-
97.5% of Accuracy Gains
80.0
Accuracy Gaina Preserved
87.5
75
85.0
82.5
70
80.0
10
10
10
65
10"
102
10a
Verification Compute Budget (ExaFLOPs)
Verification Compute Budget (ExaFLOPs)
Figure 16: WEAVER Distilled Pareto Frontiers: "We train/evaluate on an 80:20 split.
Table 22: Distillation Comparison of WEAVER and Naive Ensemble Across Different Training
Set Sizes
Training Set Percentage of Entire Dataset
Full
Methodology
Dataset
System
5%
10%
20%
50%
80%
MATH500
78.4%
80.7%
83.9%
88.2%
91.4%
93.4%
GPQA Diamond
42.6%
46.8%
52.7%
63.1%
71.8%
73.2%
WEAVER
MMLU College
83.5%
85.2%
87.6%
91.0%
93.1%
94.9%
MMLU Pro
69.2%
72.5%
76.8%
83.7%
87.8%
90.2%
MATH500
77.8%
79.6%
82.1%
86.4%
89.1%
92.4%
GPQA Diamond
42.1%
44.7%
48.9%
56.2%
62.8%
66.2%
NaiveEnsemble
MMLU College
84.0%
85.3%
87.2%
90.8%
93.5%
95.1%
MMLU Pro
69.5%
71.8%
74.9%
80.3%
84.7%
87.4%
42

---

C.7 Individual Verifier Optimization
While WEAVER primarily focuses on aggregating multiple weak verifiers to improve overall verification quality, this appendix explores complementary techniques for optimizing individual verifiers. As mentioned earlier in the paper, existing weak verifiers often suffer from high false positive rates [73], which can limit their effectiveness, even within ensemble.
As we scale the number of repeated samples and employ multiple verifiers, the precision of each individual verifier becomes increasingly important relative to recall. When many candidate solutions are available, a verifier can afford to miss some correct solutions (false negatives) as long as its positive predictions are highly reliable (high precision).
This observation motivates exploring methods to enhance individual verifier quality through methods such as prompt optimization tailoring verifier prompts to maximize performance, particularly precision, with minimal or no labeled data.
C.7.1 LM Judge Prompt Optimization
LM judges often suffer from biases such as position bias (favoring answers in certain positions), verbosity bias (preferring longer answers), and self-enhancement bias (preferring answers similar to their own generation patterns) [40, 95], suggesting sensitivity to system and input prompt design.
Throughout our WEAVER experiments, we used fixed, manually engineered prompts for our LM judge verifiers. However, optimizing these prompts could potentially improve individual verifier precision and reliability. Multi-AAent Verification [41] demonstrates this by crafting specialized prompts for specific verification aspects.
We explored systematically optimizing verifier prompts using DSPy [34], an open-source library that provides algorithms for optimizing language model prompts through discrete search over prompt candidates guided by a metric function. DSPy optimization works by generating, evaluating, and refining prompts that maximize task performance on small labeled dataset.
We investigate two dimensions of prompt optimization: (1) optimization space scaling, where we progressively expand what the optimizer can modify from system instruction only (O-shot) to including 3 demonstrations (3-shot) and 5 demonstrations (5-shot); and (2) training data size scaling, where we vary labeled data from 1% 16% to determine how much data is necessary for effective prompt optimization.
Experimental Setup:
Our experimental setup uses training examples containing datasets have multiple generations per instruction (up to 100), we group examples by instruction before splitting to prevent data leakage between train and validation sets. We hold out 50% of the dataset instructions (each paired with 100 candidate generations) for evaluation. For the optimization space scaling experiment, we randomly select n generations such that x len(dataset) 250, maximizing training set diversity while maintaining fixed training set size. For the data scaling experiment, we train on different percentages (1%, 2%, 4%, and 16%) of the dataset by calculating the number of instructions as [num_ Baauun__ooeem______ in__datast x perrennpereettgee// 100)] and selecting repeated samples for each instruction with samples- = min (mam(4 (o.xx(4,nm (4, aauum_problemee x 20) avoid overfitting. We use a consistent random seed to ensure identical dataset splits between optimization runs.
pairs. Since
Figure 17 shows results across different datasets and optimization configurations. While don't observe clear scaling relationships across all datasets (possibly due to the increased stochasticity of L.Ld-based optimization), we observe an average precision gain of 3.8% of the best judge over the chain-of-thoughh (CoT) baseline judge. MATH500 shows the largest jump in precision of 9% and shows clear improvement in precision as the optimization space is scaled.
Results:
The scaling behavior with training data size (Figure 18) shows slight log-linear improvements in precision as we increase training data, though gains differ by dataset. MMLU-Colee shows minimal benefit from additional data, while the remaining datasets see an average boost of 3.2% in precision when scaling the training data size from 1% to 16% of the original dataset.
(Figure 19) reveals that optimized prompts often improve both precision and accuracy by reducing false positive rates essentially making judges more conservative in their correctness assessments. This is particularly valuable in the repeated sampling regime, where higher precision improves overall verification quality.
43

---

067
0.57
Oprinized Optiniced (3ohhc))
Optimized shob
0.54
0.50 952
Precision
0.32 A33
0.10
63
00
GPQA
College
Figure 17: judge prompt optimizatio using 250 labeled examples consistently yields precision gains. Baseline methods (CoT and Custom) are compared against DSPy-optimized prompts with varying numbers of demonstrations (O-shot, 3-shot, and 5-shot).
AIMO
GPOA
MMLUU College
MMLU Pro
0.5
Precision
0.4
10a
10a
Training Data Percentage
Figure 18: Scaling LM judge prompt optimization training data leads modest precision gains. The x-axis shows the percentage of training data used (log scale), and the y-axis shows precision.
These findings suggest that prompt optimization can be a valuable complement to WEAVER's aggrega- tion approach. Even with limited labeled data, targeted prompt engineering can enhance individual verifier quality, benefiting the ensemble as whole. Further research is needed to define a more systematic recipe for verifier prompt optimization. Additionally, it remains question of whether can extend prompt optimization to discriminative reward models to enjoy similar gains in performance.
D Miscellaneous
D.1 Compute Requirements
Hardware Infrastructure. Our experiments were conducted using 4 compute nodes, each equipped with NVIDIA H100 GPUs (80GB HBM3 memory per GPU), for a total of 32 H100 GPUs. Each node configured with high-bandwidh NVLink connections between GPUs and inter-node commu- nication was facilitated via NVIDIA NVLink Switch System to minimize communication overhead during distributed training and inference.
Model Parallelism and Distribution. our 72B parameter language models, we employed hybrid parallelism strategy combining tensor parallclism, pipeline parallelism, and data parallelism:
44

---

FPR Reduction vs Precision Gain
FPR Reduction vs Accuracy Gain
AMO
AIMO
GPOA
SPQA
MATH 300
MATH 500
MMLU College
MMLU College
Precision Gain
Accuracy Gain
3.4
0.04
2.3
0.02
0.00
93
-0.02
0.0
0.2
0.
0.2
0.1
C.6
Reduction False Positive Rate
Reduction in False Positive Rate
Figure 19: Optimized prompts often improve LM judge performance by reducing false positive rates.
8 way tensor parallelism across GPUs within each node
4 way pipeline parallelism across nodes
Data parallelism for batch processing
Storage Requirements. Processing datasets of 100GB+ required significant storage infrastructure:
4TB NVMe SSDs per node for dataset caching and checkpoints
100TB shared network storage for full dataset repository
Software Stack. Our experiments were powered by:
NVIDIA CUDA 12.2
Py'oorch 2.1 with NVIDIA NCCL for distributed communication
DeepSpeed ZeRO Stage 3 for memory optimization
Distributed data loading with webdataset format for efficient streaming
45


Reviews:
[{"review_id": "Sgi8klJi5R", "reviewer": "Reviewer_XeAb", "summary": "This paper tackles the challenge of verifying and selecting correct responses from a language model’s outputs in order to improve the model’s effective performance. The authors propose WEAVER, a framework that combines multiple weak verifiers (such as language-model-based judges and reward models) to approximate a much stronger verification signal without requiring any labeled training data. Empirically, WEAVER leads to large gains in downstream task accuracy. To address the computational cost of using many verifiers, the authors further distill the ensemble’s behavior into a single 400M-parameter cross-encoder model. This distilled verifier retains about 98.7% of the full ensemble’s accuracy while reducing verification compute cost by ~99.97%, making the overall approach far more efficient. Overall, the paper’s contributions include a novel unlabeled ensemble method for response verification (WEAVER), a demonstration of significantly improved selection of correct answers (shrinking the generation-verification gap), and a practical distillation step to save runtime cost.", "questions": "See weakness.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n*  WEAVER significantly outperforms standard baselines (e.g., majority vote, self-verification) and nearly matches state-of-the-art models like GPT-4o and o3-mini, despite using a cheaper generator.\n\n* WEAVER combines multiple weak verifiers using weak supervision techniques without requiring labeled ground-truth data, which is a practical and scalable contribution.\n\n* The distillation of WEAVER into a lightweight 400M cross-encoder retains ~98.7% accuracy while reducing inference compute by 99.97%, enabling real-world applicability.\n\n* Applies latent variable modeling to verifier combination in a new setting, overcoming output inconsistencies and low-quality verifiers.\n\nWeakness:\n* Performance depends on having access to a broad and complementary pool of weak verifiers.\n\n* WEAVER is most useful in repeated sampling scenarios; its gains are less relevant where only a single response can be generated.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of Weaver's significant performance improvements and your characterization of  the approach as practical and scalable. \n\nWe address your two concerns regarding limitations related to verifier diversity and applicability below, by including an extended discussion on the growing role of repeated sampling in real-world applications, and a new set of experiments that evaluate Weaver’s robustness under reduced verifier diversity and its effectiveness in low-sample regimes. We plan to incorporate these additions into the final version of the paper and hope you will consider updating your score in light of these improvements.\n\n**W1: \"Performance depends on having access to a broad and complementary pool of weak verifiers.\"**\n\nWeaver offers performance improvements over naive ensembling even when the pool of verifiers is not broad or complementary. Below, we provide **1)** results from the paper using smaller sets of verifiers **2)** new experiments using a set of more correlated verifiers from the same model family, and **3)** context on why broad, diverse verifier pools are becoming increasingly accessible in practice. \n\n- **(1) Weaver performs well even with few verifiers.** Figure 4 shows how Weaver performs as we vary the number of verifiers from 1 to 15, a significantly smaller set than the 33 used in our main experiments. Even with just the top-3 verifiers, Weaver outperforms naive ensembling on GPQA, MATH500, and MMLU Pro. This demonstrates that a broad pool of verifiers is not necessary for Weaver to be effective.\n\n- **(2) Weaver is robust to correlated verifiers.** To test Weaver’s performance on a non-complementary set of verifiers, we constructed a pool of 13 verifiers all derived from the Llama family (e.g., Llama 3.1 8B/70B variants), fine-tuned with similar RLHF data and objectives. These verifiers are highly correlated, with an average pairwise Pearson correlation of 0.491. As shown in Table 1 below, Weaver achieves a +4.3% average improvement over naive ensembling—only modestly below the +6.8% improvement we observe with the full, more diverse pool (Table 2; average correlation 0.281). These results indicate that while diversity enhances performance, Weaver remains effective even when verifiers are highly redundant. We will include the two tables below in the updated draft.\n\n- **(3) Having access to a broad, diverse set of verifiers is increasingly common in practice.** While we have shown above that Weaver does not require such verifier pools, the ecosystem of pre-trained models and tools that can be used for verification continues to grow in scale and variety. Figure 7 of our paper shows that there are now over 150 reward models and 250 LLM judges currently available. Furthermore, new community leaderboards continue to emerge (e.g., RewardBench V2: https://huggingface.co/spaces/allenai/reward-bench and ChatBotArena: https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leaderboard), offering us access to the newest high-quality reward models and judges. \n\n**Table 1 (New Results: Weaver with Llama Verifiers):**  \n| Model | GPQA Diamond | MATH500 | MMLU Pro | Average |\n|:---|:---:|:---:|:---:|:---:|\n| Weaver with Llama Verifier | 59.4% | 91.1% | 87.4% | 79.3% |\n| Naive Ensemble | 52.1% | 88.9% | 84.1% | 75.0% |\n| Weaver with Llama Verifiers - Improvement % | +8.3% | +2.2% | +3.3% | +4.3% |\n\n**Table 2 (Existing Results: Weaver with All Verifiers, Currently Table 1 and Figure 3):**  \n| Model | GPQA Diamond | MATH500 | MMLU Pro | Average |\n|:---|:---:|:---:|:---:|:---:|\n| Weaver | 66.4% | 93.4% | 90.2% | 83.3% |\n| Naive Ensemble | 54.5% | 90.6% | 84.3% | 76.5% |\n| Weaver Improvement % | +11.9% | +2.8% | +5.9% | +6.8% |\n\n**W2: \"Weaver is most useful in repeated sampling scenarios; its gains are less relevant where only a single response can be generated.\"**\n\nWe agree that Weaver is primarily designed for the repeated sampling setting; we will clarify its scope in the discussion section. However, we note that 1) Weaver still provides advantages in limited sampling regimes, 2) repeated sampling is increasingly common in practice, and 3) repeated sampling is becoming increasingly practical to deploy.\n\n- **(1) Weaver still provides advantages in limited sampling regimes.** Specifically, Weaver improves the selection accuracy even when only a few (e.g., 2–4) responses are available, as shown in Figure 3 of the paper.\n- **(2) Repeated sampling is increasingly common in practice,** driven by its effectiveness and by recent findings that scaling test-time compute (e.g., sampling more outputs) can yield gains comparable to training larger models [1, 2, 3]. This has led to broader adoption of repeated sampling across high-value applications:\n\n   - *Inference-time use cases*:\n       - Mathematical reasoning and theorem proving (e.g., MiniF2F, MATH datasets) [9]\n       - Scientific question answering (e.g., GPQA for expert-level queries) [10]\n       - Business, engineering, and humanities tasks requiring multi-hop reasoning (e.g., MMLU-Pro) [11] \n       - Code generation and debugging (e.g., LiveCodeBench) [12]\n       - Creative writing or content generation where diversity improves quality (e.g. ArenaHardAuto) [5]\n       - Deployed assistants and production systems (e.g., GPT-4o, Claude, Gemini), which internally sample and verify multiple responses before emitting a final output to improve safety and reliability. [13]\n   - *Training-time workflows*:\n       - Reinforcement learning (GRPO / DPO) algorithms requiring 16+ generations per query with associated scores [6, 7]\n       - Data filtering and synthetic data creation for model training [4] \n\n- **(3) Repeated sampling is becoming increasingly practical to deploy.** Several trends support the growing viability of repeated sampling in real-world systems. Recent ML systems work has made repeated sampling more compute and memory efficient. Inference engines like Tokasaurus, vLLM, and SGLang [8, 14,15] achieve 2-6x throughput gains through dynamic prefix sharing, optimized memory management with PagedAttention, and structured execution with RadixAttention. FlexGen [16] and Sarathi-Serve [17] enable high-throughput inference on constrained hardware, though they may require tailored programming or larger batch sizes to maximize efficiency. These advancements allow real-world systems to handle high-throughput language model calls with greater efficiency, making repeated sampling increasingly practical for deployment.\n\n\n[1] Schaeffer, Rylan, et al. \"How Do Large Language Monkeys Get Their Power (Laws)?.\" arXiv preprint arXiv:2502.17578 (2025).\n\n[2] Brown, Bradley, et al. \"Large language monkeys: Scaling inference compute with repeated sampling.\" arXiv preprint arXiv:2407.21787 (2024).\n\n[3] Snell, Charlie, et al. \"Scaling llm test-time compute optimally can be more effective than scaling model parameters.\" arXiv preprint arXiv:2408.03314 (2024).\n\n[4] Guha, Etash, et al. \"OpenThoughts: Data Recipes for Reasoning Models.\" arXiv preprint arXiv:2506.04178 (2025)\n\n[5] Li, Tianle, et al. \"From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.\" arXiv preprint arXiv:2406.11939 (2024).\n\n[6] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" Advances in neural information processing systems 36 (2023): 53728-53741.\n\n[7] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).\n\n[8] Juravsky, Jordan, et al. \"Tokasaurus: An LLM Inference Engine for High-Throughput Workloads.\" Scaling Intelligence, 2025, scalingintelligence.stanford.edu/blogs/tokasaurus/.\n\n[9] Zheng, Kunhao, Jesse Michael Han, and Stanislas Polu. \"Minif2f: a cross-system benchmark for formal olympiad-level mathematics.\" arXiv preprint arXiv:2109.00110 (2021).\n\n[10] Rein, David, et al. \"Gpqa: A graduate-level google-proof q&a benchmark.\" First Conference on Language Modeling. 2024.\n\n[11] Wang, Yubo, et al. \"Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.\" Advances in Neural Information Processing Systems 37 (2024): 95266-95290.\n\n[12] Jain, Naman, et al. \"Livecodebench: Holistic and contamination free evaluation of large language models for code.\" arXiv preprint arXiv:2403.07974 (2024).\n\n[13] Team, Gemini, et al. \"Gemini: a family of highly capable multimodal models.\" arXiv preprint arXiv:2312.11805 (2023).\n\n[14] Kwon, Woosuk, et al. \"Efficient Memory Management for Large Language Model Serving with PagedAttention.\" arXiv preprint arXiv:2309.06180 (2023).\n\n[15] Zheng, Lianmin, et al. \"SGLang: Efficient Execution of Structured Language Model Programs.\" arXiv preprint arXiv:2312.07104 (2023).\n\n[16] Sheng, Ying, et al. \"FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU.\" arXiv preprint arXiv:2303.06865 (2023).\n\n[17] Agrawal, Amey, et al. \"Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve.\" arXiv preprint arXiv:2403.02310 (2024)."}, {"review_id": "V53lIepcuF", "reviewer": "Reviewer_xhJS", "summary": "Verification of responses to instructions has several valuable use cases, but existing verifiers are noisy. This paper explores how to combine multiple weak verifiers in the experimental setting of improving response selection under repeated sampling. There are three key contributions:\n\n1. Given a large corpus of labeled query-response pairs, they show that using logistic regression to learn verifier weights outperforms naive averaging of verifiers or choosing the top verifier by a large margin.\n2. To handle data-limited settings, they introduce WEAVER, which adapts weak supervision to build a latent variable model to estimate verifier weights.\n3. They show that the WEAVER ensemble can be distilled into a 400M cross-encoder verifier, which retains 99% of the performance while using only 0.03% of the inference FLOPS.", "questions": "1. In §4.1 and §5 (the main experiments), which dataset is used for learning the verifier weights? How much labeled training data does WEAVER use?\n2. Should the x-axis label in Figure 4 be \"Number of verifiers\" instead of \"Top-k Verifiers\"?\n3. What are the frontier approaches in Table 1?\n4. What questions do you use for training WEAVER-Distilled? I.e., do you train on WEAVER's pseudolabels for the same questions that you evaluate on?\n5. Why do you think that a small, distilled verifier performs so well? Would this verifier perform well on RewardBench?\n\n**Other suggestions:**\n- In Table 1, it may be a good idea to add a column for the number of verifiers. E.g., it would be 0, 0, 1, 10 for the first 4 rows, and then 33 for WEAVER. \n- Line 294: remove comma after \"Figure 6\"\n- Line 324: \"combining many weak verifier\" $\\rightarrow$ \"combining many weak verifiers\"", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n1. The empirical results are strong. WEAVER achieves significant improvements over existing verification methods for identifying the best sample from repeated generations (Table 1), and scales better with test-time compute (Figure 5). The authors provide analysis on how WEAVER scales along many dimensions, including the number of samples, the model size, the number of verifiers, and inference compute.\n2. The problem studied is of great interest to the community. Better verifiers have many applications (as discussed by the authors), such as data filtering, alignment, and test-time compute scaling.\n\n### Weaknesses:\n1. Another baseline that I would like to see is the approach considered in §4.1, where linear regression using a large amount of training data is used to learn weights over the verifiers. While this approach requires more data, it paints a better picture of how much performance is preserved by WEAVER in the data-constrained regime.\n2. What is the motivation for studying the data-constrained setting? I believe there is an abundance of instruction-response pairs for learning weighted ensembles of verifiers.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions of our work, including the importance of improved verification under repeated sampling. We appreciate your acknowledgement of Weaver’s strong empirical performance, scalability, and its practical utility for alignment and test-time compute efficiency.\n\nBelow, we address your concerns on: **W1)** adding a supervised regression baseline trained with abundant labels to contextualize Weaver’s performance and **W2)** clarifying our focus on the data-constrained regime despite the existence of large instruction-response datasets. We also answer each of your questions and will incorporate your helpful presentation feedback into the final version. In addition, we now include **new results** evaluating Weaver-distilled on RewardBenchv2. We hope this fully addresses your concerns and would be grateful if you would consider updating your score based on these improvements.\n\n**W1:   Linear regression with abundant data as a baseline**\n- We included a supervised Logistic Regression (LR) baseline as motivation in Figure 2, and a comparative benchmark in Tables 4 and 15, where we vary the amount of labeled data for both Weaver and LR. These results show that Weaver’s performance comes close to Logistic Regression when we have large amounts of training data (i.e. with 1.0x of the labeled development set), but more importantly, Weaver outperforms Logistic Regression in label-scarce regimes (i.e., with 0.01x of the dev set). This advantage stems from Weaver’s reliance on labeled data only for calibrating parameters such as class balance and binarization thresholds—unlike LR, which learns a model directly from the labels. **We will update our paper to more clearly display the comparison between these methods at varying amounts of labeled data.**  \n\n**W2: \"What is the motivation for studying the data-constrained setting? I believe there is an abundance of instruction-response pairs for learning weighted ensembles of verifiers.\"**\n- We agree that large instruction-response datasets exist in general domains. However, in many high-value tasks, such as expert-level reasoning (e.g., GPQA Diamond), advanced mathematics (e.g., MATH-500), or emerging domains, high-quality labels are scarce and costly, often requiring domain experts (e.g., PhD-level annotators for GPQA). In these label-scarce settings, supervised methods like logistic regression and Naive Bayes degrade significantly as shown in Table 15. Weaver is designed for precisely this regime: it learns to aggregate verifiers using unlabeled responses, making it well-suited for real-world applications where labeled data is limited. **We will clarify this motivation in Sections 1 and 4.2, citing examples like GPQA’s annotation costs [1].**\n\n**Q1: \"In Section 4.1 and Section 5 (the main experiments), which dataset is used for learning the verifier weights? How much labeled training data does WEAVER use?\"**\n- In both Section 4.1 and Section 5, the labeled data comes from the same benchmark datasets under evaluation—**MATH-500, GPQA Diamond, MMLU Pro, and MMLU College.** In Section 4.1, we use 100% of the labeled data from these datasets to learn verifier weights. As noted in the caption of Figure 2, this oracle baseline illustrates the best possible performance when full label access is available. In section 5, we shift to the data-constrained setting, and use only 1% of the data, a dev split from the same benchmark. \n- Tables 4 and 15 report the performance across a range of training set sizes (from 0.01× to 1.0×) for additional baselines including, Logistic Regression and Naive Bayes, to illustrate the impact of data availability across different approaches. We note that Weaver does not use labeled data for training. Instead, it fits moment-based estimates on the unlabeled set of query-response pairs, $D^{\\text{test}}$.  A small dev set (1% of the data) is used only for auxiliary calibration – specifically to set the reward thresholds, filter noisy verifiers and estimate class balance (as detailed in Appendix B.2.2). In contrast, supervised baselines like Logistic Regression use this same 1% dev set to directly fit model parameters, making their performance more sensitive to the amount and quality of labeled data. Weaver, by comparison, learns its aggregation weights entirely from unlabeled data and remains robust even when only a handful of labeled examples are available. **In Table 4, we ablate the dev set size and find that Weaver maintains strong performance even with ~5-50 examples across datasets.** \n\n**Q2: \"Should the x-axis label in Figure 4 be 'Number of verifiers' instead of 'Top-k Verifiers'?\"**\n- Thank you for this clarification question. We label the x-axis as \"Top-k Verifiers\" rather than \"Number of Verifiers\" to reflect the specific methodology used in our evaluation. Instead of arbitrarily increasing the number of verifiers, we add them in descending order of individual performance, from the strongest to the weakest, in an oracle setting where we know the ranked performances of the verifiers a priori. This approach allows us to assess Weaver's performance as we incorporate progressively lower-quality verifiers into the ensemble, isolating the effect of verifier strength and eliminating noise from randomly scaling the verifier pool. We will revise the caption to clarify this setup and make the evaluation methodology more explicit.\n   - **Updated Caption**: \"Weaver Benefits from Adding Top-K Verifiers: In an oracle setting where we know the top performing verifiers for a benchmark, we find Weaver can improve verifier ensembling beyond naive ensembling even as lower-quality verifiers are added, leading to an average improvement of 8.5% across GPQA, MATH500, and MMLU Pro.\"\n\n**Q3: \"What are the frontier approaches in Table 1?\"**\n- The \"Frontier LM\" rows in Table 1 refer to state-of-the-art closed-source models like o3-mini (OpenAI's latest reasoning model), evaluated on the same tasks as Weaver. We replicate their officially reported results and include them in the table with their citations. We also include Oracle Verification (pass@100), which involves always selecting the correct generation—this is the upper bound for any verification system such as Weaver (see Section 3 “Evaluation Metrics” for more information).\n\n**Q4: \"What questions do you use for training WEAVER-Distilled? I.e., do you train on WEAVER's pseudolabels for the same questions that you evaluate on?\"**\n- We train Weaver-Distilled on pseudolabels generated by Weaver using an 80% held-out development split of query-response pairs, with no overlap with the 20% evaluation set. As shown in Figure 16 and Appendix C.6, we use cross-entropy loss and preserve 98.7% of Weaver’s accuracy. This enables us to deploy the distilled model on **unseen queries** without needing to run the full Weaver ensemble at inference time. **Full training details (e.g., Adam optimizer, learning rate 3e-6, batch size 64) are provided in Appendix C.6.**\n\n**Q5: \"Why do you think that a small, distilled verifier performs so well? Would this verifier perform well on RewardBench?\"**\n- **Weaver-distilled preserves 98.7% of the original Weaver model's accuracy on unseen queries** by leveraging the full textual context of query-response pairs, unlike Weaver, which relies on aggregating binary verifier scores. Trained directly on these pairs, the distilled cross-encoder captures nuanced patterns of correctness through richer input representations, effectively approximating the verifier ensemble's decision boundary. This approach aligns with the strengths of cross-encoders for pairwise scoring tasks like reward modeling, as they efficiently distill large language model knowledge into precise classifiers, avoiding generation biases and reinforcement learning instabilities observed in fine-tuning smaller generative models [2, 3, 4]. By drawing upon earlier work on employing cross-encoders in related classification tasks—such as sentence pair similarity scoring, information retrieval re-ranking, semantic search, pair classification, recommendation systems, and semantic textual relatedness—our method extends these proven techniques to enhance accuracy and efficiency in reward modeling and verification.\n- Per your suggestion, **we further evaluate Weaver distilled on RewardBench V2.** As shown below, we see that it achieves good performance and a top-50 position. Note that Weaver distilled is trained without having access to ground truth labels (unlike other verifiers in RewardBench) yet **it achieves 69.13% on the \"Math\" subset, a broader dataset compared to MATH-500.** We will include these additional results in Appendix C.9.\n\n|Model|Model Type|Score|Factuality|Precise IF|Math|Safety|Focus|Ties|\n|---|---|---|---|---|---|---|---|---|\n|Skywork/Skywork-Reward-V2-Llama-3.1-8B (#1)|Seq. Classifier|84.13|84.63|66.25|77.60|96.67|98.38|81.24|\n|allenai/Llama-3.1-Tulu-3-70B-SFT-RM-RB2 (#20)|Seq. Classifier|72.20|80.84|36.88|67.76|86.89|77.78|83.08|\n|anthropic/claude-3-opus-20240229 (#50)|Generative|57.44|53.89|33.13|51.37|83.78|66.46|56.01|\n|**Weaver Distilled (Top-50)**|Seq. Classifier|**58.27**|57.91|41.76|69.13|51.29|67.84|61.71|\n\n\n[1] Rein, David, et al. \"Gpqa: A graduate-level google-proof q&a benchmark.\" First Conference on Language Modeling. 2024.\n\n[2] Reimers, Nils, et al. \"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.\" EMNLP. 2019.\n\n[3] Nogueira, Rodrigo, et al. \"Passage Re-ranking with BERT.\" arXiv preprint arXiv:1901.04085. 2019.\n\n[4] Lu, Meng, et al. \"Cross-Encoder Rediscovers a Semantic Variant of BM25.\" arXiv preprint arXiv:2502.04645. 2025."}, {"review_id": "auY59bHKIX", "reviewer": "Reviewer_jVww", "summary": "This paper proposes WEAVER: an unsupervised probabilistic model that learns per-verifier weights to fuse dozens of weak LLM judges.  Applied to 100-sample decoding, WEAVER boosts pass\\@1 accuracy over the base model and also over majority vote on four reasoning benchmarks, letting a 70 B generator rival GPT-4. Furthermore, the authors distill a 400 M distilled cross-encoder retains ≈99 % of the gain while slashing verification Flops.", "questions": "Same as the weakness section.\nIf the authors could solve my questions, I will raise my score.\n\n1. How sensitive is WEAVER to the diversity and independence of verifiers, particularly if reward models or judges are trained on similar data or architectures? If the verifier models are similar, will this lead to performance degradation?\n2. Has WEAVER been evaluated on tasks beyond math QA, such as code generation or subjective reasoning? Are there known challenges or adaptation requirements for those domains?\n3. Can the overall process be interpreted as learning weighted coefficients from a small subset using traditional machine learning methods? If so, would tree-based models be applicable?\n4. Does the effectiveness of the ensemble method rely on the small subset and test set being drawn from the same distribution? If not, would the learned coefficients become unreliable?\n5. For the distillation method, if a small LLM is fine-tuned on a specific dataset, would it achieve results comparable to those reported in the paper?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The motivation is clear and well-written.\n2. The results show significant improvements across four benchmarks.\n3. The distillation method significantly reduces computational cost while maintaining strong performance.\n\n## Weaknesses\n\n1. How sensitive is WEAVER to the diversity and independence of verifiers, particularly if reward models or judges are trained on similar data or architectures? If the verifier models are similar, will this lead to performance degradation?\n2. Has WEAVER been evaluated on tasks beyond math QA, such as code generation or subjective reasoning? Are there known challenges or adaptation requirements for those domains?\n3. Can the overall process be interpreted as learning weighted coefficients from a small subset using traditional machine learning methods? If so, would tree-based models be applicable?\n4. Does the effectiveness of the ensemble method rely on the small subset and test set being drawn from the same distribution? If not, would the learned coefficients become unreliable?\n5. For the distillation method, if a small LLM is fine-tuned on a specific dataset, would it achieve results comparable to those reported in the paper?", "was_revised": true, "final_justification": "After reading this paper, my biggest concern is the authors use the test set (benchmark) to split a dev set. I worry this will involve the test set leakage. However, during the discussion phase, the authors have solved this question by utilizing training sets to make their dev sets.\n\nHence, I give a positive score.", "rebuttal": "We thank you for your thoughtful feedback and recognition of Weaver's clear motivation, significant improvements across benchmarks, and effective distillation for compute reduction. \n\nBelow, we address your questions regarding verifier diversity, domain generalization, interpretability of the aggregation method, distribution shift robustness, and the effectiveness of our distillation strategy. We supplemented our original results with **five new experiments**, each designed to directly address one of your concerns. We believe these additions significantly strengthen the paper, and we would greatly appreciate your consideration in updating your score based on these improvements.\n\n**Q1: \"How sensitive is WEAVER to verifier diversity/independence, especially if models are trained on similar data?\"**\n- Weaver is designed to leverage diversity among verifiers, but it offers performance improvements over naive ensembling even when the verifiers used are similar. **To examine this, we add the following experiment:** we subsampled 13 of the 33 verifiers from our main experiments, all fine-tuned versions of Llama 8B/70B family models. These verifiers are more correlated than the full, more diverse pool (average Pearson correlation of 0.491 vs 0.281, respectively). As shown in the table below, Weaver achieves a +4.3% average improvement over naive ensembling—only modestly below the +6.8% improvement we observe with the full pool (Table 2). These results indicate that while diversity enhances performance, Weaver remains effective even when verifiers are similar. We will include the two tables below in the updated draft.\n\n**Table 1: New Results - Weaver with Llama Verifiers**\n|Model|GPQA|MATH500|MMLU Pro|Average|\n|---|---|---|---|---|\n|Weaver with Llama Verifiers|59.4%|91.1%|87.4%|79.3%|\n|Naive Ensemble|52.1%|88.9%|84.1%|75.0%|\n|Weaver with Llama - Improvement %|+8.3%|+2.2%|+3.3%|+4.3%|\n\n**Table 2: Existing Results - Full Weaver with All Verifiers**\n|Model|GPQA|MATH500|MMLU Pro|Average|\n|---|---|---|---|---|\n|Weaver|66.4%|93.4%|90.2%|83.3%|\n|Naive Ensemble|54.5%|90.6%|84.3%|76.5%|\n|Weaver Improvement %|+11.9%|+2.8%|+5.9%|+6.8%|\n\n**Q2: \"Has WEAVER been evaluated beyond math QA (e.g., code generation, subjective reasoning)? What are the associated challenges and adaptations, if any?\"**\n- Per your suggestion, **we add the following experiment:** we evaluate Weaver on ArenaHard Auto [5], a benchmark that emphasizes subjective and open-ended reasoning to produce a response. As shown below, Weaver performs well in this setting, without any domain-specific adaptation.\n- In our paper, we also evaluated Weaver across a diverse set of reasoning tasks beyond math QA, including **multi-hop reasoning (MMLU Pro/College) [2, 3], PhD-level science (GPQA) [1], and commonsense reasoning (BBH) [4].** Weaver’s framework is **domain-agnostic**, relying solely on verifier outputs rather than task-specific features. \n- **ArenaHard Auto Results:** ArenaHard Auto [5] is an automatic evaluation benchmark focused on challenging real-world user queries requiring subjective reasoning and creative generation. For this new task, we focus on Weaver's performance compared to supervised logistic regression (LR), where Weaver achieves 85.1%—a 9.0% absolute improvement over the 76.1% of the supervised LR baseline. Furthermore, Weaver outperforms frontier LMs, surpassing models like o3-mini (55.6%), Claude 3.7 Sonnet (63.9%), and Llama 4 Maverick (10.5%). We include the complete results in a table below and plan to add this result to our revised version:\n\n| Model | ArenaHard Auto Score |\n|:---:|:---:|\n| Weaver | 85.1% |\n| Weaver Distilled | 81.6% |\n| Supervised LR | 76.1% |\n| o3-mini | 55.6% |\n| Claude 3.7 Sonnet | 63.9% |\n| Llama 4 Maverick | 10.5% |\n\n**Q3: \"Is this just learning weighted coefficients via traditional ML? Would tree-based models work?\"**\n- Not exactly. You're right that Weaver aims to learn weighted coefficients over verifier scores, but it does so in a fundamentally different way from traditional supervised learning methods. \n- Traditional supervised learning methods—such as logistic regression/Naive Bayes (Table 15) or tree-based models—can be effective when sufficient labeled data is available. However, Weaver is designed for the more challenging setting **where labeled data is scarce**. It fits a latent variable model to the verifier scores (Section 4.2.1). This model treats the true correctness labels as latent and, by modeling the relationship between verifier outputs and these unobserved labels, learns both the accuracy of each verifier and the optimal way to aggregate their output, entirely from unlabeled data. In Weaver, the small labeled development set is not used for training but rather for calibration—specifically, to estimate the class balance Pr(y=1), making Weaver more label-efficient than supervised alternatives.\n- Below, we discuss some results on **1)** Weaver versus traditional supervised learning methods, and **2)** performance of the supervised tree-based methods you suggested: \n    - **1)** From Tables 15 and 4, supervised logistic regression performs better than Weaver when trained on the full development set (e.g., 97.2% vs. 92.4% on  MATH-500 using all 500 samples), but Weaver clearly outperforms it when labeled data is limited (e.g., 70.5% vs. 90.4% when using 0.01x of the dev set). \n    - **2)** Following your suggestion, **we also include additional experiments:** we find that tree-based models (AdaBoost, Decision Tree, 1.0x dev set) underperformed Logistic Regression (1.0x dev set) by -7.5% and Weaver (0.01x dev set) by -3.4% on average. We will include these results in the revised draft.\n\n| Model | GPQA Diamond | MATH500 | MMLU Pro | Average |\n|:---|:---:|:---:|:---:|:---:|\n| Supervised LR (1.0x Dev Set) | 72.9% | 97.2% | 92.0% | **87.4%** |\n| Supervised AdaBoost (1.0x Dev Set) | 67.1% | 90.2% | 84.1% | **80.5%** |\n| Supervised DecisionTree (1.0x Dev Set) | 64.5% | 87.1% | 86.3% | **79.3%** |\n| Supervised LR (0.01x Dev Set) | 55.9% | 70.5% | 72.1% | **66.2%** |\n| Weaver (0.01x Dev Set) | 66.4% | 93.4% | 90.2% | **83.3%** |\n\n**Q4: \"Does effectiveness rely on dev/test sets being from the same distribution?”**\n- Like most traditional ML methods, Weaver assumes the development set is representative of the test distribution. However, even when the estimated class balance is wrong, the pairwise agreement statistics among verifiers remain largely unchanged, and moment-matching can still help us recover sensible estimates of each verifier’s true positive and false positive rates. To evaluate the sensitivity of Weaver, **we add the following experiment:** We sweep over class balance values to evaluate robustness. Specifically, we vary the assumed Pr⁡(y=1) from 0.1 to 0.9, inclusive, and observe that performance degrades gracefully, indicating that **Weaver is not overly sensitive to mild miscalibration** (i.e. range of 0.3 above/below the ground truth class balance).  This experiment directly tests the impact of a mismatch between the development and test distributions, and suggests that Weaver’s aggregation procedure is resilient to moderate deviations. We will add the table below to Appendix C.3, where the bolded values correspond to the estimated class balance from the dev set:\n\n**Weaver Accuracy across Dataset Distributions - Sensitivity to Class Prior**\n| | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 |\n|---|---|---|---|---|---|---|---|---|---|\n| GPQA Diamond | 59.0% | 60.8% | 64.0% | **66.4%** | 64.1% | 62.5% | 61.0% | 50.2% | 42.7% |\n| MATH500 | 54.2% | 71.5% | 85.1% | 90.6% | 91.7% | 92.6% | 93.1% | **93.4%** | 93.0% |\n| MMLU Pro | 56.3% | 70.1% | 85.4% | 87.1% | 88.0% | 89.5% | **90.2%** | 89.7% | 88.2% |\n\n**Q5: \"Would fine-tuning a small LLM on the dataset achieve comparable results?\"**\n- Following your suggestion, **we add the following experiment:** we post-train small LMs (e.g., Qwen 4B, Llama 3B, and Llama 8B) using GRPO and DPO on generations from our Llama 70B model. However, due to off-policy RL issues (e.g., distribution mismatch between generations and the small model's capabilities [9, 10]), we observed only modest gains in sampling accuracy (e.g., 4-6% improvements), whereas **the distilled cross-encoder delivered +15% gains on average.** \n- We will add the supervised post-training results below (including Qwen 4B and Llama 3B) to the manuscript and discuss the findings of these RL baselines.\n\n| Post-Trained Generator | Generations Source | Post-Training Approach | GPQA Diamond | MATH500 | MMLU Pro | Average |\n|---|---|---|---|---|---|---|\n| Llama 3.1 8B (Majority Voting) (Baseline) | N/A | N/A | 30.5% | 69.0% | 56.4% | 52.0% |\n| Llama 3.1 8B | Llama 70B | DPO | 35.1% | 77.4% | 61.3% | 58.0% |\n| Llama 3.1 8B | Llama 70B | GRPO | 36.4% | 76.7% | 62.4% | 58.5% |\n| Weaver (Baseline) | Llama 8B | N/A | 47.1% | 80.0% | 67.2% | 64.8% |\n\n[1] https://arxiv.org/abs/2311.12022\n\n[2] https://arxiv.org/abs/2406.01574\n\n[3] https://arxiv.org/abs/2009.03300\n\n[4] https://arxiv.org/abs/2210.09261\n\n[5] https://arxiv.org/abs/2406.11939\n\n[6] https://arxiv.org/abs/1908.10084\n\n[7] https://arxiv.org/abs/1901.04085\n\n[8] https://arxiv.org/abs/2502.04645\n\n[9] https://arxiv.org/abs/2402.03300\n\n[10] https://arxiv.org/abs/2503.14286"}, {"review_id": "9T5f6twlqL", "reviewer": "Reviewer_cAMM", "summary": "The authors introduce Weaver, a method for improving verification of LLM outputs. This method, given N \"verifiers\" (or \"reward models\") and a development set of labeled examples, assigns weights for each of the verifiers, which are used to aggregate the verifiers scores. The authors show that their method performs better compared to basic averaging. Additionally, it is shown that Weaver -- en ensemble of multiple verifiers (LLMs) -- can be effectively distilled into one model.", "questions": "1. If I understood correctly, before aggregating the verifier scores, you binarize them. Why? It looks like discarding relevant information -- if a verifier gives a score of, say 0.55, it may mean it is uncertain and it will not skew the average in either direction. Also, binarization requires an additional step of selecting the right threshold. Did you experiment with un-binarized scores?\n\n2. Did you consider comparing Weaver with other methods for obtaining weights?\n\n3. How the base LLM was sampled? To achieve diversity of samples, crucial for the majority voting and verifier-dependent approaches, one need to sample with non-zero temperature. But too high temperature degrades the quality of individual samples. So this is a crucial parameter, but it is not optimized nor discussed... \n\n4. What development set (D^dev) was used to tune Weaver's weights? How large?\n\n5. How top-K verifiers were selected? Based on the performance on the development sets?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper tackles an interesting and important problem -- when multiple samples are extracted from an LLM, often some are correct whereas the others are not; but an additional approach is needed to recognize the correct ones -- thus the need for good verifiers.\n\nThe paper contains substantial number of experiments involving multiple verifiers.\n\n**Weaknesses**\n\n1. Experimental setup seems a bit weak / not entirely adequate:\n\n1.1. In addition to the down-stream task performance on the benchmarks it would be good to see the performance of the Weaver scores as measured by, e.g., AUC, and compared to the other approaches. This would be more direct assessment of the Weaver performance. \n\n1.2. In addition to the top-K approach it would be good to see how Weaver compares to some more straightforward methods for obtaining weights, e.g. logistic regression (mentioned in lines 168-170). Also, logistic regression with L1 regularization seems as an interesting alternative, as it would allow to reduce the number of verifiers in the ensemble.\n\n1.3. The authors assume that the 33 verifiers used in the experiments are de-correlated. However, this likely is not the case -- perhaps some of the verifiers were obtained by training on similar data? This may be the main reason why the simple averaging does not work. Perhaps selecting uncorrelated subset verifiers would be better that assigning non-zero weights for each of the verifiers?\n\n2. Important technical details missing (temperature of sampling, size of D^dev, how top-K were selected). See questions.\n\n3. Over-exaggerated claims / questionable presentation of some aspects:\n\n3.1. You write: \"Naively aggregating weak verifiers is insufficient for reliable verification\", but as far as I can tell, Weaver cannot achieve reliable verification either -- it is just to some degree better that normal (\"naive\"...) averaging.\n\n3.2. You emphasize that Weaver does not need much labeled data, and that alternative approaches may require more labels (lines 58-60). But did you actually compare Weaver with the alternative approaches? Maybe logistic regression can achieve better performance on the same number of labeled examples (D^dev)?\n\n4. The general idea of calling multiple (33 in the experiments) verifiers seems not very practical -- the verification becomes quite expensive. I understand that this motivated the distillation experiment, but there, in turn, there is added cost of training.\n\n5. The authors experiment with many verifiers but only with one base model (Llama 3.3 70B). Seeing the results for other base models would make the conclusions stronger.\n\n6. As the authors emphasize the compute-time efficiency of verification, it would be perhaps worth discussing the recent literature concerning this topic, where efficient verification is achieved by learning from the base LLM's internal representations -- see, e.g., [1,2].\n\n[1] Correctness Assessment of Code Generated by Large Language Models Using Internal Representations (https://arxiv.org/abs/2501.12934) \\\n[2] Lightweight Latent Verifiers for Efficient Meta-Generation Strategies (https://arxiv.org/abs/2504.16760)", "was_revised": true, "final_justification": "During the discussion the authors were active and answered all my questions as well as provided useful additional experiments. I believe that incorporating the explanations and new results is important and will improve the paper. \n\nIn general, I find the presented approach of combining multiple verifiers using \"moment matching\" on the unlabeled test set and using small dev set for calibration an interesting and useful approach. The core idea of it is perhaps not novel, but adapting it to the verification setting is original. However, the work has its weaknesses:\n1. Only two Llama models tested as generators of base responses.\n2. Not using more direct metric like AUC for assessing the quality of ranking responses from various (combinations of) verifiers.\n3. The suite of experiments could be more consistent and better designed (e.g., I got confused during the rebuttal what the dev set is used for in various experiments: class balance estimation? threshold estimation?)\n4. The algorithm itself could be explained better. For instance, if it's similar to Naive Bayes in it's first approximation, it would be helpful to emphasize it.\n5. Now I think that the paper could spend more time analyzing or interpreting the results. Is there some intuition why Weaver outperforms other approaches? Maybe seeing the weights Weaver and other methods assign to the 33 verifiers would be a good addition to the appendix. \n\nI'm rising the rating but only by one point because of the above reasons.", "rebuttal": "We appreciate your recognition of the problem, our experiments and constructive suggestions. \n\nBelow, we address your concerns, grouping related weaknesses and questions together. We include **new experiments** on: L1 regularization, Weaver on correlated verifiers, additional baselines, and using continuous verifier scores. We have included these results in the paper and would be grateful if you would consider **updating your score** in light of these improvements.\n\n**W1.1:  Performance of the Weaver scores vs other approaches as measured by AUC:**\n\nFollowing your suggestion, **we computed AUC for Weaver's scores**. Weaver achieves an average AUC of 0.7 (e.g. on GPQA for dev set 0.01x), outperforming naive ensembles (0.6) and logistic regression (0.65). The results reflect Weaver's improved ranking of correct vs. incorrect responses. \n\n**W(1.2, 2, 3.2, Q2, Q4): Comparing Weaver with other methods for obtaining weights and dependence on dev set size**\n\nComparisons with logistic regression (LR) and Naive Bayes (NB) using different sized dev sets are provided in Tables 4 and 15.  Weaver outperforms LR at smaller dev sizes (0.01x) and remains close for  large dev sets (0.5x). This advantage stems from how Weaver leverages labeled data to calibrate the class balance but not for training. For the main experiments, Weaver uses a dev set equivalent to 1% of the benchmark (e.g. ~5 problems for MATH-500, ~6 for GPQA’s 640). \n\nThe LR experiments in the paper use L2 regularization (λ=1). **We add L1 regularization experiments below** (λ=0.01 tuned via cross-validation). While LR +L1 regularization drops up to 50% of the verifiers, it does not reach the same performance as LR with L2 regularization.\n\n**Table 1: L1 vs. L2 Logistic LR with 1.0x Dev Set**\n|Model|GPQA|MATH500|MMLU Pro|Average|\n|---|---|---|---|---|\n|Supervised L1 LR|73.1%|92.5%|90.2%|**85.3%**|\n|Supervised L2 LR|78.1%|97.2%|92.0%|**89.1%**|\n|Weaver|72.7%|92.4%|90.4%|**85.2%**|\n\n**Table 2: L1 vs. L2 Logistic LR with 0.1x Dev Set**\n|Model|GPQA|MATH500|MMLU Pro|Average|\n|---|---|---|---|---|\n|Supervised L1 LR|54.1%|72.4%|73.5%|**66.7**|\n|Supervised L2 LR| 55.9%|70.5%|72.1%|**66.2%**|\n|Weaver|67.1%|90.4%|87.0%|**81.5%**|\n\nWe also added **two new supervised baselines, AdaBoost and Decision Tree.** We find that tree-based supervised strategies underperform.\n\n|Model|GPQA Diamond|MATH500|MMLU Pro|Average|\n|---|---|---|---|---|\n|AdaBoost (1.0x Dev Set)|67.1%|90.2%|84.1%|**80.5%**|\n|DecisionTree (1.0x Dev Set)|64.5%|87.1%|86.3%|**79.3%**|\n\n**W1.3: Concern that verifier correlation may undermine simple averaging, and suggestion to instead select an uncorrelated subset of verifiers.**\n\nWeaver does not assume de-correlated verifiers. Instead, it assumes conditional independence given the true labels, meaning that verifier dependencies can be explained through their relationship to the ground truth. To illustrate Weaver’s effectiveness when verifiers are highly correlated, **we add the following experiment**: We selected all 13 verifiers from our main experiments that were derived from the Llama family (e.g., Llama 3.1 8B/70B variants) and fine-tuned on similar RLHF data, representing the subset most likely to be highly correlated. These verifiers have an average Pearson correlation of 0.491. Table 1 below shows that Weaver achieves a +4.3% average improvement over naive ensembling—a small drop from  the +6.8% improvement when we use the full, more diverse pool (Table 2; average correlation 0.281). These results indicate that while diversity enhances performance, Weaver remains effective even when verifiers are highly redundant. \n\nWhile selecting an uncorrelated subset of verifiers might improve naive ensemble performance, there is a tradeoff involved in such a selection process. The optimal ensemble balances diversity and individual verifier accuracy [7], a tradeoff that Weaver learns to navigate using unlabeled data.\n\n**Table 1: New Results - Weaver with Llama Verifiers**\n|Model|GPQA|MATH500|MMLU Pro|Average|\n|---|---|---|---|---|\n|Weaver with Llama Verifiers|59.4%|91.1%|87.4%|79.3%|\n|Naive Ensemble|52.1%|88.9%|84.1%|75.0%|\n\n**Table 2: Existing Results - Full Weaver with All Verifiers**\n|Model|GPQA|MATH500|MMLU Pro|Average|\n|---|---|---|---|---|\n|Weaver|66.4%|93.4%|90.2%|83.3%|\n|Naive Ensemble|54.5%|90.6%|84.3%|76.5%|\n\n**W3.1: \"Weaver cannot achieve reliable verification either - it is just to some degree better than normal (\"naive\") averaging.\"**\n\nThank you for the thoughtful comment. We agree that Weaver does not turn weak verifiers into a perfect oracle. Rather, it narrows the generation-verification gap (e.g., by 14.5% on average; see Figure 1, Table 1) by improving verification accuracy through scalable ensembling. We will revise the wording to better reflect this goal and appreciate the opportunity to clarify.\n\n**W4: Concern about the practicality of using 33 verifiers and the additional distillation cost**\n\nWe respectfully disagree that using multiple verifiers is impractical and raise the following three points:\n\n**1) Verification is parallelizable and can leverage distributed/idle compute.**\n Verifier models run independently and require only a single forward pass—no autoregressive decoding—making them highly parallelizable across devices. Many verifiers (8B–30B) can run on consumer GPUs, enabling distributed or overnight batch verification using idle compute.\n\n**2) Weaver performs well even with few verifiers.** As shown in Figure 4, Weaver yields strong gains even with limited ensembles. Instead of the full 33 used in experiments, we see that with just 3 verifiers, we can still outperform naive ensembling on GPQA, MATH500, and MMLU Pro.\n\n**3) Distillation pays off quickly.** Training the 400M distilled verifier requires ~10 GPU-hours (Appendix C.6). This small training cost is quickly amortized; after roughly 1,000 inferences, the distilled model becomes more compute-efficient than the full ensemble while retaining 98.7% of its accuracy.\n\n**W5: “Only one base model (Llama 3.1 70B) is used.”**\n\nOur paper also includes results for Llama 3.1 8B as a generator in Tables 3 and 18, where our results demonstrating Weaver’s gains still hold. In our final draft, we will include results on additional model families, such as the Qwen models.\n\n**W6: \"Add discussion of recent literature on efficient verification via learning from internal representations [1, 2]”.**\n\nWe appreciate the valuable references [1,2] and have incorporated them. Both propose verifiers that use LLM hidden states to predict generation correctness. In contrast, Weaver is a black-box aggregation framework, designed for flexibility and compatibility across open- and closed-source models. These internal verifiers are fully compatible with Weaver and could be included in its pool to provide complementary signals and reduce verification cost.\n\n**Q1: Concern that binarizing scores discards relevant information**\n\nWe explain the motivation behind binarization and present results using a continuous version of our algorithm. \n\n**Due to differences in how verifiers are trained, their continuous scores vary significantly in distribution.** Verifier scores can range from unbounded log-likelihoods to learned regression scores. In our analysis, we found that even after normalization (e.g., min-max scaling or quantile adjustment), verifier scores exhibited sharp multimodality, heavy skew toward extremes, flat or noisy profiles, and inconsistent calibration—e.g., a score of 0.8 could signal high confidence in one verifier and uncertainty in another. These inconsistencies complicate continuous modeling, as they introduce additional parameters or hierarchical structures that increase the complexity of learning the latent variable model.\n\n**Modeling binary scores aligns better weak supervision frameworks,** enabling the use of a tractable model with fewer parameters and efficient moment-matching for latent variable estimation. These models are theoretically grounded and have proven effective in noisy-label scenarios.\n\n**Smart binarization does not degrade performance**: To assess what is lost from binarization, we ran a new experiment where we compared several strategies in a logistic regression setting. These included: (1) no binarization, (2) class balance: choosing a threshold so that the verifier balance matches the class balance; and (3) quantile: assign positive labels to the top 15% of scores. While continuous scores achieved strong AUC when ample labels were available, the class balance and quantile binarization matched or outperformed no binarization under low-label regimes. \n\n**Continuous Weak Supervision algorithm performs poorly**: We also implemented a recent continuous weak supervision method based on Gaussian moment matching [6], which can operate on raw verifier scores.  This approach consistently underperforms compared to the binarized model. The continuous model struggles with the non-Gaussian and asymmetric nature of real-world verifier scores, leading to inaccurate reliability estimates and degraded aggregation performance. \n\n|Model|GPQA|MATH500|MMLU Pro|Average|\n|---|---|---|---|---|\n|Continuous Weaver|63.1%|89.6%|88.4%|80.4%|\n|Discrete Weaver|66.4%|93.4%|90.2%|83.3%|\n\n**W2, Q3: How is the base LLM sampled?**\n\nWe sample the base LLM temperature 0.7 to balance diversity and sample quality, following prior work for reasoning and math tasks [3, 4, 5].\n\n**Q5: \"How top-K verifiers were selected? Based on the performance on the development sets?\"**\n\nIn the experiments for Figure 2 and 4, the top-K verifiers were computed in an oracle setting using the full dataset.\n \n[1] https://arxiv.org/abs/2501.12934\n\n[2] https://arxiv.org/abs/2504.16760\n\n[3] https://arxiv.org/abs/2502.17578\n\n[4] https://arxiv.org/abs/2407.21787\n\n[5] https://arxiv.org/abs/2408.03314\n\n[6] https://arxiv.org/abs/2112.03865\n\n[7] https://arxiv.org/abs/2302.00704"}]

Author rebuttal:

Meta-review: This paper introduces Weaver, a verification framework of language model outputs by aggregating signals from multiple "weak" verifiers, such as language model judges and reward models. The central claim is that by combining these imperfect verifiers in a principled, data-efficient manner, one can create a strong, unified verifier that significantly improves the selection of correct responses from a pool of candidates generated at test time.

Based on my reading and the reviewers' assessments, the key findings are:
1.  **Superior Aggregation:** The paper demonstrates that a weighted aggregation of verifiers substantially outperforms naïve methods like simple averaging or majority vote. Weaver adapts weak supervision techniques, using a small amount of labeled data for calibration and a large amount of unlabeled data to learn the accuracy of each verifier.
3.  **Compute Efficiency via Distillation:** Recognizing that querying dozens of verifiers can be computationally expensive, the authors show that the aggregated signal from the Weaver ensemble can be distilled into a compact 400M parameter cross-encoder model. This distilled verifier retains 98.7% of the full ensemble's performance while reducing the verification compute cost by up to 99.97%, making the approach highly practical.
2.  **Performance:** By using Weaver to select the best response from 100 samples generated by Llama 3.3 70B, the system achieves performance on par with or exceeding that of much larger, models.


### (b) Strengths of the Paper

*   **Significance and Novelty:** The paper addresses the critical and highly relevant problem of improving  verification . The proposed method of using weak supervision for label-efficient verifier ensembling is novel and practical.
*   **Strong Empirical Results:** The experimental results are impressive and demonstrate significant gains over strong baselines across multiple challenging benchmarks . The ability to elevate the performance of a 70B model to the level of frontier models is a compelling demonstration of the framework's power.
*   **Practicality and Scalability:** The distillation of the large verifier ensemble into a small, efficient cross-encoder is a crucial contribution that makes the system viable for real-world deployment. The label-efficient nature of Weaver, requiring only a small calibration set, further enhances its practicality, especially for domains where labeled data is scarce or expensive.
*   **Thorough Analysis:** The authors provide a comprehensive analysis of Weaver's performance, exploring its scaling properties with respect to the number of samples, verifiers, and model size.

### (c) Weaknesses of the Paper

The reviewers raised several valid concerns, many of which were addressed during the rebuttal period:
*   **Dependence on Verifier Diversity:** A key concern was how Weaver's performance would be affected by a lack of diversity or high correlation among the weak verifiers, which is a realistic scenario (Reviewer XeAb, Reviewer jVww, Reviewer cAMM).
*   **Experimental Setup:** There were initial questions about the fairness of the evaluation setup, particularly regarding the use of a development set drawn from the same benchmarks used for testing. This raised concerns about potential test set leakage (Reviewer jVww, Reviewer cAMM). Reviewers also requested comparisons against more sophisticated baselines, such as regularized logistic regression and tree-based models (Reviewer cAMM).
*   **Generalizability:** The evaluation was primarily focused on mathematical and logical reasoning tasks using a single family of generator models (Llama). Reviewers questioned the framework's applicability to other domains like code generation or subjective reasoning (Reviewer jVww).
*   **Clarity and Missing Details:** Some technical details were initially unclear, such as the motivation for binarizing verifier scores and the precise mechanism for estimating certain probabilities in the model (Reviewer cAMM).

### (d) Recommendation: Accept

I strongly recommend accepting this paper. The work is technically solid, addresses a problem of high importance, and presents a novel, practical, and effective solution. The empirical results are compelling and demonstrate a significant step forward in making language models more reliable through enhanced verification.
Decision: Accept (poster)