Title: Weaver: Shrinking the Generation-Verification Gap by Scaling Compute for Verification
Venue: NeurIPS 2025
Accepted: yes
Keywords: test-time compute

repeated sampling

weak supervision

weak verification
Abstract: Verifiers can improve language model (LM) capabilities by providing feedback or selecting the best response from a pool of generated candidates. Currently, high-quality verifiers are either unscalable (e.g., humans) or limited in utility (e.g., tools like Lean for formal proofs). While LM judges and reward models have become broadly useful as general-purpose verifiers, a significant performance gap remains between them and oracle verifiers. To help close this gap, we introduce Weaver, a framework for designing a strong verifier by combining multiple weak, imperfect verifiers. First we find that weighted ensembles of verifiers, which typically require learning from labeled data, significantly outperform unweighted combinations due to differences in the verifiers. To reduce the dependency on labeled data, Weaver leverages weak supervision to estimate each verifier’s accuracy and combines their outputs into a unified score that better reflects true response quality. However, directly applying weak supervision algorithms poses several challenges, including inconsistent verifier output formats and handling low-quality verifiers. Weaver addresses these challenges by using dataset statistics to normalize outputs and filter specific verifiers. We study the effectiveness of Weaver in repeated sampling settings, where a model generates multiple candidate responses at test time and a verifier is used to select the correct one. Our evaluations demonstrate that Weaver significantly improves the pass@1 performance across several reasoning and math tasks, achieving o3-mini level accuracy with Llama 3.3 70B Instruct (a much cheaper non-reasoning model) as the generator, and an ensemble of smaller judge and reward models as the verifiers (86.2% average). This gain mirrors the jump achieved between GPT-4o and o3-mini (69.0% vs. 86.7%), which required extensive finetuning and post-training interventions. To make Weaver more efficient, we train a compact 400M cross-encoder using Weaver's combined output scores. This distilled model retains 98.7% of Weaver's full accuracy while reducing verification compute by up to 99.97%.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Weaver, a method for improving verification of LLM outputs. This method, given N \"verifiers\" (or \"reward models\") and a development set of labeled examples, assigns weights for each of the verifiers, which are used to aggregate the verifiers scores. The authors show that their method performs better compared to basic averaging. Additionally, it is shown that Weaver -- en ensemble of multiple verifiers (LLMs) -- can be effectively distilled into one model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe paper tackles an interesting and important problem -- when multiple samples are extracted from an LLM, often some are correct whereas the others are not; but an additional approach is needed to recognize the correct ones -- thus the need for good verifiers.\n\nThe paper contains substantial number of experiments involving multiple verifiers.\n\n**Weaknesses**\n\n1. Experimental setup seems a bit weak / not entirely adequate:\n\n1.1. In addition to the down-stream task performance on the benchmarks it would be good to see the performance of the Weaver scores as measured by, e.g., AUC, and compared to the other approaches. This would be more direct assessment of the Weaver performance. \n\n1.2. In addition to the top-K approach it would be good to see how Weaver compares to some more straightforward methods for obtaining weights, e.g. logistic regression (mentioned in lines 168-170). Also, logistic regression with L1 regularization seems as an interesting alternative, as it would allow to reduce the number of verifiers in the ensemble.\n\n1.3. The authors assume that the 33 verifiers used in the experiments are de-correlated. However, this likely is not the case -- perhaps some of the verifiers were obtained by training on similar data? This may be the main reason why the simple averaging does not work. Perhaps selecting uncorrelated subset verifiers would be better that assigning non-zero weights for each of the verifiers?\n\n2. Important technical details missing (temperature of sampling, size of D^dev, how top-K were selected). See questions.\n\n3. Over-exaggerated claims / questionable presentation of some aspects:\n\n3.1. You write: \"Naively aggregating weak verifiers is insufficient for reliable verification\", but as far as I can tell, Weaver cannot achieve reliable verification either -- it is just to some degree better that normal (\"naive\"...) averaging.\n\n3.2. You emphasize that Weaver does not need much labeled data, and that alternative approaches may require more labels (lines 58-60). But did you actually compare Weaver with the alternative approaches? Maybe logistic regression can achieve better performance on the same number of labeled examples (D^dev)?\n\n4. The general idea of calling multiple (33 in the experiments) verifiers seems not very practical -- the verification becomes quite expensive. I understand that this motivated the distillation experiment, but there, in turn, there is added cost of training.\n\n5. The authors experiment with many verifiers but only with one base model (Llama 3.3 70B). Seeing the results for other base models would make the conclusions stronger.\n\n6. As the authors emphasize the compute-time efficiency of verification, it would be perhaps worth discussing the recent literature concerning this topic, where efficient verification is achieved by learning from the base LLM's internal representations -- see, e.g., [1,2].\n\n[1] Correctness Assessment of Code Generated by Large Language Models Using Internal Representations (https://arxiv.org/abs/2501.12934) \\\n[2] Lightweight Latent Verifiers for Efficient Meta-Generation Strategies (https://arxiv.org/abs/2504.16760)\n\nQUESTIONS:\n1. If I understood correctly, before aggregating the verifier scores, you binarize them. Why? It looks like discarding relevant information -- if a verifier gives a score of, say 0.55, it may mean it is uncertain and it will not skew the average in either direction. Also, binarization requires an additional step of selecting the right threshold. Did you experiment with un-binarized scores?\n\n2. Did you consider comparing Weaver with other methods for obtaining weights?\n\n3. How the base LLM was sampled? To achieve diversity of samples, crucial for the majority voting and verifier-dependent approaches, one need to sample with non-zero temperature. But too high temperature degrades the quality of individual samples. So this is a crucial parameter, but it is not optimized nor discussed... \n\n4. What development set (D^dev) was used to tune Weaver's weights? How large?\n\n5. How top-K verifiers were selected? Based on the performance on the development sets?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes WEAVER: an unsupervised probabilistic model that learns per-verifier weights to fuse dozens of weak LLM judges.  Applied to 100-sample decoding, WEAVER boosts pass\\@1 accuracy over the base model and also over majority vote on four reasoning benchmarks, letting a 70 B generator rival GPT-4. Furthermore, the authors distill a 400 M distilled cross-encoder retains ≈99 % of the gain while slashing verification Flops.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The motivation is clear and well-written.\n2. The results show significant improvements across four benchmarks.\n3. The distillation method significantly reduces computational cost while maintaining strong performance.\n\n## Weaknesses\n\n1. How sensitive is WEAVER to the diversity and independence of verifiers, particularly if reward models or judges are trained on similar data or architectures? If the verifier models are similar, will this lead to performance degradation?\n2. Has WEAVER been evaluated on tasks beyond math QA, such as code generation or subjective reasoning? Are there known challenges or adaptation requirements for those domains?\n3. Can the overall process be interpreted as learning weighted coefficients from a small subset using traditional machine learning methods? If so, would tree-based models be applicable?\n4. Does the effectiveness of the ensemble method rely on the small subset and test set being drawn from the same distribution? If not, would the learned coefficients become unreliable?\n5. For the distillation method, if a small LLM is fine-tuned on a specific dataset, would it achieve results comparable to those reported in the paper?\n\nQUESTIONS:\nSame as the weakness section.\nIf the authors could solve my questions, I will raise my score.\n\n1. How sensitive is WEAVER to the diversity and independence of verifiers, particularly if reward models or judges are trained on similar data or architectures? If the verifier models are similar, will this lead to performance degradation?\n2. Has WEAVER been evaluated on tasks beyond math QA, such as code generation or subjective reasoning? Are there known challenges or adaptation requirements for those domains?\n3. Can the overall process be interpreted as learning weighted coefficients from a small subset using traditional machine learning methods? If so, would tree-based models be applicable?\n4. Does the effectiveness of the ensemble method rely on the small subset and test set being drawn from the same distribution? If not, would the learned coefficients become unreliable?\n5. For the distillation method, if a small LLM is fine-tuned on a specific dataset, would it achieve results comparable to those reported in the paper?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nVerification of responses to instructions has several valuable use cases, but existing verifiers are noisy. This paper explores how to combine multiple weak verifiers in the experimental setting of improving response selection under repeated sampling. There are three key contributions:\n\n1. Given a large corpus of labeled query-response pairs, they show that using logistic regression to learn verifier weights outperforms naive averaging of verifiers or choosing the top verifier by a large margin.\n2. To handle data-limited settings, they introduce WEAVER, which adapts weak supervision to build a latent variable model to estimate verifier weights.\n3. They show that the WEAVER ensemble can be distilled into a 400M cross-encoder verifier, which retains 99% of the performance while using only 0.03% of the inference FLOPS.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n1. The empirical results are strong. WEAVER achieves significant improvements over existing verification methods for identifying the best sample from repeated generations (Table 1), and scales better with test-time compute (Figure 5). The authors provide analysis on how WEAVER scales along many dimensions, including the number of samples, the model size, the number of verifiers, and inference compute.\n2. The problem studied is of great interest to the community. Better verifiers have many applications (as discussed by the authors), such as data filtering, alignment, and test-time compute scaling.\n\n### Weaknesses:\n1. Another baseline that I would like to see is the approach considered in §4.1, where linear regression using a large amount of training data is used to learn weights over the verifiers. While this approach requires more data, it paints a better picture of how much performance is preserved by WEAVER in the data-constrained regime.\n2. What is the motivation for studying the data-constrained setting? I believe there is an abundance of instruction-response pairs for learning weighted ensembles of verifiers.\n\nQUESTIONS:\n1. In §4.1 and §5 (the main experiments), which dataset is used for learning the verifier weights? How much labeled training data does WEAVER use?\n2. Should the x-axis label in Figure 4 be \"Number of verifiers\" instead of \"Top-k Verifiers\"?\n3. What are the frontier approaches in Table 1?\n4. What questions do you use for training WEAVER-Distilled? I.e., do you train on WEAVER's pseudolabels for the same questions that you evaluate on?\n5. Why do you think that a small, distilled verifier performs so well? Would this verifier perform well on RewardBench?\n\n**Other suggestions:**\n- In Table 1, it may be a good idea to add a column for the number of verifiers. E.g., it would be 0, 0, 1, 10 for the first 4 rows, and then 33 for WEAVER. \n- Line 294: remove comma after \"Figure 6\"\n- Line 324: \"combining many weak verifier\" $\\rightarrow$ \"combining many weak verifiers\"\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of verifying and selecting correct responses from a language model’s outputs in order to improve the model’s effective performance. The authors propose WEAVER, a framework that combines multiple weak verifiers (such as language-model-based judges and reward models) to approximate a much stronger verification signal without requiring any labeled training data. Empirically, WEAVER leads to large gains in downstream task accuracy. To address the computational cost of using many verifiers, the authors further distill the ensemble’s behavior into a single 400M-parameter cross-encoder model. This distilled verifier retains about 98.7% of the full ensemble’s accuracy while reducing verification compute cost by ~99.97%, making the overall approach far more efficient. Overall, the paper’s contributions include a novel unlabeled ensemble method for response verification (WEAVER), a demonstration of significantly improved selection of correct answers (shrinking the generation-verification gap), and a practical distillation step to save runtime cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n*  WEAVER significantly outperforms standard baselines (e.g., majority vote, self-verification) and nearly matches state-of-the-art models like GPT-4o and o3-mini, despite using a cheaper generator.\n\n* WEAVER combines multiple weak verifiers using weak supervision techniques without requiring labeled ground-truth data, which is a practical and scalable contribution.\n\n* The distillation of WEAVER into a lightweight 400M cross-encoder retains ~98.7% accuracy while reducing inference compute by 99.97%, enabling real-world applicability.\n\n* Applies latent variable modeling to verifier combination in a new setting, overcoming output inconsistencies and low-quality verifiers.\n\nWeakness:\n* Performance depends on having access to a broad and complementary pool of weak verifiers.\n\n* WEAVER is most useful in repeated sampling scenarios; its gains are less relevant where only a single response can be generated.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-18"}