Title: Majority of the Bests: Improving Best-of-N via Bootstrapping
Venue: NeurIPS 2025
Accepted: yes
Keywords: llms

test-time compute

best-of-n

bootstrap

inference-time computation
Abstract: Sampling multiple outputs from a Large Language Model (LLM) and selecting the most frequent (Self-consistency) or highest-scoring (Best-of-N) candidate is a popular approach to achieve higher accuracy in tasks with discrete final answers. Best-of-N (BoN) selects the output with the highest reward, and with perfect rewards, it often achieves near-perfect accuracy. With imperfect rewards from reward models, however, BoN fails to reliably find the correct answer and its performance degrades drastically. We consider the distribution of BoN’s outputs and highlight that, although the correct answer does not usually have a probability close to one under imperfect rewards, it is often the most likely outcome. This suggests that the mode of this distribution can be more reliably correct than a sample from it. Based on this idea, we propose Majority-of-the-Bests (MoB), a novel selection mechanism that estimates the output distribution of BoN via bootstrapping and selects its mode. Experimental results across five benchmarks, three different base LLMs, and two reward models demonstrate consistent improvements over BoN in 25 out of 30 setups. We also provide theoretical results for the consistency of the bootstrapping. MoB serves as a simple, yet strong alternative to BoN and self-consistency, and more broadly, motivates further research in more nuanced selection mechanisms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA popular trend in inference-time scaling of LLMs involves sampling many responses, and using some strategy to aggregate their answers to produce a single final answer.  This work is motivated by the shortcomings of common approaches like Best-of-N (BoN) and self-consistency (SC), proposing to improve on BoN by applying a bootstrap approach to debiasing the choice of the noisy reward model.  Theoretical and empirical evidence is provided to support the approach, with experiments showcasing applications in Math, Chemistry and QA.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThis paper offers a solution to a high-priority problem for the LLM community: broadly, what is the best way to scale inference-time compute, and in particular, how should we combine solutions from many samples.  As opposed to the many heuristic approaches that have been taken to this problem, the bootstrapping method proposed here is underpinned by rigorous theory. Also, the compute cost compares favorably with other methods.\n\nIn terms of the paper presentation, the method description (and accompanying empirical demonstrations) offer solid support for the proposed algorithm.\n\n## Weaknesses\n\nWhile I appreciate the simplicity and motivation of the MoB algorithm, I find the experimental evaluation to have significant weaknesses that prevent the reader from gaining a thorough impression of when and why this algorithm might be helpful.\n\nFirst, it is not clear why certain experiment details are chosen.  For example, why is evaluation done on this selection of 3 math, 1 chemistry, and 1 QA tasks?  Are these domains where you expect the method to be particularly useful, or that are helpful to reveal the strengths and weaknesses of the algorithm?  Also, does it make sense to use these two reward models for all of these tasks?  One could imagine that different reward models might be chosen for different tasks, e.g., math or QA.  Also, these do not seem to be the strongest available reward models of their size (according to rewardbench).  Finally, why do some experiments use few-shot prompting, and some use zero-shot CoT?  \n\nNext, related to the last question, it seems that baseline implementations may not match the approach suggested in the original work.  For Wang 2022, I believe that SC is always meant to be combined with chain of thought (as opposed to few-shot prompting).  Similarly, for WBoN with Li 2022 as a reference, I believe the original method involves some diverse prompting and (crucially) step-level labeling, which do not seem to be applied here.\n\nFinally, and potentially my biggest concern with the experiments, is that the presentation of results seems aimed at boosting the perceived performance of MoB.  For example, I am quite confused by how items are bolded or not in tables.  Across the 5 tables, I found 9 examples of unbolded results that have overlapping CIs with the best result.  Accordingly, I am further confused by the claim in the conclusion that MoB outperforms all baselines in 25 of 30 settings.  This does not reflect the results in the tables.\n\nOverall, I feel that the experiment section can benefit from more focus on “when” and “why?” questions about the algorithm’s performance, as opposed to focusing purely on outperforming baselines.  Given that the algorithm is not technically deep, the weaknesses of the experimental results outweigh the strengths of the paper in my opinion.\n\nAlso, ideally code would be included.\n\nQUESTIONS:\nPlease see weaknesses section.\n\nLIMITATIONS:\nThe authors address limitations in one sentence in the conclusion.  More discussion (and demonstration) of the algorithm’s limitations would be welcomed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Majority-of-the-Bests, a method to improve Best-of-N decoding for LLMs by using bootstrapping to estimate the distribution of BoN outputs and selecting the most frequent outcome. By aggregating rewards over resampled subsets, MoB aims to mitigate errors from noisy reward models and shows consistent empirical gains across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- Conceptually simple methodology\n- Consistent experimental results on a wide range of settings\n\n# Weaknesses\n- The paper frames BoN’s randomness as a problem to be mitigated. But if reward scores are noisy and correctness is inherently uncertain, is forcing the \"mode\" always desirable? Sometimes randomness can reflect epistemic uncertainty, not just noise.\n- Bootstrapping is a statistical resampling technique that works well when the sample size is large and the statistic of interest is smooth and not too sensitive to outliers. However, this work applies bootstrapping to a relatively small sample set and attempts to approximate the extreme statistic: the maximum reward. So I am a bit suspicious about the validity and soundness of applying the bootstrapping technique.\n- In Figure 2, I am curious why the authors assigned differently-sized bins for the histogram. To claim that \"BoN has a significant chance of returning the correct answer but fails to do so reliably\", a more fine-grained histogram is needed to demonstrate the skewness of the success probability distribution.\n- The BoN itself is a computationally heavy algorithm that is not often used in practice. Although authors disclose that the CPU computation overhead is marginal, I suspect MoB will take longer to compute overall. It would be helpful to have a full comparison of computation time (latency) among methods: Best-of-1, BoN, MoB, and BoN+SC.\n\nQUESTIONS:\n- How do you define the Oracle final answer distribution? I am not sure how one can access the true pi_N.\n\nLIMITATIONS:\nThere does not seem to be a separate limitations section or a societal impact section. Please add them.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes **Majority-of-the-Bests (MoB)**, a hyper-parameter-free alternative to Best-of-N (BoN) sampling for inference-time reasoning with LLMs.\n\n- **Motivation.** BoN’s success probability can saturate well below 1 with imperfect rewards. If the correct answer is the most probable output of BoN, we are likely to fix the errors of BoN by doing majority voting over BoN results.\n\n- **Key idea.** After sampling *N* candidate traces and scoring them with a reward model, MoB repeatedly bootstraps *m*-sized subsets (with replacement), keeps only the best-of-m trace in each subset, and select the mode of these “best” traces.\n\n- **Theory.**  The authors proves that:\n  - An *m-out-of-N* bootstrap consistently estimates BoN’s output distribution. That is, MoB converges to its oracle version on large N.\n  - Their algorithm of adaptively select nearly optimal m with theoretical gaurantee under certain situation. Empirical results also show that adaptive m performance closely follows the optimal m variant.\n- **Experiments.**  On five benchmarks, three open-weight LLMs and two reward models, MoB outperforms BoN, Self-Consistency and Weighted BoN in **25 / 30** settings; at $N = 128$ it raises accuracy by largely over BoN.\n- **Practicality.**  The method adds only marginal CPU overhead; the computational-heavy sampling part remains unchanged.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Solid theoretical analysis and a proof of bootstrap consistency; extensive evaluation across 30 settings; results reported with confidence intervals.\n- Motivation is intuitive; algorithm and adaptive-*m* procedure are clearly described with helpful figures.\n- MoB can serve as a drop-in replacement of BoN with marginal overhead, offering systematic gains without tuning.\n\n**Weaknesses**\n\n- Models stop at 9B parameters and open-weight models; no ablation on bootstrap budget *B*.\n- A few typos remain; discussion of failure cases is brief and buried in the appendix.\n- Share the same limited use cases as SC. Open-ended generation (summaries, codegen) is not supported, unlike BoN.\n- Relies on combining previous methods (bootstrap, majority voting, BoN); originality is limited.\n\nQUESTIONS:\n1. **When does MoB underperform BoN?** The appendix hints that very hard questions (base success rate <1%) may favor larger m or even plain BoN since the correct answer may no longer be the most probable one for BoN under small m. Could you add a difficulty-conditioned analysis and a practical decision rule?\n2. **Larger / proprietary models.** Can you report at least a subset of results on proprietary LLMs and larger open-weight LLMs like GPT-4o, Gemini 2.5, and Qwen3-30B-A3B, perhaps on fewer datasets, to further demonstrate scalability and generalizability?\n3. **Comparing with SC.** Given that MoB shares the same limitation of not supporting open-ended generation with SC, could you also add SC as an additional baseline and report MoB improvements over SC?\n\n**Rating swing.** Demonstrating (1) a principled failure-mode analysis and (2) results on at least one >10B or proprietary model with consistent results would increase my overall score by 1 point (to 5: Accept).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes \"Majority of the Bests (MoB)\" a simple method to improve best-of-n sampling with LLMs using  bootstrapping. The proposed method works by sampling multiple times from generated LLM outputs, selecting the highest rewarded output, followed by selecting the majority answer from the filtered outputs. The paper provides both theoretical and experimental support for superiority of their method. When evaluated on multiple datasets, 3 generator models and 2 different reward models, the method outperforms both BoN and weighted BoN.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n1. The paper proposes a simple method to improve the performance of LLMs, while showcasing strong results across a broad range of datasets and models.\n2. The method can act as a replacement for the widely used Weighted Best-of-N method across the domains tested.\n3. The paper is well-presented and easy to follow.\n\n---\n\n## Weaknesses\n1. The MoB method works based on the assumption that reward model outputs are noisy. While generally true, the paper does not compare against the latest strong reward models. If the authors could show results with them (e.g., by choosing the strongest reward model from RewardBench on respective domains), the effectiveness of MoB could be further strengthened.\n2. In the text and tables, the authors continuously emphasize comparison with BoN instead of WBoN (e.g., the improvement row in Table 2). Further, the authors claim \"MoB is a drop-in replacement for BoN\". However, this framing is arguably unfair, since evaluations are a.) only done on tasks with a discrete output space and b.) majority is calculated over final answers and not full outputs.\n3.  There is limited analysis in the paper. For instance, the authors could add what are some failure modes of the method, on which questions the methods fail, how scaling RM size changes the performance, or the effects of hyperparameters. The complete results for the stated \"30 setups\" are also not presented.\n4. The paper claims the method requires no hyperparameter tuning; however, both variables q and B have to be chosen, and an analysis studying their effects is missing.\n5. Scaling the number of model outputs is one dimension, while scaling the number of outputs from generative reward models (which are often used) is another. Comparing the effect, and how MoB can be used with generative reward models, would be an interesting study. \n6. The paper claims the method is computationally efficient. However, it may still require significant CPU resources because of the large value of B (10,000), which might increase latency and cost in real-world usage. Results showing actual wall time and CPU usage are missing.\n7.  The paper lacks a dedicated Related Work section. While a minor issue, omitting one makes it harder to understand the work's novelty.\n---\n\nOverall, the proposed method is promising with some exciting results. However, a more thorough analysis is needed to further validate the claims.\n\nQUESTIONS:\nPlease See the Weaknesses Above.\n\nAdditionally:\n1. The results in Table 1, Row: WBoN and Column: CSQA are abnormally low. Is there a typo?\n2. I particularly like the Figure 6 showcasing how Accuracy varies with number of generations. Form graph it seems, the performance difference is particularly higher for smaller N. Could authors provide results for other setups, and plot performance difference as a function of N? The analysis might further strengthen the paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-06"}