Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

## Abstract

Abstract Scaling the amount of compute used to train language models has dramatically improved their capabilities. However, when it comes to inference, we often limit the amount of compute to only one attempt per problem. Here, we explore inference compute as another axis for scaling by increasing the number of generated samples. Across multiple tasks and models, we observe that coverage – the fraction of problems solved by any attempt – scales with the number of samples over four orders of magnitude. In domains like coding and formal proofs, where all answers can be automatically verified, these increases in coverage directly translate into improved performance.
When we apply repeated sampling to SWE-bench Lite, the fraction of issues solved with DeepSeek-Coder-V2-Instruct increases from 15.9% with one sample to 56% with 250 samples, outperforming the single-attempt state-of-the-art of 43% which uses more capable frontier models.
Moreover, using current API pricing, amplifying the cheaper DeepSeek model with five samples is more cost-effective and solves more issues than paying a premium for one sample from GPT-4o or Claude 3.5 Sonnet. Interestingly, the relationship between coverage and the number of samples is often log-linear and can be modelled with an exponentiated power law, suggesting the existence of inference-time scaling laws.
Finally, we find that identifying correct samples out of many generations remains an important direction for future research in domains without automatic verifiers.
When solving math word problems from GSM8K and MATH, coverage with Llama-3 models grows to over 95% with 10,000 samples.
However, common methods to pick correct solutions from a sample collection, such as majority voting or reward models, plateau beyond several hundred samples and fail to fully scale with the sample budget. 0 0 footnotetext: ∗ Equal Contribution. Work done by BB as a visiting researcher at Stanford.

## 1 Introduction

The ability of large language models (LLMs) to solve coding, mathematics, and other reasoning tasks has improved dramatically over the past several years [, ]. Scaling up model training has been a consistent driver of these gains. Investments in larger models, larger pre-training datasets, and more extensive post-training (e.g. through collecting human preference labels) has led to remarkably capable generalist systems [, , , ].

In contrast, a comparatively limited investment has been made in scaling the amount of computation used during inference. Larger models do require more inference compute than smaller ones, and prompting techniques like chain-of-thought [] can increase answer quality at the cost of longer (and therefore more computationally expensive) outputs. However, when interacting with LLMs, users and developers often restrict models to making only one attempt when solving a problem. In this work, we investigate repeated sampling (depicted in Figure 1) as an alternative axis for scaling inference compute to improve LLM reasoning performance.

Figure: Figure 1: The repeated sampling procedure that we follow in this paper. 1) We generate many candidate solutions for a given problem by sampling from an LLM with a positive temperature. 2) We use a domain-specific verifier (ex. unit tests for code) to select a final answer from the generated samples.
Refer to caption: x1.png

The effectiveness of repeated sampling is determined by two key properties:

- 1.
Coverage: As the number of samples increases, what fraction of problems can we solve using any sample that was generated?
- 2.
Precision: In settings where we must select a final answer from the collection of generated samples, can we identify correct samples?

With unlimited attempts, any model that assigns a non-zero probability to every sequence will achieve perfect coverage. However, repeated sampling is only practical if we can improve coverage with a feasible budget.
Moreover, without the ability to decide between samples, the applications of repeated sampling are limited. Existing work provides encouraging evidence along both of these directions, showing examples of repeated sampling improving performance in math, coding, and puzzle-solving settings [, , ]. Notably, AlphaCode [], a state-of-the-art system for competitive programming, finds that performance continues to improve with a million samples per problem.

Here, we show that repeated sampling is an effective methodology for improving coverage across a range of tasks, models, and sample budgets.
For example, when solving CodeContests [] programming problems using Gemma-2B [], we increase coverage by over 300x as we scale the number of samples, from 0.02% with one attempt to 7.1% with 10,000 attempts.
Interestingly, the relationship between $\log(\text{coverage})$ and the number of samples often follows an approximate power law. With Llama-3 [] and Gemma models, we observe that coverage grows nearly log-linearly with the number of samples over several orders of magnitude.

In settings where all of a model’s solutions can be automatically verified, such as with proof checkers or unit tests, these increases in coverage translate directly into improved task performance. When applying repeated sampling to competitive programming and writing Lean proofs, models like Llama-3-8B-Instruct can exceed the single-attempt performance of much stronger ones like GPT-4o []. This ability to amplify weaker models extends to the challenging SWE-bench Lite dataset of real-life GitHub issues [], where the current single-attempt state-of-the-art (SOTA), achieved by a mixture of GPT-4o and Claude 3.5 Sonnet models, is 43% []. When given a single attempt, DeepSeek-Coder-V2-Instruct [] achieves only 15.9% on the benchmark. By simply increasing the number of attempts to 250, we increase the fraction of solved problems to 56%, exceeding the single-attempt state-of-the-art by 13%.

In addition to improving model quality, repeated sampling provides a new mechanism for minimizing LLM inference costs. When holding the total number of inference FLOPs constant, we find that on some datasets (e.g. MATH) coverage is maximized with a smaller model and more attempts, while on others (e.g CodeContests) it is better to use a larger model. We also compare API prices between DeepSeek-Coder-V2-Instruct, GPT-4o, and Claude Sonnet 3.5 in the context of solving SWE-bench Lite issues. When keeping the agent framework (Moatless Tools []) constant, sampling five times from the weaker and cheaper DeepSeek model solves more issues than single attempts from Claude or GPT, while also being over 3x cheaper.

Finally, in math word problem settings, where answers cannot be automatically verified by existing tools, we identify a large gap between coverage and the performance of common methods for deciding on a final answer. When solving MATH [] problems with Llama-3-8B-Instruct, coverage increases from 79.8% with 100 samples to 95.3% with 10,000 samples. However, methods such as majority voting and using reward models plateau with a lower sample budget, scaling only from 38.7% to 39.8% over the same range. These results highlight that building robust verifiers remains an open problem.

In summary, our primary observations are:

- 1.
We demonstrate that scaling inference compute through repeated sampling leads to large improvements in coverage across a variety tasks, models, and sample budgets. This makes it possible, and sometimes cost-effective, to amplify weaker models with many samples and outperform single attempts from more capable models. Notably, we are able to solve 56% of issues from SWE-bench Lite by sampling 250 times from DeepSeek-Coder-V2-Instruct, exceeding the single-attempt SOTA of 43%.
- 2.
We show that the relationship between coverage and the number of samples can often be modelled using an exponentiated power law, suggesting a form of scaling laws for inference-time compute.
- 3.
In domains without automatic verifiers, we show that common approaches to verification like majority voting and reward model scoring plateau beyond approximately 100 samples. This leads to a growing gap between the performance achieved with these methods and the coverage upper bound.

## 2 Scaling Repeated Sampling

We focus on pass-fail tasks where a candidate solution can be scored as right or wrong. The primary metric of interest for these tasks is the success rate: the fraction of problems that we are able to solve. With repeated sampling, we consider a setup where a model can generate many candidate solutions while attempting to solve a problem. The success rate is therefore influenced both by the ability to generate correct samples for many problems (i.e. coverage), as well as the ability to identify these correct samples (i.e. precision).

The difficulty of the precision problem depends on the availability of tools for sample verification. When proving formal statements in Lean, proof checkers can quickly identify whether a candidate solution is correct. Similarly, unit tests can be used to verify candidate solutions to coding tasks. In these cases, precision is handled automatically, and improving coverage directly translates into higher success rates. In contrast, the tools available for verifying solutions to math word problems from GSM8K and MATH are limited, necessitating additional verification methods that decide on a single final answer from many (often conflicting) samples.

We consider the following five tasks:

- 1.
GSM8K: A dataset of grade-school level math word problems []. We evaluate on a random subset of 128 problems from the GSM8K test set.
- 2.
MATH: Another dataset of math word problems that are generally harder than those from GSM8K []. Similarly, we evaluate on 128 random problems from this dataset’s test set.
- 3.
MiniF2F-MATH: A dataset of mathematics problems that have been formalized into proof checking languages []. We use Lean4 as our language, and evaluate on the 130 test set problems that are formalized from the MATH dataset.
- 4.
CodeContests: A dataset of competitive programming problems []. Each problem has a text description, along with a set of input-output test cases (hidden from the model) that can be used to verify the correctness of a candidate solution. We enforce that models write their solutions using Python3.
- 5.
SWE-bench Lite: A dataset of real world Github issues, where each problem consists of a description and a snapshot of a code repository []. To solve a problem, models must edit files in the codebase (in the Lite subset of SWE-bench that we use, only a single file needs to be changed). Candidate solutions can be automatically checked using the repository’s suite of unit tests.

Among these tasks, MiniF2F-MATH, CodeContests, and SWE-bench Lite have automatic verifiers (in the form of the Lean4 proof checker, test cases, and unit test suites, respectively).
We begin by investigating how repeated sampling improves model coverage. Coverage improvements correspond directly with increased success rates for tasks with automatic verifiers and in the general case provide an upper bound on the success rate.
In coding settings, our definition of coverage is equivalent to the commonly-used pass@k metric [], where $k$ denotes the number of samples per problem. We use this metric directly when evaluating on CodeContests and SWE-bench Lite. For MiniF2F the metric is similar, with a “pass” defined according to the Lean4 proof checker. For GSM8K and MATH, coverage corresponds to using an oracle verifier that checks if any sample “passes” by outputting the correct final answer. To reduce the variance when calculating coverage, we adopt the unbiased estimation formula from Chen et al. []. In each experiment, we first generate $N$ samples for each problem index $i$ and calculate the number of correct samples $C_{i}$. We then calculate the pass@k scores at each $k\leq N$ of interest according to:

$$ pass@k $\displaystyle=\frac{1}{\text{\# of problems}}\sum_{i=1}^{\text{\# of problems} }\left(1-\frac{\binom{N-C_{i}}{k}}{\binom{N}{k}}\right)$ (1) $$

We use the numerically stable implementation of the above formula suggested in Chen et al. [].
Data and code is available at [https://scalingintelligence.stanford.edu/pubs/large_language_monkeys/](https://scalingintelligence.stanford.edu/pubs/large_language_monkeys/).

### 2.1 Repeated Sampling is Effective Across Tasks

Figure: Figure 2: Across five tasks, we find that coverage (the fraction of problems solved by at least one generated sample) increases as we scale the number of samples. Notably, using repeated sampling, we are able to increase the solve rate of an open-source method from 15.9% to 56% on SWE-bench Lite.
Refer to caption: x2.png

Here, we establish that repeated sampling improves coverage across multiple tasks and a range of sample budgets. We evaluate Llama-3-8B-Instruct and Llama-3-70B-Instruct on CodeContests, MiniF2F, GSM8K, and MATH, generating 10,000 independent samples per problem. For SWE-bench Lite, we use DeepSeek-Coder-V2-Instruct [], as the required context length of this task exceeds the limits of the Llama-3 models. As is standard when solving SWE-bench issues, we equip our LLM with a software framework that provides the model with tools for navigating through and editing codebases. In our work, we use the open-source Moatless Tools library []. Note that solving a SWE-bench issue involves a back-and-forth exchange between the LLM and Moatless Tools. One sample/attempt for this benchmark refers to one entire multi-turn trajectory. To minimize costs, we restrict the number of attempts per issue to 250, with all attempts made independently of one another.

We report our results in Figure 2. We also include the single-attempt performance of GPT-4o on each task, as well the single-attempt state-of-the-art for SWE-bench Lite (CodeStory Aide [] which uses a combination of GPT-4o and Claude 3.5 Sonnet). Across all five tasks, we find that coverage smoothly improves as the sample budget increases. When all LLMs are given a single attempt, GPT-4o outperforms the Llama and DeepSeek models at every task. However, as the number of samples increases, all three of the weaker models exceed GPT-4o’s single-attempt performance. In the case of SWE-bench Lite, we solve 56% of problems, exceeding the single-attempt SOTA of 43%.

### 2.2 Repeated Sampling is Effective Across Model Sizes and Families

Figure: Figure 3: Scaling inference time compute via repeated sampling leads to consistent coverage gains across a variety of model sizes (70M-70B), families (Llama, Gemma and Pythia) and levels of post-training (Base and Instruct models).
Refer to caption: x3.png

The results from Section 2.1 indicate that repeated sampling improves coverage. However, we only show this trend for three recent, instruction-tuned models with 8B or more parameters. We now show that these trends hold across other model sizes, families, and levels of post-training. We expand our evaluation to include a broader set of models:

- •
Llama 3: Llama-3-8B, Llama-3-8B-Instruct, Llama-3-70B-Instruct.
- •
Gemma: Gemma-2B, Gemma-7B [].
- •
Pythia: Pythia-70M through Pythia-12B (eight models in total) [].

We restrict evaluation to the MATH and CodeContests datasets to minimize inference costs, reporting results in Figure 3. Coverage increases across almost every model we test, with smaller models showing some of the sharpest increases in coverage when repeated sampling is applied. On CodeContests, the coverage of Gemma-2B increases by over 300x, from a pass@1 of 0.02% to a pass@10k of 7.1%.
Similarly, when solving MATH problems with Pythia-160M, coverage increases from a pass@1 of 0.27% to a pass@10k of 57%.

The exception to this pattern of increasing coverage across models is with the Pythia family evaluated on CodeContests. All Pythia models achieve zero coverage on this dataset, even with a budget of 10,000 samples. We speculate that this due to Pythia being trained on less coding-specific data than Llama and Gemma.

### 2.3 Repeated Sampling Can Help Balance Performance and Cost

One takeaway from the results in Sections 2.1 and 2.2 is that repeated sampling makes it possible to amplify a weaker model’s capabilities and outperform single samples from stronger models.
Here, we demonstrate that this amplification can be more cost-effective than using a stronger, more expensive model, providing practitioners with a new degree of freedom when trying to jointly optimize performance and costs.

We first consider FLOPs as a cost metric, examining the Llama-3 results from Section 2.1. We re-plot our results from Figure 2, now visualizing coverage as a function of total inference FLOPs instead of the sample budget. Since Llama-3 models are dense transformers where the majority of parameters are used in matrix multiplications, we approximate inference FLOPs with the formula:

$$ FLOPs per token $\displaystyle\approx 2*\left(\text{num parameters}+2*\text{num layers}*\text{ token dim}*\text{context length}\right)$ total inference FLOPs $\displaystyle\approx\text{num prompt tokens}*\text{FLOPs per token}$ $\displaystyle+\text{num decoded tokens}*\text{FLOPs per token}*\text{num completions}$ $$

We present our re-scaled results for MiniF2F, CodeContests, MATH, and GSM8K in Figure 4. Interestingly, the model that maximizes coverage varies with the compute budget and task. On MiniF2F, GSM8K and MATH, Llama-3-8B-Instruct always obtains higher coverage than the larger (and more expensive) 70B model when the FLOP budget is fixed.
However for CodeContests, the 70B model is almost always more cost effective.
We note that examining FLOPs alone can be a crude cost metric that ignores other aspects of system efficiency [].
In particular, repeated sampling can make use of high batch sizes and specialized optimizations that improve system throughput relative to single-attempt inference workloads [, , ].
We discuss this in more detail in Section 5.

Figure: Figure 4: Comparing cost, measured in number of inference FLOPs, and coverage for Llama-3-8B-Instruct and Llama-3-70B-Instruct. We see that the ideal model size depends on the task, compute budget, and coverage requirements. Note that Llama-3-70B-Instruct does not achieve 100% coverage on GSM8K due to an incorrectly labelled ground truth answer: see Appendix E.
Refer to caption: x4.png

We also examine the dollar costs of repeated sampling when solving SWE-bench Lite issues using current API pricing. Keeping the agent framework (Moatless Tools) constant, we consider making a single attempt per issue with Claude 3.5 Sonnet and GPT-4o, as well as repeated sampling using DeepSeek-Coder-V2-Instruct. We report the average cost per issue and issue resolution rate with each approach in Table 1. While the DeepSeek model is weaker than the GPT and Claude models, it is also over 10x cheaper. In this case, repeated sampling provides a cheaper alternative to paying a premium for access to strong models while achieving a superior issue solve rate.

**Table 1: Comparing API cost (in US dollars) and performance for various models on the SWE-bench Lite dataset using the Moatless Tools agent framework. When sampled more, the open-source DeepSeek-Coder-V2-Instruct model can achieve the same issue solve-rate as closed-source frontier models for under a third of the price.**
| Model | Cost per<br>attempt<br>(USD) | Number of<br>attempts | Issues<br>solved (%) | Total cost<br>(USD) | Relative<br>total cost |
| --- | --- | --- | --- | --- | --- |
| DeepSeek-Coder-V2-Instruct | 0.0072 | 5 | 29.62 | 10.8 | 1x |
| GPT-4o | 0.13 | 1 | 24.00 | 39 | 3.6x |
| Claude 3.5 Sonnet | 0.17 | 1 | 26.70 | 51 | 4.7x |

## 3 Characterizing the Benefits of Repeated Sampling

The relationship between an LLM’s loss and its training compute has been well-characterized with training scaling laws [, , ].
These laws have empirically held over many orders of magnitude and inspire confidence in model developers that large investments in training will pay off.
Inspired by training scaling laws, here we aim to better characterize the relationship between coverage and the sample budget (i.e. the amount of inference compute), presenting two interesting observations:

- 1.
The relationship between coverage and the number of samples can often be modelled with an exponentiated power law.
- 2.
For a given task, the coverage curves of different models from the same family resemble S-curves with similar slopes but distinct horizontal offsets.

### 3.1 Scaling Laws for Repeated Sampling

Figure: Figure 5: The relationship between coverage and the number of samples can be modelled with an exponentiated power law for most tasks and models. We highlight that some curves, such as Llama-3-8B-Instruct on MiniF2F-MATH, do not follow this trend closely.
Refer to caption: x5.png

Here, we develop an explicit model for the relationship between coverage and the number of samples. The GPT-4 technical report [] finds that the relationship between a model’s mean-log-pass-rate on coding problems and its training compute can be modelled well using a power law. We start by adopting the same function class, but now modelling the log of coverage $c$ as a function of the number of samples $k$:

$$ $\displaystyle\log(c)\approx ak^{-b}$ (2) $$

where $a,b\in\mathbb{R}$ are fitted model parameters.
In order to directly predict coverage, we exponentiate both sides, ending up with the final model of:

$$ $\displaystyle c\approx\exp(ak^{-b})$ (3) $$

We provide examples of fitted coverage curves in Figure 5, and additional curves in Appendix C.2. While these laws are not as exact as training scaling laws (most strikingly on MiniF2F-MATH), they provide encouraging early evidence that the benefits of inference scaling can be characterized.

### 3.2 Similarities in Coverage Curves Across Models

Figure: Figure 6: Overlaying the coverage curves from different models belonging to the same family. We perform this overlay by horizontally shifting every curve (with a logarithmic x-axis) so that all curves pass through the point (1, $c$). We pick $c$ to be the maximum pass@1 score over all models in the plot. We note that the similarity of the curves post-shifting shows that, within a model family, sampling scaling curves follow a similar shape.
Refer to caption: x6.png

Interestingly, when comparing the coverage curves (with a logarithmic x-axis) of different models from the same family on the same task (see Figure 3), it appears that the traced S-curves have the same slope, but unique horizontal offsets. To investigate this further, we overlay the coverage curves of different models from the same family in Figure 6. We do this by picking an anchor coverage value $c$, and shifting every curve leftward (in log-space) so that each passes through the point (1, $c$). This corresponds to a leftward shift by $\log(\text{pass@k}^{-1}(c))$, where $\text{pass@k}^{-1}(c)$ denotes the closest natural number $k$ such that $\text{pass@k}=c$. We pick $c$ to be the maximum pass@1 score over all models from the same family. These similarities demonstrate that across models from the same family, the increase in the log-sample-budget (or equivalently, the multiplicative increase in the sample budget) needed to improve coverage from $c$ to $c^{\prime}$ is approximately constant.

## 4 Harnessing Repeated Sampling Requires Precision

So far, we have focused on measuring model coverage, characterizing the benefits of repeated sampling under the best-case scenario where we can always identify correct model samples. We now turn to the complementary problem of precision: given a collection of model samples, can we identify the correct ones? In Section 4.1, we evaluate two common verification methods (majority voting and reward model scoring) on GSM8K and MATH. Additionally, in Section 4.2, we discuss potential pitfalls when relying on unit tests to identify correct software programs.

### 4.1 Common Verification Methods Don’t Always Scale with the Sample Budget

Figure: Figure 7: Comparing coverage (performance with an oracle verifier) to mainstream methods available for picking the correct answer (majority voting, reward model selection and reward model majority voting) as we increase the number of samples. Although near-perfect coverage is achieved, all sample selection methods fail to reach the coverage upper bound and saturate before reaching 100 samples. For every k value, we calculate the metric on 100 subsets of size k then plot the mean and one standard deviation across subsets.
Refer to caption: x7.png

Of the five tasks we evaluate, only GSM8K and MATH lack tools for automatically verifying solutions. Here, we evaluate two common approaches to deciding on a final answer: calculating a majority vote across samples and using a reward model to assign a score to each sample. We test these techniques on their ability to identify correct solutions from the 10,000 samples we generated with Llama-3-8B-Instruct and Llama-3-70B-Instruct in Section 2. We benchmark three methods:

- 1.
Majority Vote: We pick the most common final answer [].
- 2.
Reward Model + Best-of-N: We use a reward model [] to score each solution, and pick the answer from the highest-scoring sample.
- 3.
Reward Model + Majority Vote: We calculate a majority vote where each sample is weighted by its reward model score.

We use ArmoRM-Llama3-8B-v0.1 [] as a reward model, which currently has the highest reasoning score on the RewardBench leaderboard [] among open-weight models. We report our results in Figure 7 as we increase the number of samples. While success rate initially increases with the number of samples for all three methods, it plateaus around 100 samples. Meanwhile, coverage continues to increase with the number of samples and exceeds 95%.

In the case of majority voting, this success rate saturation is easily explainable. As the number of samples increases, the proportion of votes allocated to each answer stabilizes, and therefore the success rate plateaus. For some GSM8K and MATH problems, correct solutions are sampled with a probability of 1% or lower (see Figure 8), making them a minority of samples. As the number of samples increases, rare correct solutions will appear for more problems, increasing coverage but not the success rate with majority voting. In order to fully benefit from repeated sampling, sample identification methods must be able to solve these “needle-in-a-haystack” cases and identify rare, correct samples.

Given the poor performance of the reward model, it is reasonable to wonder how “hard” it is to verify a candidate solution. With GSM8K and MATH, only a sample’s final answer is used for assessing correctness, with the intermediate chains of thought being excluded. If models generated only non-sensical chains of thought before guessing a correct final answer, verification may not be any easier than solving the problem in the first place. We investigate this question by manually evaluating 105 chains of thought from correct Llama-3-8B-Instruct solutions to GSM8K problems, reporting our results in Table 2.
We find that over 90% of the chains of thought that we graded are faithful, even among problems where correct answers are generated infrequently.
These correct reasoning steps indicate that there is signal for a verifier to exploit when identifying correct samples. Interestingly, during this process we also identified one GSM8K problem that has an incorrect ground truth answer (see Appendix E).
This incorrect GSM8K problem is also the only one that Llama-3-70B-Instruct did not generate a “correct” sample for across 10,000 attempts.

**Table 2: Human evaluation of the validity of the Chain-of-Thought reasoning in Llama-3-8B-Instruct answers to GSM8K problems. 3 chains of thought were graded per problem. Even for difficult questions, where the model only gets $\leq 10\%$ of samples correct, the CoTs almost always follow valid logical steps. For the model generations and human labels, [see here](https://docs.google.com/spreadsheets/d/1D-suvkheNA4fjLsO2TuwHNqwx2TIECmp).**
| Pass@1 | # Problems | # CoT Graded | Correct CoT | Incorrect CoT | Incorrect Ground Truth |
| --- | --- | --- | --- | --- | --- |
| 0-10% | 5 | 15 | 11 | 1 | 1 problem, 3 CoTs |
| 10-25% | 10 | 30 | 27 | 3 | 0 problems |
| 25-75% | 29 | 30 | 28 | 2 | 0 problems |
| 75-100% | 84 | 30 | 30 | 0 | 0 problems |

### 4.2 Verifiers and Software Tasks: Two Cautionary Tales

Software development tasks can occupy a middle-ground with respect to available verification tools.
On one hand, the ability to execute and test code allows for a higher degree of automatic verification than is possible with unstructured language tasks.
However, tools like unit tests take a black-box approach to verifying a piece of code and are not as comprehensive as methods like proof checkers.
These imperfections in the verification process can lead to false positives and/or false negatives that are important to consider when applying repeated sampling. Below we provide two examples of software verifier imperfections that we encountered when generating our results from Section 2.1.

#### 4.2.1 Flaky Tests in SWE-bench Lite

When producing our results on SWE-bench Lite, we identified that 11.3% of problems have flaky test suites that do not produce consistent results when running them on the same candidate solution.
These flaky tests occasionally classify even the dataset’s ground-truth issue solutions as incorrect. Additionally, the test suites for some issues can be non-determinstic depending on the candidate solution.
For example, two SWE-bench Lite issues involve manipulating Python sets, which are naturally unordered.
The gold solutions for these issues explicitly order the items in the set and pass the test suites reliably. However, some model-generated candidate solutions do not impose such an ordering, and therefore pass the tests on some “lucky” runs and not others.
In Appendix B, we list all of the problem IDs where we identified flaky tests. We also report our SWE-bench Lite results from Figure 2 with the problematic issues removed, finding similar results to our evaluations on the whole dataset.

#### 4.2.2 False Negatives in CodeContests

Each problem from the CodeContests dataset comes with a set of input-output test cases used to asses the correctness of solutions. These test cases are more comprehensive than those from earlier coding benchmarks like APPS [], cutting down on the frequency of false positive solutions that pass all test cases but do not fully solve the described problem. However, the construction of the CodeContests test suites leads to false negative solutions that are correct but fail the tests.

For some CodeContests problems, the problem description allows for multiple distinct correct outputs for a given test input. However, the corresponding test cases do not handle these scenarios, instead requiring that one particular correct output is emitted. Additionally, many CodeContests test cases have been programmatically generated by mutating original test cases from the problem. Some mutated inputs violate the problem’s input specifications (e.g. a mutated input being zero when the description promises a positive integer).
These malformed test cases can lead to inconsistent behaviour between different correct solutions.

We assess the prevalence of these issues by running each problem’s test suite on the list of correct solutions that CodeContests provides. Of the 122 problems in the test set that have Python3 solutions, we find that 35 problems have “correct” solutions that fail the corresponding tests. Since we do not allow models to view all of a problem’s test cases (and their peculiarities), applying repeated sampling to these problems contains an element of “rolling the dice” to generate a solution that is not only correct, but emits the particular outputs that pass the tests.

Figure: Figure 8: Bar charts showing the fraction of samples (out of 10,000 samples) that are correct, for each problem in the subsets of GSM8K and MATH we evaluate on. There is one bar per problem, and the height of the bar corresponds to the fraction of samples that arrive at the correct answer. Bars are green if self-consistency picked the correct answer and are red otherwise. We highlight that there are many problems with correct solutions, where the correct solutions are sampled infrequently.
Refer to caption: extracted/5858723/figures/hist2.png

## 5 Discussion and Limitations

In this work, we explore repeated sampling as an axis for scaling compute at inference time in order to improve model performance. Across a range of models and tasks, repeated sampling can significantly improve the fraction of problems solved using any generated sample (i.e. coverage). When correct solutions can be identified (either with automatic verification tools or other verification algorithms), repeated sampling can amplify model capabilities during inference. This amplification can make the combination of a weaker model and many samples more performant and cost-effective than using fewer attempts from a stronger, more expensive model.

Improving Repeated Sampling: In our experiments, we explore only a simple version of repeated sampling where all attempts to a problem are generated independently of one another using the exact same prompt and hyperparameters. We believe that this setup can be refined to improve performance, particularly along the following directions:

- 1.
Solution Diversity: We currently rely on a positive sampling temperature as the sole mechanism for creating diversity among samples. Combining this token-level sampling with other, higher-level approaches may be able to further increase diversity. For example, AlphaCode conditions different samples with different metadata tags.
- 2.
Multi-Turn Interactions: Despite automatic verification tools being available when solving CodeContests and MiniF2F problems, we use only a single-turn setup where models generate a solution without any ability to iterate on it. Providing models with execution feedback from these tools should improve solution quality. We are interested in the tradeoffs associated with multi-turn interactions, since each attempt becomes more expensive, but also may be more likely to succeed.
- 3.
Learning From Previous Attempts: Currently, our experiments fully isolate attempts from each other. Access to existing samples, particularly if verification tools can provide feedback on them, may be helpful when generating future attempts.

Repeated Sampling and Inference Systems: Repeated sampling is a distinct LLM inference workload from serving chatbot requests. Production chatbot deployments place an emphasis on low response latencies, and adhering to latency targets can force a lower per-device batch size and reduce hardware utilization. In contrast, when sampling many completions to a single prompt, a larger emphasis can be placed on overall throughput and maximizing hardware utilization. Additionally, repeated sampling can benefit from specialized attention optimizations that exploit overlaps in prompts across sequences [, , ]. Repeated sampling inference can therefore be accomplished at a lower cost than naively making many parallel requests to a chatbot-oriented API. These cost savings can further motivate choosing to sample many times from a cheaper model instead of fewer times from a more expensive one.

Verifiers: Our results from Section 4 highlight the importance of improving sample verification methods when tools for automatically doing so are unavailable. Equipping models with the ability to assess their own outputs will allow repeated sampling to be scaled to far more tasks. Of particular interest is applying repeated sampling to unstructured tasks like creative writing, which can require a more subjective comparison between different samples than the pass-fail tasks we consider. An alternative direction to developing model-based verifiers is to design converters that can make an unstructured task verifiable, for example by formalizing an informal math statement into a language like Lean so that proof checkers can be applied.

## 6 Related Work

Scaling Inference Compute: Methods that perform additional computation during inference have been successful across many areas of deep learning. Across a variety of game environments, state-of-the-art methods leverage inference-time search to examine many possible future game states before deciding on a move [, , ]. Similar tree-based methods can also be effective in combination with LLMs, allowing models to better plan and explore different approaches [, , , ]. Another axis for increasing LLM inference compute allows models to spend tokens deliberating on a problem before coming to a solution [, , ]. Additionally, multiple models can be ensembled together at inference time to combine their strengths [, , , , ]. Yet another approach involves using LLMs to critique and refine their own responses [, ].

Repeated Sampling: Previous work has demonstrated that repeated sampling can improve LLM capabilities in multiple domains. One of the most effective use cases is coding [, , ], where performance continues to scale up to a million samples and verification tools (e.g. unit tests) are often available to automatically score every candidate solution.
Recently, Greenblatt [] shows that repeated sampling is effective when solving puzzles from the ARC challenge [], observing log-linear scaling as the number of samples increases.
In chat applications, repeated sampling combined with best-of-N ranking with a reward model can outperform greedily sampling a single response [].
In domains without automatic verification tools, existing work shows that using majority voting [], prompting an LLM [], or training a model-based verifier [, , , , ], to decide on a final answer can improve performance on reasoning tasks relative to taking a single sample. Nguyen et al. [] finds that performing majority voting over answers that exceed a threshold length can outperform voting across all answers. Concurrent with our work, Song et al. [] finds that using the best available sample improves LLM performance on chat, math, and code tasks, sweeping up to a max of 128 samples. Additionally, Hassid et al. [] find that when solving coding tasks, it can be more effective to draw more samples from a smaller model than draw fewer samples from a larger one.

Scaling Laws: Characterizing how scaling affects model performance can lead to more informed decisions on how to allocate resources. Scaling laws for LLM training find a power law relationship between loss and the amount of training compute and provide estimates for the optimal model and dataset size given a fixed compute budget  [, , ].
Jones [] finds scaling laws in the context of the board game Hex, observing that performance scales predictably with model size and the difficulty of the problem.
Interestingly, they also show that performance scales with the amount of test-time compute spent while performing tree search.
Recently, Shao et al. [] observe scaling laws when augmenting LLMs with external retrieval datasets, finding that performance on retrieval tasks scales smoothly with the size of the retrieval corpus.

## 7 Acknowledgements

We thank Together AI for partially sponsoring the compute for this project, as well as Rahul Chalamala and Ben Athiwaratkun for their help managing this infrastructure. We thank John Yang for his advice and support when running our SWE-bench experiments. Finally, we are grateful to Mayee Chen, Neel Guha, Quinn McIntyre, Jon Saad-Falcon, and Benjamin Spector for their helpful discussions and feedback throughout this project.

We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF2247015 (Hardware-Aware), CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under Nos. W911NF-23-2-0184 (Long-context) and W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under Nos. N000142312633 (Deep Signal Processing); Stanford HAI under No. 247183; NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Meta, Google, and VMWare. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.

This work was completed with the support of the Clarendon Fund Scholarships.

## Appendix A Sampling Experimental Setup

### A.1 Lean Formal Proofs

We report results on the $130$ questions in the test set of the [lean4 MiniF2F dataset](https://github.com/rah4927/lean-dojo-mew/blob/main/MiniF2F/Test.lean) that correspond to formalized MATH problems. This dataset is derived from the [fixed version](https://github.com/facebookresearch/miniF2F) of the original MiniF2F dataset created by Zheng et al. []. We sample with a temperature of $0.5$ and do not use nucleus sampling. We generated $10,000$ samples per problem. We use proofs of the following 5 theorems from the [validation set](https://github.com/rah4927/lean-dojo-mew/blob/main/MiniF2F/Validation.lean) as few-shot examples:

- •
mathd_algebra_116
- •
amc12_2000_p5
- •
mathd_algebra_132
- •
mathd_algebra_11
- •
mathd_numbertheory_84

Our prompt consists of:

- 1.
Few shot examples.
- 2.
Header imports present in each problem in the HuggingFace dataset cat-searcher/minif2f-lean4 dataset, an upload of the lean4 MiniF2F dataset.
- 3.
The theorem definition. In order to avoid leaking information about how to solve the theorem from its name, we replace the name of the theorem with theorem_i. $i\in\{1,2,3,4,5\}$ for the few-shot examples and $i=6$ for the current problem.

We set $200$ as the max token length for the generated solution. To grade solutions, we use the lean-dojo 1.1.2 library with lean version 4.3.0-rc2. We set a timeout of $10$ seconds for every tactic step.

### A.2 CodeContests

We report results on the $140$ test set questions that do not include image tags in the problem description. We sample with a temperature of $0.6$ and a top-p value of $0.95$ following the experiments in CodeLlama []. We generate 10,000 samples per problem. We use two few-shot examples from the training set that are randomly sampled per-problem. We set $1024$ as the max token length for the generated solution. We use the same answer comparison function as [] and use the concatenation of public, private, and generated tests to validate correctness of solutions.

### A.3 MATH

We report results on $128$ randomly selected test-set problems. We sample with a temperature of $0.6$ and do not use nucleus sampling. We use the fixed $5$ few-shot example from [] for each problem. We generate $10,000$ samples per problem. We set $512$ as the max token length for the generated solution. To grade solutions, we use the minerva_math functions from LMEval [] to extract the model’s final answer. We then check correctness if the extracted answer is an exact string match to the ground truth, or if the is_equiv function from minerva_math in LMEval evaluates to true.

### A.4 GSM8K

We report results on $128$ randomly sampled test-set problems. We sample with a temperature of $0.6$ and do not use nucleus sampling. We use $5$ few-shot examples from the training set that are randomly sampled per-problem. We generate $10,000$ samples per problem. We set $512$ as the max token length for the generated solution. To grade solutions, we follow LMEval [] and extract answers using a regular expression that extracts the string after the quadruple hashes. Similar to MATH, we then assess correctness by checking if the extracted answer is an exact string match to the ground truth or if is_equiv evaluates to true.

## Appendix B SWE-bench Lite

### B.1 Experimental Setup

For our experiments, we use DeepSeek-Coder-V2-Instruct with the Moatless Tools agent framework (at commit a1017b78e3e69e7d205b1a3faa83a7d19fce3fa6).
We use Voyage AI [] embeddings for retrieval, the default used by Moatless Tools. We make no modifications to the model or framework, using them entirely as off-the-shelf components.

With this setup, we sample 250 independent completions for each problem using standard temperature-based sampling. To determine the optimal sampling temperature, we conducted a sweep on a random subset of 50 problems from the test set, testing temperatures of 1.0, 1.4, 1.6, and 1.8. Based on these results, we selected a temperature of 1.6 for our main experiments.

### B.2 Test Suite Flakiness

During our analysis, we identified 34 problems in SWE-bench Lite whose test suites had flaky tests. Using the SWE-bench testing harness provided by the authors of SWE-bench, we tested each solution repeatedly: for some solutions, sometimes the solution was marked as correct, and other times it was marked as incorrect. In 30 of these 34 cases, we observed flakiness even on the correct solutions provided by the dataset authors. Table 3 lists the problem IDs of the 34 instances with flaky tests.

**Table 3: Instance IDs of problems from SWE-bench Lite that have flaky tests.**
| Repository | Instance IDs |
| --- | --- |
| django | django__django-13315, django__django-13447, django__django-13590,<br>django__django-13710, django__django-13757, django__django-13933,<br>django__django-13964, django__django-14017, django__django-14238,<br>django__django-14382, django__django-14608, django__django-14672,<br>django__django-14752, django__django-14915, django__django-14997,<br>django__django-14999, django__django-15320, django__django-15738,<br>django__django-15790, django__django-15814, django__django-15819,<br>django__django-16229, django__django-16379, django__django-16400,<br>django__django-17051 |
| sympy | sympy__sympy-13146, sympy__sympy-13177, sympy__sympy-16988 |
| requests | psf__requests-863, psf__requests-2317,<br>psf__requests-2674, psf__requests-3362 |
| scikit-learn | scikit-learn__scikit-learn-13241 |
| matplotlib | matplotlib__matplotlib-23987 |

An additional instance, astropy__astropy-6938, was flaky on some machines and not others. The authors of SWE-bench were able to reproduce the flakiness; however, we were unable to. Our preliminary investigation indicates this specific issue is due to unpinned versions of dependencies in the docker environments that run the unit tests.

Here, we include results on a subset with the problems in Table 3 removed (266 problems). For the full dataset evaluation, on any problem that has flaky tests, we run the test suite 11 times and use majority voting to determine whether a solution passed or failed. For the evaluation on the subset without flaky tests, all baselines we compare against release which problems they correctly solve, so we simply removed the problems with flaky tests and recomputed their scores.

Figure: Figure 9: SWE-bench Lite results, without and with problems that have flaky tests. For the graph on the left, all problems in Table 3 are excluded. For the graph on the right, all problems are included. We note that the trend is the same with or without the flaky tests.
Refer to caption: x8.png

## Appendix C Scaling Law Details

### C.1 Experimental details

To fit exponentiated power laws to coverage curves, we first sample $40$ points spaced evenly along a log scale from 0 0 to $10,000$ and remove duplicates. We then use SciPy’s [] curve_fit function to find the $a$ and $b$ parameters from Equation 3 that best fit these points.

### C.2 Additional results

In Figure 10, we show additional results fitting power laws to coverage curves for an expanded set of datasets and models.

Figure: Figure 10: Fitting exponentiated power laws to coverage curves for an expanded set of tasks and models.
Refer to caption: x9.png

## Appendix D Precision Details

To calculate the Majority Vote, Reward Model + Best-of-N and Reward Model + Majority Vote metrics, we use the same $128$ problem subsets for both MATH and GSM8K datasets introduced in Section 2.
Each problem corresponds to 10,000 samples for each model we test.
For each verification method, we take $100$ random subsets of size $k$ and calculate the success rate using each subset.
We report the mean and standard deviation across subsets in Figure 7.
To calculate the Majority Vote answer, we take the plurality answer in each subset (note that two answers are considered equivalent if they are exact string matches or if is_equiv evaluates to true).
For the Reward Model + Best-of-N, we take the answer with the highest score assigned by the reward model.
For the Reward Model + Majority Vote metric, we sum the reward model score across all the samples with the same final answer, and use the final answer with the highest sum.

## Appendix E GSM8K incorrect answer

As discussed in 4.1, we identify that [a problem in the GSM8K test set (index 1042 on HuggingFace)](https://huggingface.co/datasets/openai/gsm8k/viewer/main/test?row=1042) has an incorrect ground truth solution.

The mistake is in the second line of the answer: on the third race, Johnny’s dad lost $$16.5$, not $$15$, meaning he made $$11$ and lost $\$16.5+\$5=\$21.5$. So, the answer is an average loss of $\$3.5$ per race, not $\$3$ per race (the answer in the dataset).

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 6
presentation: 2
questions: - Maybe I missed discussion on this but is there any attempt to also parameterize the coverage scaling law in terms of the model size?
- Can the authors compare their work with parallel work from Snell et. al. 2024 (Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters)? In particular it would be nice to see discussion on whether the exponentiated power law is also a reasonable to fit scaling curves generated by forms of test-time compute methods, like sequential corrections explored by Snell et al.
soundness: 2
strengths: - The paper presents an extensive analysis of coverage on math and coding tasks and plots empirical scaling laws (and fitted ones) on a multiple model families like Gemma, Pythia and Llama. One of the most interesting trends to note is that models from the same family only affected the offset of the log-linear plot, and not the slope. This is different from typical scaling laws for pre-training loss seen in prior works.
- The authors also extend analysis to inference time compute, and show that sometimes drawing multiple samples from a smaller (and less capable model) is more compute-efficient (in terms of FLOPs), compared to a single sample from a larger and more capable model. This analysis can be immediately used to reduce the inference time for models deployed in practice, without any loss in performance, at least when a reliable verifier is available. 
- The analysis in Figure 6,7 is particularly compelling to put more effort in improving verification, since it suggests that most test-time methods for reasoning/safety etc., are not failing due to lack of coverage, but more due to the inaccuracy of the verifier.
summary: The paper studies scaling laws for a new axis of compute for LLMs: inference time compute. They empirically find that coverage (pass@N) improves with the number of inference time samples log-linearly and can be modeled with an exponentiated power law. 
In domains like coding, where automatic verifiers exist, i.e., verification is much easier than generation, the performance improves from 15.9% with one sample to 56% with 250 samples, outperforming the single-sample state-of-the-art of 43%. In domains lacking automatic verifiers, the authors observe that typical approaches for selecting from a set of IID sampled responses, such as majority voting and reward models, reach a performance plateau after several hundred samples and do not effectively scale with an increasing sample budget, and thus the performance on these problems is bottlenecked by the accuracy of the verifier.
weaknesses: - While the coverage analysis expands on different models and tasks, it does not model the affect of other common parameters of interest like temperature, top-K etc. 
- Some recent works like (Snell et al. 2024, Setlur et al. 2024) show that beam search against an automated verifier improves compute efficiency, with a fixed beam size. Having analysis on this direction of compute use would make the paper stronger.
- While the authors fit scaling laws for coverage, it would be much more useful to see if scaling laws can also be fit for the setting with trained verifiers, that may not be perfect, accounting for the error in the verifier. Currently it is unclear how the size/data used for the trained verifier affects the compute scaling laws for best-of-n inference. 
- In general, the paper presents interesting results for best-of-N inference, but they are quite narrow and expected. Broadening the laws to other forms of compute usage, or identifying how the "learnability" of a verifier affects these laws can help to make the work more complete.

[1] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (C. Snell, J. Lee, K. Xu, A. Kumar)
[2] Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, A. Kumar)

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: (See above)
soundness: 3
strengths: * While it isn't surprising that repeated sampling improves performance, the authors have quantified this phenomena in a precise way.  In particular, they find a relatively clean scaling relation between the coverage and number of samples.
* There is good diversity in the models and datasets used, from more "agentic" tasks like SWE-bench to standard math/code reasoning.
summary: This work considers the question of scaling inference compute through repeated sampling.  The authors find that repeated sampling can greatly improve the performance on reasoning/code tasks, particularly when there is an external verifier that can be used to check the result.  At a high level, this work targets an interesting research question and the methodology looks sound.  However, parts of the paper are poorly written/confusing (particularly in explicating the experimental setup + results with/without the verifier), and could benefit from some major revisions.  I am willing to raise my score if the authors address my questions/concerns.
weaknesses: * The authors should include a few more details about the experimental setup or streamline the existing writing.  For example, one may have chosen a couple of different approaches on how to incorporate the verifier: the first is an iid setup, where the verifier is just used to pick the final answer out of several of attempts.  The second is where the model is provided some verifier signal (e.g., is the answer right/wrong?) between attempts, and asked to review its reasoning trace.  Both scenarios interesting, but there were some experimental choices made in the paper that bear some justification.  
* The improved performance with repeat sampling is not meaningful in and of itself; if you have enough attempts, the fraction that you will get right will of course go up.  However, it may be useful for the reader to show an example of repeat sampling where the model gets the right answer after a small number of samples -- which part of the reasoning trace gets "fixed" between samples typically?  Can we characterize the mistakes that the models make better?
* I am a bit unclear on the relevant baseline here.  How should we compare spending the inference compute on sampling (versus, say, doing X-of-thought approaches)?
* A common assumption in the literature is that verification is easier than generation.  Interestingly, this work seems to show that while non-oracle verification can provide some wins, the improvement is overall quite small.  It would be interesting to dig into this question a bit more, and study some explicit examples where the reward model (or majority vote) fails.  How much of the lack of improvement is due to the inherent noisiness of the reward model?  Can the authors characterize the failure mode here a bit more precisely (is non-oracle verification almost as hard as generation in this case)?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: See weaknesses
soundness: 2
strengths: 1. The paper provides a clear, rigorous study of an important and basic topic in LLM inference. The paper tests many different benchmarks and base models and finds that the main results to be robust. A lot of these may exist in prior work, but not in a coherent and systematic way as presented here. 

2. The paper is well-written and easy to follow.
summary: This paper presents a systematic evaluation of repeated sampling for LLM generation across models and benchmark problems considering both oracle verifiers (pass@k) and reward models to select answers. The paper finds smooth scaling behavior with number of samples across all settings with oracle verifiers and proposes a parametric for the inference time scaling laws, even showing that repeated sampling from smaller models can outperform FLOP-matched sampling from larger models. Finally, the paper finds that there remains a large gap between the performance of oracle verifiers and reward models or other mechanisms like majority voting.
weaknesses: 1. The paper needs to be careful about claims of novelty. The paper does discuss some related work in the intro, but could be more clear that existing work has made many of these observations. The main diffference is that this paper gives a more systematic analysis. For example alphacode [1] and others have figures that look almost identical scaling figures (Fig 6 in alphacode paper), and [2] has similar figures about scaling samples with reward models. 

2. I think the scaling laws presentation is missing the somewhat basic discussion that this is totally the expected behavior for computing pass@k for independent bernoulli samples, no LLMs needed. For example, try running this numpy code and you will reproduce the same kinds of scaling law curves on a log scale:
```
p = 0.001
T = 10000
n_trials = 1000

samples = np.random.random((n_trials, T)) < p
passes = np.cumsum(samples, axis=1) >= 1
pass_rate = np.mean(passes, axis=0)
```
Of course this is a simplification since there should be a different value of p for each problem in the test set, but the basic idea is that when you take independent samples of a bernoulli variable, this is the expected behavior. There is likely a closed form for this simple process. It would probably make sense to model this directly rather than fitting a heuristic scaling law.

3. It is not clear how the presented scaling laws could be useful. The usual usefulness of pre-training scaling laws is to predict the optimal model size at a FLOP budget beyond those used for training. This kind of extrapolation is not tested in the paper. Moreover, the scaling laws are different for each specific model/task combinations and seem cheap to estimate, so it is not clear how prescriptive they are.

4. The stated conclusions about verifiers seem to be too strong given that the paper only uses one reward model that is not particularly tailored to the problems studied. The off-the-shelf RM is chosen for performance on reward bench reasoning, but to my understanding this benchmark is mostly about humaneval-style coding and then the RM is being applied on math reasoning problems. For example, [2] trains task-specific RMs and does not observe the same sort of saturation.

5. Some discussion of related work is missing when discussing how to improve verifiers in the future work directions. For example, [3] proposes an objective for reward modeling to allow LMs to evaluate themselves with a linear model on their representations and [4] uses models to evaluate themselves.

[1] Li, Yujia, et al. "Competition-level code generation with alphacode." Science 378.6624 (2022): 1092-1097.

[2] Lightman, Hunter, et al. "Let's verify step by step." arXiv preprint arXiv:2305.20050 (2023).

[3] Li, Kenneth, et al. "Q-Probe: A Lightweight Approach to Reward Maximization for Language Models." arXiv preprint arXiv:2402.14688 (2024).

[4] Yuan, Weizhe, et al. "Self-rewarding language models." arXiv preprint arXiv:2401.10020 (2024).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Please refer to the **weakness** part.
soundness: 2
strengths: - The research problem is interesting and well-motivated. Scaling up training compute has led to remarkable success in deep learning. It is important to consider scaling inference compute.
- The paper is easy to read.
- The evaluation covers 5 different benchmark datasets and show consistent trends.
summary: This paper studies the potential of scaling inference compute of LLMs. The authors show that, by generating repeated samples from language models, we can increase the “coverage”, i.e., the fraction of problems that are solved by any generated sample. The improved coverage directly translates to better performances when an oracle verifier is available.  The author also conduct experiments in the setting without oracle verifiers and find that the performances plateau quickly given repeated samples.
weaknesses: - One main claim, **scaling inference compute through repeated sampling leads to large improvements in coverage, seems trivial**. I believe that this fact is generally known in the community. Empirically, it has been observed in prior works like [1,2]. Mathematically, it is a simple consequence of equation 1. It is easy to prove that pass@$k$ monotonically increases with $k$ as long as there exists some $C_i>0$. The scaling curves are simply numerical calculation of equation 1 (correct me if this is not the case!). Thus, the novelty of this analysis seems limited.

- The paper **lacks in-depth analysis on scaling laws**. In Section 3, the proposed formula (equation 2) is simply adopted from the GPT-4 technical report. Then the “curve fitting” directly fit the power law to the curve generated by a _known formula_ (i.e., equation 1). I believe that meaningful scaling laws should be distilled from experimental observations. It’s not clear to me what is the insight of fitting some curves when the underlying formula is already known.

- I like the analysis on domains without automatic verifiers since it is a more realistic setting than the experiments based on oracle verifiers.  However, this section is very short and **lacks a deeper exploration of verifiers**. The conclusions here are based on experiments with a single existing 8B verifier. Strengthening this analysis with verifiers of varying sizes and other well-known verification approaches (e.g., process supervision [3]) would significantly enrich this section.

- Minor issues: Line 394: $k$ should be italic.

[1] Program Synthesis with Large Language Models

[2] Competition-level code generation with AlphaCode

[3] Let’s Verify Step by Step

Meta-review: This paper studies scaling laws for LLMs as a function of the inference time budget. It makes empirical observations that coverage improves with increasing inference time budget and can be modeled with an exponentiated power law. For domains with automatic verifiers (e.g., code generation), the paper shows significant improvement in performance with increased inference time budget. For domains without automatic verifiers, the paper shows that performance plateaus after few hundred samples due to inaccurate verifier.

The reviewers liked the experimental findings acknowledging that most of the findings are known to the community. The only non-trivial contribution is the scaling curve and its claimed predictive power. However, two reviewers noted that this part is not well done and the analysis is somewhat insufficient. I agree with this observation.

Therefore, I'm recommending to reject this paper and strongly encourage the authors' to improve the scaling curve and its analysis based on the feedback from reviewers' for resubmission.

Decision: Reject