The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
Authors: Michael Hassid
Tal Remez
Jonas Gehring
Roy Schwartz
Yossi Adi
Venue: COLM (2024)
Keywords: code-generation, budget constraint, model size
TL;DR: We compare between small and large LMs in code-generation tasks when both are given the same budget constraint; we show that smaller models often outperform large ones in this setting

Abstract: It is a common belief that large language models (LLMs) are better than smaller-sized ones. However, larger models also require significantly more time and compute during inference. This begs the question: what happens when both models operate under the same budget? (e.g., compute, run-time). To address this question, we analyze code generation LLMs of various sizes and make comparisons such as running a 70B model once vs. generating five outputs from a 13B model. We consider a standard unit-test setup, which can be used to select the correct output from the smaller model. Our findings reveal that the repeated use of smaller models can yield consistent improvements, with gains of up to 15% across five tasks. On the other hand, in scenarios where unit-tests are unavailable, a ranking-based selection of candidates from the smaller model falls short of the performance of a single output from larger ones. Our results highlight the potential of using smaller models instead of larger ones, and the importance of studying approaches for ranking LLM outputs.

Published as a conference paper at COLM 2024
The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
Michael Hassid Tal Remez* ,Jonas Gehring', Roy Schwartz', Yossi Adi FAIR Team, Meta
2The Hebrew University of Jerusalem (michael. hassid) amail..mmm....aa...........aaaa. hujji. ac. i1
Abstract
It is common belief that large language models (LLMs) are better than smaller-sized ones. However, larger models also require significantly more time and compute during inference. This begs the question: what happens when both models operate under the same budget? (e.g,, compute, run-time). To address this question, we analyze code generation LLMs of various sizes and make comparisons such running a 70B model once vs. generating five outputs from 13B model. We consider standard unit-test setup, which can be used to select the correct output from the smaller model. Our findings reveal that the repeated use of smaller models can yield consistent improvements with gains of up to 15% across five tasks. On the other hand, scenarios where unit-tests are unavailable, a ranking-baas candidates from the smaller model falls short of the performance of a single output from larger ones. Our results highlight the potential of using smaller models instead of larger ones, and the importance of studying approaches for ranking LLM outputs.
selection
Introduction
common wisdom in deep learning, and language modeling in particular, is that investing more compute leads to improved performance (Kaplan et al., 2020). The standard way of implementing this principle training larger models. A simpler, yet often overlooked way to increase compute budget is to run smaller model multiple times, and select the best output using some metric (Chen et al., 2021). In this work we systematically compare between these two approaches: we ask whether, given a fixed compute budget, it is best to run a large model once, or a smaller model multiple times (Figure 1). Our results show that, perhaps surprisingly, given the same compute budget, running 7B or 13B models can not only match the performance of a 70B model, but also substantially surpass it.
Addressing our research question requires method for selecting the best LLM output from a given set of candidates. In this work we focus on execution based oode-generation tasks, which assume the availability of unit-tests (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021). We consider the widely-used pass@k metric (Kulal et al., 2019), which evaluates model's sss performance on code generation problems by generating k outputs and assigning a point if any of them passes all tests. To adapt this metric for our purposes, we take models of different sizes, and for each generate as many outputs as possible given fixed compute budget, e.g., floating point operations (FLOPs) or wall-time.
We apply this setup to evaluate the Code Llama (Roziere al., 2023) model family (7B, 13B, 34B, and 70B) across five tasks: HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and the three splits of APPS (Hendrycks al., 2021). For the HumanEyal and MBPP benchmarks, we additionally use the recent Llama-3 (AI@Meta, 2024) model family (8B and 70B). Surprisingly, we find that for the two popular tasks, HumanBval and MBPP, the
Equal contribution
Data is avalible at https://ggttie http:///thbb.caaaappppa//aateeee
1

---

Published as a conference paper at COLM 2024
Ouaite
x10 Model Size
x1 Generation
VS.
x1 Model Size
x10 Generations
Figure 1: Different ways to improve LLM performance by increasing compute budget. Top: the standard approach of increasing model size, while generating a single output. Bottom: our approach- using small model to generate multiple outputs, and select the best one.
smaller models (7B, 8B and 13B) outperform the larger ones (34B and 70B) margin of up to 15%. Importantly, this is observed using both budget types (FLOPs and wall-time) and across all computation budgets. When considering the challenging APPS benchmark, we find that the 13B model performs best across a almost all budgets, with consistent margin of 5% when considering the hardest competionmtt
We then proceed to examine the scenario where unit-tests are unavailable, such as in an IDE code-completion setup. In such cases, an efficient policy is required to select a single solution from all generated ones. We consider a simple LLM-based policy, which ranks solutions based on the negative log likelihood of the LLM. We also augment this policy with variant of a recent ranking apPEO (Ni et al., 2023). We experiment with the 7B model, and rank its outputs using each of the models. Our results show that, as expected, ranking-baan based selection improves with the increase in compute budget, and with the size of the ranking LLM. Nonetheless, this procedure still falls short of the performance achieved by running the larger model independently with the same budget.
Our results highlight the potential using smaller models instead of larger ones, a practice that has many benefits. First, small models are far computationally cheaper to pre-train." Further, at inference time, they are considerably more hardware-friendly; a 13B model can be accommodated on a single A100 GPU, a feat unachievable for a 70B model (Dettmers et al., 2022). Finally, as we have shown, when controlling for the compute budget, smaller models may actually outperform larger ones.
Our findings also emphasize the importance of developing effective ranking approaches for LLM outputs. This is especially important in cases where no unit-tests or other verification methods are available (Zou et al., 2021; Uesato al., 2022; Sun et al., 2023). To support such research direction, we release 2, 000 Code Llama 7B outputs for each example in HumanEval and MBPP-a total of more than 1M outputs.!
2 Evaluation under Compute Restrictions
To study our main research question-what is the optimal way of using given LLM com- pute budget-we consider a code-generation setup with unit-tests (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021). Below we discuss our methodology for code generation evaluation under computational restrictions. We begin by describing pass@k (Kulal et al., 2019), the current main approach for evaluating code generation tasks (Section 2.1). We then transition to describe our variant of code generation metrics under computational restrictions (Section 2.2).
F.g., 2 7B x10X faster to pre-train compared to the 70B variant (Touvron al., 2023).
2

---

Published as a conference paper at COLM 2024
2.1 Standard Code Generation Evaluation
To evaluate LLM code-gennration abilities, a common setup assumes set of coding ques- tions, each with set of unit-tests. The LLM is fed with each question, and fixed number of output generations (labelled k) are sampled. The evaluation protocol considers each question for which at least one output passes all unit-tests correct. To estimate the per- formance of a model that generates k outputs, it is common to generate larger number of outputs (> k) and compute:
pass@k
(1)
Problems
where S. is the number of examples that pass the unit-tests, The above mentioned metric results in an unbiased estimator as was shown by Chen et al. (2021).
2.2 Comparing LLMs of Different Sizes under a Fixed Budget
Our goal is compare between LLMs of different sizes under a fixed compute budget. To do so, we allow smaller models, which consume fewer resources, to generate more outputs. This results in models of different sizes requiring roughly the same amount of compute.
We consider two types of compute budgets: the number of FLOPs and wall-time. For each type, specific resource limit is set (e.g,, 10k Tera-FLOPs or seconds), and the model generates examples up to the point where the compute limit is reached. That is:
pasSsops @f pass@k where: k max K, flops(k)
(2)
pasStime @! == pass@k where: k == max K,
time(k))
where flops(k) and time(k) are functions that return the FLOPs/wall-ting usage of given model that generates outputs. Notably, the FLOPs restriction is more theoretical com- putational restriction, as it assumes perfect utilization of the hardware. On the other hand, the wall-time restriction more realistic, but is hardware specific, and thus not directly comparable across different machines.
3
Experimental Setup
In this section describe our experimental setup, focusing the code benchmarks used (Section 3.1), our metrics (Section 3.2), and our experiments (Section 3.3).
3.1 Benchmarks
We experiment with three python code benchmarks: HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021) and APPS (Hendrycks et al., 2021). The HumanEval bench- mark consists of 164 function declarations alongside their documentation. The Code-LLM's task is to complete the function according to the provided documentation. MBPP consists of 500 test examples, each one is an instruction for a code function. Here, the Code-LLM is required to generate the full function. Lastly, the test subset of APPS is composed of 5k programming problems at various levels of difficulty: introductory (1k), interview (3k) and competition (1k). In the APPS tasks, the Code-LLM is required to generate the complete python file, which includes import declarations, class definitions, and so on.
3.2 Metrics
Computing the passnops @f and pasStime @t metrics requires an estimation of the flops(k) and time(k) functions from Equations (2) and (3). To estimate FLOPs usage, we use the calflops library (xiaoju ye, 2023), with input sequence length of 128. We measure wall-time while
3

---

Published as a conference paper at COLM 2024
Table 1:
Code Llama FLOPS and wall-time usage per model size, along with normalized
values with respect to the 7B model.
Model Size
FLOPs (Teras)
FLOOPs (norm.)
wall-time (seconds)
wall-time (norm.)
7B
1.69
1.00
395
1.00
13B
3.29
1.95
667
1.69
34B
8.58
5.08
2,994
7.58
70B
17.60
10.41
5,605
14.19
assuming optimal throughput utilization of the hardware. Specifically, we use a node of A100 GPUs, optimize the batch size per model and measure the time it takes each model to generate a subset of 1k examples from our datasets. We report the Code Llama results in Table 1, for readability we also report the normalized factor with respect to the 7B model.
3.3 Experiments
We experiment with the Code Llama family (Roziere al., 2023), finetuned version of Llama (Touvron et al., 2023). Code Llama comes in various sizes, which we use for our experiments: 7B, 13B, 34B and 70B. For the smaller benchmarks, HumanEval and MBPP, also consider the Llama-3 family (SB and 70B).
We follow Roziere et al. (2023), and use zero-shot setting for HumanEval, a 3-shot prompt- ing strategy for MBPP and 2-shot prompts for APPS, and limit the generation length to 512/256/ 256 tokens for HumanEval/MBPP APPS. For the sampling process, we use nu- cleus sampling (Holtzman et al., 2019) with top-p 0.95 and temperature of 0 0..//0.6 for HumanEval/ CMMPPAAPP with all models sizes (Roziere et al., 2023). Finally, we also report, the pass@1 results using a greedy decoding method for all models.
To compare models in varying sizes, we select the maximal number of generations for each model with respect to the values in Table 1. Specifically, for the smaller bench- marks, HumanEval and MBPP, we generate и = 2, 000/ 000/11000/4/00 0000/400/2000 answers for the 7-8B ABB/11B//0 111///3/////// /7B models, respectively. For the larger benchmarks, the three splits of To get a robust estimation of these measures, we
APPS, we use n 1, 10000000////000 follow Chen et al. (2021) and Roziere et al. (2023), and report for all benchmarks maximal value of = z
for the pass@k metric, while using all available unit-tests.
Small Models Outperform Large Ones under Fixed Compute Budget
Results for HumanEval and MBPP using the Code Llama models are presented Figures and 3, respectively.' The corresponding results for the Llama-3 models can be found in Figures 10 and 11 (Appendix A). We first note that, as expected, the pass@k metric improves both with model scale, and with the number of generations k (sub-figure (a) in all figures). However, perhaps surprisingly, when considering the passnoops @ and passtime @t metrics (sub-figures (b) and (c)), we see different trend-given fixed compute budget, smaller models yield better results than larger ones. Specifically, the 7B/ /888/13 13B models outperform the larger models across all compute budgets. Particularly, in the small budget regime (up to 32 normalized FLOPs units and 64 wall-time units) the performance gap reaches 5-15%.
Another way of looking at our results is by observing that smaller models match the performance of larger ones using substantially lower budgets. For instance, in HumanEval, the Code Llama 7B and 13B models achieve score of 60% using one quarter of the time it takes the larger models to reach that score. This efficiency gap further increases with the
LLlama-3 8B/70B presents similar usage to Code Llama 7B/70B, with a difference of up to 7%.
"Tables 2 and 3 in Appendix B presents detailed results.
4

---

Published as a conference paper at COLM 2024
80
pasSirpx @p
80
80
pass-gk
passumu
60
80
60
138
348
40
4.0
40
708
Greedy
4
16
64
256
1024
16
64
256
1024
16
64
256
1024
FLOPs (norm.)
wall-time
(a)
(b)
(c)
Figure Code Llama performance (Y axis) as a function of compute (X axis, exponential scale) for the HumanEval benchmark. Larger models perform better in general (Figure 2a), but under fixed compute budget (Figures and 2c), smaller models (7B and 13B) sub- stantially outperform larger ones (34B and 70B). Greedy decoding is marked by star.
80
80
80
passikk
pasSm.p-
pasSume@0
78
130
60
60
60
348
40
40
40
Greedy
16
64
1021
256
1024
256
1021
FLOPs (norm.)
wall wall-Eime (norm.)
(a)
(b)
(c)
Figure 3: Code Llama performance vs. compute for the MBPP benchmark. As in Hu- manEval (Figure 2), larger models perform better as function of (Figure 3a), but worse under fixed compute budget (Figures 3b and 3c).
138
passiak
40
pasSitn-08
40
40
34B
708
Greedy
20
20
20
32
128
512
8
32
128
512
2
128
512
FLOPs (norm.)
wall-tme (norm.)
(a)
(b)
(c)
Figure 4: Code Llama performance compute for the APPS benchmark, introductory split. The 13B model is superior to the 34B model and comparable to the 70B model under fixed budget. In contrast, the 7B model underperforms the larger models.
Llama-3 models (Figure 10c). Finally, we compare small models greedy decoding with larger models, which generally performs better than sampling. We observe that even in this setup, using the smaller models several times is equivalent or preferable in all cases.
We next turn to discuss the Code Llama results over the three splits of the APPS bench- mark (Figures 4 to 6).5 We first consider the 13B model, and observe the same trends as in HumanEval and MBPP: this model achieves the best performance in almost all fixed com- pute budgets. Specifically for the competition split (Figures 6b and 6c), the most challenging APPS split, the 13B model outperforms other models in all compute budgets, with consistent margin of ~5%% from the 70B model when considering the wall-time budget. We
TTable 4 in Appendix B presents detailed results.

---

Published as a conference paper at COLM 2024
30
30
30
pass@K
20
pasSinex @0
20
paSkum
20
70B
348
Greedy
2
8
32
128
512
2
8
32
128
512
2
32
128
512
FLOPs
wall-time (norm.)
(a)
(b)
(c)
Figure 5: Code Llama performance vs. compute for the APPS benchmark, interview split. Similarly to the introductory split (Figure 4), the 13B model is superior to the 34B model and comparable to the 70B model under fixed wall-time, while the 7B model is inferior to the larger models.
15
15
348
passiak
20
pacsungg
20
70B
Grondy
2
8
32
128
512
2
8
32
128
512
2
32
128
512
FLOPs (norm.)
wall-Bime (norm.)
(a)
(b)
(c)
Figure 6: Code Llama performance vs. compute for the APPS benchmark, competition split (the most challenging one). The 13B model is superior to both 34B and 70B models under fixed wall-time, and comparable to the 70B under fixed number of FLOPs.
further observe that the 13B model achieves similar or better performance the greedy approach of all models in all three splits. Finally, when fixing the performance, the 13B model 2-4 times more efficient than the 70B model (both for FLOPs and wall-time).
We next observe that the 7B model is also competitive with larger models in small budget regimes (up to 8 normalized FLOPs units and 16 wall-time units). Nonetheless, it slightly underperforms the other models larger budgets. This can be attributed to the 7B model's inability to generate a sufficient number correct answers for the task, and may suggest that there is minimum size requirement for certain level of task difficulty.
Our results indicate that small models can match or even outperform large ones under fixed compute budget, assuming the availability of unit-tests. An intriguing aspect of our research question is what happens when unit-tests are unavailable, and single selection among several generations must be made. We delve into this topic in the following section.
Evaluating Code Generation without Unit-tests
We examine the scenario where unit-tests are not available (e.g,, IDE code-completion setup). In this case, an efficient selection policy strategy may be used to select one answer from the model's generations. the previous cases (Section 2), unit-tests served as this policy. Here we investigate using ranking selection policy. In Section 5.1 we show how to estimate the performance model given such a strategy, and in Section 5.2 we analyze the performance of larger models rankers for small model.

---

Published as a conference paper at COLM 2024
def rank_scoro_at_kka k, pass_sorted);
:param n: total number samples
.param k: k in rank-score@u
:param pass, sorted: binary list of pass scores. The list is sorted by the ranks assigned examples by ranker. ***
7
numerator_sun _uu for 1 in range (1, n-k+2):
numerator_ sum +- math comb (n-i, k-1) _cores_and_pes [i-1]
10
score (numerator_ sum math. comb(n, k)) 100 return score
11
Figure 7: A Python implementation of rank-score@k as presented in Equation (4).
5.1 Evaluating Rankers
We assume a model that generates outputs, and a policy that ranks them. To estimate the performance of such setup, count the number of groups containing k generations where the highest-ranked generation within them a correct one. That is:
pass; ].
A Σ
rank-score@k:= == E
(4)
Problems
(3)
k k-1
where n(>k) is the number of answers generated for the estimation, and [pass,, ppsssy.......................... pass,] E [0,1]" are the pass scores sorted according to the ranking policy. if the example ranked according to the policy correct, and otherwise. See Figure 7 for a python implementation of aco-sccre@k..
That is, pass;
Similarly to Equations (2) and (3), also define:
rank-scoreflops @f := rank-score@k where:
max K, (ops(k'ss]
(5)
rank-scoretime @t rank-score@k
where: k K',
(6)
time(k))
where flops(k) and time(k) are the same functions as Section 2.2. Next, we evaluate the performance of large models rankers using the above metrics.
5.2 Large Language Models as Rankers
We examine the usage of LLMs rankers. To produce ranking order over a set of generations, we use the averaged Negative Log Likelihood (NLL) the LLM assigns to each generation (excluding the prompt), and rank the generations according that score. It should be noted that extracting the NLL of model over given generation can be done in a parallel manner (i.e., non-autoregressively) which substantially faster than traditional token-by-toker generation. The score given by a model to generation G (W)..., wyy given a prompt
score = NLI ( (G|P)
- Σ log (Pmodel (w 1...., WT, P)).
To study the performance of LLMs as rankers we use the HumanEval and MBPP benchmarks. We use 2, 000 generations produced by Code Llama 7B as described in Section 3.3. As rankers we use all four Code Llama model sizes. We discard any generation that fails to complete, ie. reached the maximal number of generated tokens without producing end-of-sequence token. We also report the performance of running each model independently with one generation budget (both greedy and sampling).
7

---

Published as a conference paper at COLM 2024
50
rank Scoremee @tt
@t
60
Pank-scoreumi
40
50
30
40
7B
138
Groedy dec.
Sampling dec.
20
30
22
8
16
1
2
16
wall-time (norm.)
wall-time (norm.)
Figure 8: rank-scoreéme @f as a function of wall-time for HumanEval (left) and MBPP (right), using different rankers (different lines). Greedy sampling is marked as star, and top-p sampling as circle. While ranking results improve with the size of the ranker and with compute budget, they still fall short of greedy decoding with larger models.
Our results are presented in Figure 8. As can seen, using LLMs as rankers over generations obtained from smaller models improves performance. Interestingly, we observe that using a 7B model as a ranker for itself can enhance its generation even further than the greedy approach, albeit with the cost generating several outputs. We also find that using larger models as rankers results in better perfomance. When considering fixed compute budget, find that it is sometimes comparable to LLMs rankers instead of sampling from them, as can be seen with the 13B and 34B models. However, this not the case for the greedy approach which consistently outperforms ranking multiple generations from smaller model given a fixed compute budget.
To further check the use of external verifiers, inte- grate the LEVER verifier model (Ni et al., 2023) with the Code Llama models. The LEVER approach aims enhance code generation by learning to verify gener- ated programs. The full LEVER pipeline involves using the NLL produced by the code generation model, er- ror pruning based on execution, and verifier trained on code generations with execution results. However, since we assume that tests are available our setting, execution pruning and execution results cannot be used. LEVER released a trained verifier over the MBPP bench- mark, which model. As shown in Figure 9, the LEVER verifier does not improve the results the test-less setting, which is expected given that one of the main components of the approach relies on execution over unit-tests.
rank- scorecmeeee
50 40
12
4
16
wall-time (norm.)
Figure 9: rank-score @f function of wall-time for MBPP, using the LEVER verfier with dif- ferent NLL rankers. Results are similar to Figure 8.
along with the NLL scores of each
In summary, there remains gap to bridge between using LLMs as rankers for smaller models and using them as generators. To further promote this line of research, we release the 2, 000 generations per example produced by the 7B model for both HumanEval and MBPP (a total of 1, 328, 000 generations).
Related Work
6.1 Model Scaling
Model scaling was found to be one of the key elements in the success of LLMs (Dehghani et al., 2023; Gu et al., 2023; Hassid et al., 2023; Rae et al., 2021; Chowdhery et al., 2023; Touvron et al., 2023), with Wei et al. (2022) demonstrating how specific abilities emerge mainly after reaching specific scale. The way language models behave when they scaled up and their ability to adjust have been a significant factor in the creation of LLMs (Hernandez et al., 2021). Kaplan et al. (2020) investigated the optimal model size to train for given
8

---

Published as a conference paper at COLM 2024
compute budget, while Hoffmann et al. (2022) demonstrated how scaling both model and dataset sizes improves performance across various tasks. Clark et al. (2022) analyzed the scaling properties of muiture-of-expert models, showing that scaling with the number of experts diminishes as model size increases. Recently, Gadre et al. (2024) provided a scaling law analysis considering downstream tasks rather than next-token prediction loss. They related the perplexity of a language model to its downstream task performance via a power law and used it to predict the top-1 error averaged over the evaluated downstream tasks. Our work differs from all of the above, as we do not claim to provide new scaling laws but rather suggest that when fixing the budget, smaller models can provide comparable or superior results to larger ones.
Recent studies by Shi et al. (2024) and Mei al. (2024) have demonstrated that under constrained compute budgets, smaller vision models can surpass their larger counterparts. Specifically, Shi al. (2024) found advantages in using multiple image scales, whereas Mei et al. (2024) observed that smaller diffusion models perform better than larger ones when the compute budget is fixed. Our approach, which generates multiple text outputs from a small model, aligns with these findings.
6.2 Verifiers and Rankers
LLM verifiers and rankers is a growing trend, which leverages LLMs to verify and rank generations obtained from weaker and smaller models (Cobbe et al., 2021b; Uesato al., 2022; Saha et al., 2024; Havrilla et al., 2024). Both Cobbe et al. (2021b) and Uesato et al. (2022) leveraged an external classifier rank LLM outputs. Specifically, in both setups the authors proposed to generate many candidate solutions and select the one ranked highest by the verifier. The authors demonstrated the applicability of using such verifiers in solving math word problems (Cobbe et al., 2021a). Qin et al. (2023) demonstrated that LLMs can serve as efficient text rankers when considering pairwise ranking.
Another line of work leveraged LLMs to evaluate the quality of smaller models (Saha et al., 2024; Dubois et al., 2023; Zheng et al., 2023; Oren et al., 2024). Although providing promising alternative, such evaluation suffers from biases in the larger model (Zheng et al., 2023) and reliance on hand-designed evaluation plans that impact the method's 's ability to generalize (Liu et al., 2023). Large models also serve as verifiers of small ones in speculative decoding setup, with the goal of speeding-up up LLM generation (Leviathan et al., 2023; Kim al., 2023; Chen et al., 2023). It also common to distill knowledge from a large model into a smaller one in order to improve efficiency (Hinton et al., 2015; Sanh et al., 2019; Xu al., 2024), see Treviso al. (2023) for survey on efficient methods in NLP.
In this work, we explore the potential of LLMs as selectors of the best output of a smaller model in a fixed budget setup. Similarly ours, Li et al. (2024) found that smaller sized LMs (7B parameters) already exhibit strong mathematical abilities when selecting the best response from different generations. When considering code generation models, Al- phaCode Team (2023) presented impressive results on challenging coding contests tasks while generating 1M samples, and later on filtering and ranking them using Gemini-Pro LLM (Team et al., 2023). Dou et al. (2024) proposed a method to improve code-generation models by learning policy model using reinforcement learning methods. Lastly, Shi et al. (2022) and Ni et al. (2023) used execution feedback in order to filter oode-generations, while Shi et al. (2022) used non-learned approaches, Ni et al. (2023) trained an external verifier on top of the generation and the execution feedback.
Discussion & Limitations
Our results show that using smaller models with the same amount of compute can improve LLM code-generation performance. An interesting question we do not fully address is whether, given enough compute, the larger models will overtake the smaller ones, or perhaps they will all saturate at a similar performance level at some point. Our HumanEval and MBPP results seem to slightly support the latter hypothesis (as all models begin to saturate, see Figures 2 and 3). However, unfortunately, due to compute constraints, our

---

Published as a conference paper at COLM 2024
setting is restricted to exploring only limited number of generations per model. We note that despite this limitation, in practice, due to these costs our conclusions apply to most practical use-cases. We defer more expensive experiments to future work.
Conclusion
In this work, we compared large language models with smaller-sized models under fixed budget constraints (i.e,, FLOPs and wall-time). We evaluated the models using execution- based code-generation tasks, which provide access to unit-tests. Our findings reveal that generating multiple outputs from a 13B model may lead to gains of up to 15% over single generation from 70B model across five tasks. This highlights the potential using smaller models instead of larger ones. In scenarios where unit tests or other solution verifiers are unavailable, we explored a simple ranking-baa -baeed approach for candidate selection. We found the proposed ranking approach falls short in performance compared to single output from the larger model. Our findings emphasize the importance of studying approaches for ranking LLM outputs, which hold great potential to not only improve model performance but also improve budget allocation. To further enhance this research direction we release over 1M samples from the Code Llama 7B models spanning both HumanEval and MBPP benchmarks.
9 Acknowledgments
We thank Miri Varshavsky Hassid for the great feedback and moral support.
References
Al@Meta. Llama 3 model card. 2024. URL httpttt/gittuub..om httppp///gtthhoooooommmaaaaaaaee blob/mainn/MDEEL.COO .dd
Google DeepMind AlphaCode Team. Alphacode technical report, 2023. URL https://storag: http:////sttraggggggleaapis.. googleapig....ooooooooooo com/ deeg:m/ii-mmddiasapppppaaeee///p _ecch_ Report.pdf.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models, 2021. URL
org/ htwp//bb/100.0772.2. arXiv:2108.07732.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023. URL https: org 2tab///00..... 01318. arXiv:2302.01318
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Vosse William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL httpt:tt///eee //axiv.. org/ abs/2107 03374. arXiv:2107.03374
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann,
For instance, generating 1, 000 answers for the 5, 000 examples of the APPS benchmark with 7B model takes about 20 days using node of 8 A100 GPUs
10

---

Published as a conference paper at COLM 2024
et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057-4086. PMLR, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021a. URL https://arxiv.orr htwpp//..wgwb//////////ee abs///a/...1..///////////// 14168. arXiv:2110.1416..
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021b. URL /tt////axxxi org/abst h/s/bb1//110..//./////// 1100.... 14168. arXiv:2110.14168.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp- 7480-7512. PMLR, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, 2022.
Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Junjie Shan, Caishuang Huang, Wei Shen, Xiaoran Fan, Zhiheng Xi, et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback, 2024. URL abs/2402.
org/
arXiv:2402.03391.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy $ Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In Advances in Neural Information Processing Systems, 2023. URL https: http:://axvvvvvviii wwwp:/rg////////ee/eee ht///3bb/335555 14387.
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, al. Language models scale reliably with over-inai and on downstream tasks, 2024. URL https:
//arxiv. .wriv.orze orgg 2203. 08540. M0Xiv:2403.085400
Yile Gu, Prashanth Gurunath Shivakumar, Jari Kolehmainen, Ankur Gandhe, Ariya Rastrow, and Ivan Bulyko. Scaling laws for discriminative speech recognition rescoring models, 2023. URL https: //tr///axxi org/ a23/ 15815. arXiv:2306.1558 15815.
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, et al. Textually pretrained speech language models. Advances in Neural Information Processing Systems, 36, 2023.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Swivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning, 2024.
URL https://axxiv.. bwtpxxivvorr///ee hogg/abs/2403. 04642. arXiv:2403.04642.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Advances in Neural Information Processing Systems, 2021.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. URL httppppp///aaa.....//a////////// 01293. arXiv:2102.01293.
11

---

Published as a conference paper at COLM 2024
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXivv1503.02551,, 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models, 2022. URL https: //ap://////aaaaaa/aaa org/ abs/2203. 15556. arXiv:2203.1555 155556..
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: / org/ hat/a/bb/201111 08361. arXiv:2001.0883
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems,, volume 36, pp. 39236-39256. Curran Associates, Inc., 2023. URL, https: /tttpee//prooceeeddnggg..o wwwriiinnnnnn..u..e.re.. www.wnew.iii..o. ccp wttp:eccpppppppeeeeeeeeeeeeeeeee files/lll//eeee wttpee/feeeeeeeeeee//ee/ /ppp//pppprrr/////eee pdf.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based pseudocode to code. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d' Alchh-Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL attps://pproeeedingg www..ne....ccoeoooooeeeeoo wwtpccppppppileeee ceaper_file//eee pat.ee//pppee/21///12eeee 72832104400440G44c66e6ffff- Paper
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proc. of ICLR, 2023. URL httttttt//aaiiii httpaaawwwi..oorraaaaraaaaaaa 2211..1992.
Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7B language models already possess strong math capabili- ties, 2024. URL https://arxiv.. httpps://aaxvv.or///a/////////a/
04706. arXiv:2403.04706.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- Eval: NLG evaluation using GPT-4 with better human alignment, 2023. URL https: //arxiv. org/ abs//303. 16634. arXiv:2303.1663.
Kangfu Mei, Zhengzhong Tu, Mauricio Delbracio, Hossein Talebi, Vishal M Patel, and Peyman Milanfar. Bigger is not always better: Scaling properties of latent diffusion models. arXiv preprint ar arXi::2404.01367 2024.
Ansong Ni, Srini lyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning verify language-to-code generation with execution. In International Conference on Machine Learning, pp. 26106-26128, PMLR, 2023.
Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz. Transformers are multi-state RNNs, 2024. URL aattps://arxte
org/abs//2000 06104. arXiv:2401.06104.
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large language models are effective text rankers with pairwise ranking prompting, 2023. URL https: //arxiv. ab3/ 17563. arXiv:2306.17563.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher, 2021. URL https: //arxiv. org/ abs///12..... 11446. a12.v.2112.11446.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code, 2023. URL https:://aaaai wrg/abs/2300 12950. arXiv:2308. 12950.
12

---

Published as a conference paper at COLM 2024
Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-solve-merge improves large language model evaluation and generation. In Proc. of NAACL, 2024. URL pestps://arxyy org/ abs/2310 15123.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXio:1910.01108, 2019.
Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models?, 2024. URL https://arxiv org/abs/ babbs/2403..m 13043. arXiv:2403.13043.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natural language to code translation with execution. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3533-3546, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v/ 10.16653v1122222mmmml /2t/2222emmmmmmimin235. URL org/2022. emnlp-main 231.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGI'T good search? investigating large language models as re-ranking agents. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14918-14937, Singapore, December 2023, Association for Computational Linguistics. doi: (0.8653/vv] 2023emmip--minn925 hRttttps:: //aclanthoooogy org/2023. emnlp-main 923.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models, 2023. URL https: /tte:///axxii http://aaii...o//aa//////3///// arg/abs/2312. 11805. mrXiv:2312.11800
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models, 2023. URL https:/s://arr arXiv:2307.0928..
hrrg/bbs/288 2ttps/2227770088.8.. 09288.
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, and Roy Schwartz. Efficient methods for natural language processing: survey. Transactions of the Association for Computational Linguistics, 11:826-860, 2023. doi: 10.1162 tacl//acl....// a_00577. URL hwwpollogy.rgg/2222000 org/2023. tacl-1.48.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback, 2022. URL battts:://rxiis org/abs abs/ab///2/ /221. 14275. arXiv:2211.14275.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL attps://openreviert net/ htt/:/frrmmm/ arrmmiddyyasUUzzaa Survey Certification.
xiaoju ye. calflops: a flops and params calculate tool for neural networks in pytorch framework, 2023. URL https://githuu: cotp///t/a///alllllllaeeeeeaeee popppppp
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. Xiv preprint ar Xix::4402.1331 2024.
13

---

Published as a conference paper at COLM 2024
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, 2023.
Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, and Dawei Yin. Pre-trained language model based ranking in baidu search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 4014 4022, 2021.
14

---

Published as a conference paper at COLM 2024
100
100
100
pasSerpx00
passum@r
pass@gk
60
60
60
8B
708
40
40
40
Greedy
2
16
64
256
1024
16
64
256
1024
2
16
64
256
1024
wall-time (norm.)
(a)
(b)
(c)
Figure 10: Llama-3 performance vs. compute for the HumanBval benchmark. The 70B model performs better in general (Figure 10a), but under fixed compute budget (Figures 10b and 10c), the 8B model substantially outperforms the larger one.
90
90
90
80
pasSmee-80 00F
80
80
pass@ax
70
70
70
60
Greedy
40
40
40
16
64
256
1024
16
64
256
1024
16
64
256
1024
FLOPs (norm.)
wall time (narm.)
(a)
(b)
(c)
Figure 11: Llama-3 performance vs. compute for the MBPP benchmark. As in Hu- manEval (Figure 10), larger models perform better as a function of k (Figure 11a), but worse under fixed compute budget (Figures 11b and 11c).
A Llama-3 Results
We present Llama-3 results for the HumanEval and MBPP benchmarks in Figures 10 and 11, respectively.
B Detailed pass@k Results
In Tables to 4 presents precise pass@k results for the datasets examined (HumanEval, MBPP and APPS, respectively). Due to the infeasibility of reporting results for all k, we provide results for selected k values. Nevertheless, values were calculated and used in the computation of the figures.
important to note that all relevant
Table Precise models' pass@k results for several k values over the HumanEval benchmark.
k for pass@k
2
4
16
64
128
256
500
1000
Model
Code Llama 7B
28.2
38.5
48.9
68.7
83.9
89.0
92.7
95.0
96.3
Code Llama 13B
32.7
44.4
56.2
77.0
89.0
92.3
94.8
96.5
Code Llama 34B
38.6
51.4
63.4
81.8
91.7
94.3
Code Llama 70B
46.7
61.1
73.2
87.3
94.4
Llama-3 8B
31.8
43.1
54.4
74.0
86.9
90.9
94.0
96.0
97.3
Llama-3 - 70B
47.3
59.3
69.8
84.7
92.8
15

---

Published as a conference paper at COLM 2024
Table 3: Precise models' pass@k results for several k values over the MBPP benchmark.
k for pass@k
2
4
16
64
128
256
500
1000
Model
Code Llama 7B
37.3
48.0
57.3
71.6
81.1
84.8
87.9
90.3
92.4
Code Llama
42.3
53.4
62.5
75.3
84.0
87.4
90.4
92.8
Code Llama 34B
49.2
60.1
68.6
79.2
85.8
88.7
Code Llama 70B
57.3
67.7
74.8
83.7
90.0
3.aam-33 8B
41.1
52.4
61.8
75.5
84.4
87.7
90.5
92.8
94.7
Llama-3 70B
61.2
71.5
78.7
87.2
92.9
Table 4: Precise models' pass@k results for several k values the different splits of the APPS benchmark.
k for pass@k
2
16
32
64
128
256
500
Model
APPS-introducee
Code Llama 7B
4.5
7.7
12.1
23.0
28.6
34.0
39.3
44.4
49.0
Code Ilama 13B
10.3
16.3
23.1
36.3
42.3
47.8
52.8
Code Llama 34B
12.6
19.7
27.4
46.8
52.5
Code Llama 70B
22.6
30.5
38.1
52.5
58.5
APPS-interview
Code Llama 7B
19.2
Code Llama 13B
2.2
3.8
5.9
11.6
14.8
18.3
21.9
Code Llama 34B
2.9
4.9
7.6
14.4
18.1
22.0
Code Llama 70B
5.5
8.7
12.6
21.8
26.6
APPS-competition
Code Llama 7B
0.1
0.3
0.5
1.6
2.6
4.1
6.1
8.6
11.6
Code Llama 13B
0.6
1.2
2.1
5.3
7.8
10.9
14.4
Code Llama 34B
0.7
1.3
2.3
6.1
9.0
12.6
Code Llama 70B
15.3
16


Reviews:
[{"review_id": "ehmWuCY7d0", "reviewer": "Reviewer_ZRmm", "summary": "The paper explores the hypothesis of whether larger models are better than smaller models even in limited computational budget scenarios. In other words, the question the paper explores is: can we generate more outputs from a smaller model so that it matches the performance of bigger models under a certain computational budget? The analysis focuses on the task of code generation. The models are large language models (LLMs) of different sizes and the smallest size is 7B parameters while the largest is 70B parameters. The results of the paper show that repeated use of smaller models can yield consistent improvements with gains of up to 15% across different tasks. In scenarios in which a candidate generation needs to be ranked and then selected as the best one, the smaller model fall short of the performance of a single greedy decoding output from a larger model. It is unclear from the paper the reason for this result.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We appreciate them approving that our paper “explores an interesting and important problem”, acknowledging the improvements shown in our results and our experimental setup being “robust and well designed”, stating that the paper is “well written and well organized”, and overall “useful for the community”.\n\nRegarding the comment about evaluation on other tasks: we note that the focus of this work is on tasks that have a fast and reliable way to verify the output. To the best of our knowledge, the major task with practical relevance that meets this criterion is execution based code generation, as, for example, test-driven development is a common approach in software development. Notice, under the text-to-image domain, recent studies found similar trends regarding smaller models outperform larger ones under fixed budget, see [2]. Nevertheless, we agree that demonstrating similar properties for other verifiable NLP downstream tasks is interesting, and would appreciate any recommendations for such tasks. \n\nWe agree that ranking LLM outputs is an interesting research area to explore. Although this is not our main contribution, we explore the use of LLMs as verifiers using their NLL score for ranking (and find that it is insufficient). \nAs the reviewer mentioned, there are studies that use trained rankers. We tried using the LEVER [1] ranking approach over the generations produced by the Code Llama 7B model for the MBPP test set (using the trained model from [1]). The LEVER approach uses the LM probability over the generation combined with a probability given by a trained ranker.*\nResults are shown in Figure 3 in tinyurl.com/ykbappvj, and show a very similar trend to our experiments with NLL scores (Figure 8 in the paper). We will add this result to the paper.\nMoreover, our work sets the environment for further research in the field, both by providing code for rank-score@k metric and by releasing (upon publication) more than 1M generation examples of CodeLlama-7B for the HumanEval and MBPP datasets.\n\n* We didn’t use execution feedback, as in our **ranking** setting we assume tests are not available.\n\n[1] arxiv.org/abs/2302.08468\n[2] arxiv.org/abs/2404.01367", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is easy to follow, well written and well organized. The experimental settings seem robust and well designed. The paper explores an interesting and important problem that is assessing whether we actually need to use the largest model available always, under every computational budget condition. This is useful for the community that works on building such models and applications with them so that they make informed decisions when designing model architectures and approaches that rely on them.", "reasons_to_reject": "Mainly related to the generalization of the results across different generation tasks or all types of tasks. The paper focuses solely on one task, code generation, for which there is a big interest in the software engineering community. However, it is difficult to generalize the conclusions obtained for code generation for every other generation task. The main reason is because the evaluation metrics change according to the task.\n \nIt is not completely clear and focused on the paper why the ranked-selection-based task (for code generation cases without unit tests) do not work well for smaller models. Previous work [1], [2] explored re-ranking approaches to other NLP problems such as machine translation in which the the ranker is an LLM trained with human-labeled supervision about the translation quality of translated outputs. In this scenario, re-ranking with this specialized ranker usually works well but no analysis of different LLMs sizes has been done. Perhaps this is an interesting research direction that complements the work here.\n\nOverall the paper could use from experiments on other tasks besides code generation to be able to claim that smaller models are better than larger ones. \n\n[1] https://arxiv.org/abs/2205.00978\n[2] https://aclanthology.org/2023.emnlp-main.733/", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RsexgT71lG", "reviewer": "Reviewer_P86J", "summary": "This paper studies an interesting questions of given the same budget of compute or wall-time, will the smaller models outperform larger models given their lower inference cost. Experiments are conducted with the CodeLLaMA models ranging from 7B to 70B, on three function-level code generation benchmarks, HumanEval, MBPP and APPS. Results show that when test cases are available, smaller CodeLLaMA models consistently outperforms larger models. However, when test cases are not available, larger models still perform better even when likelihood-based ranking methods are used.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful review. We appreciate them finding our results of “great impact,” for acknowledging that our experiments “are nicely designed,” and for considering our paper as “quite smooth” and “very easy to follow”.\n\nW1+Q1: Reg. using one model family, we point that the experiments conducted required substantial computational resources, ~100 days of 8x A100 GPUs (e.g., footnote2). Nevertheless, per the reviewer’s request, we conducted additional experiments with the LLaMA-3 [2] models (8&70B) over HumanEval and MBPP, see tinyurl.com/ykbappvj, Figures 1&2. The trends mirror those seen with the Code Llama models, i.e: under a fixed compute budget, running the 8B model multiple times substantially outperforms running the 70B model fewer times. We will include these results in the paper.\n\nW2: We will add to the paper all pass@$k$ numbers.\n\nW3+Q5: All available unit-tests are used to verify the correctness of each generation, i.e., only solutions that pass all unit-tests are considered correct. We will clarify this.\n\nQ2: Reg. trying other ranking methods, although this is not our main contribution, we explore the use of LLMs as verifiers using their NLL score for ranking.\nAs for the reviewer’s suggestion, we tried using the LEVER [1] ranking approach over the Code Llama 7Bs’ generations for MBPP (using the trained ranker from [1]). This approach uses a combination of the LM and the trained ranker probabilities.*\nResults are shown in Figure 3 in tinyurl.com/ykbappvj, and show a similar trend to our experiments with NLL scores (Figure 8 in the paper). We will add this result to the paper.\nMoreover, we do set the environment for further research, both by providing code for rank-score@k metric and by releasing (upon publication) more than 1M generations of Code Llama 7B for HumanEval and MBPP.\n\n* We didn’t use execution feedback, as in our ranking setting we assume tests are not available.\n\nQ3+Q4: We use a custom implementation tuned for fast inference, based on the xFormers library and its \"memory_efficient_attention\" primitive using FlashAttention (github.com/facebookresearch/xformers). This primitive dispatches automatically to different backends to maximally optimize inference time, based on input params, GPUs, arch., etc. We believe quantization/other efficiency methods will affect our numbers similarly for all model sizes, hence can be considered as an orthogonal direction.\n\n[1] arxiv.org/abs/2302.08468; [2] llama.meta.com/llama3/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. The idea of this paper is very interesting, and the results potentially have great impact on how the practitioners would choose models for their specific use cases;  \nS2. The experiments are nicely designed, the metrics as pass_flop@k and pass_time@k make a lot of sense to me;  \nS3. The flow of this paper is quite smooth, which makes it very easy to follow", "reasons_to_reject": "W1. The major weakness of this paper is that only one set of models (i.e., CodeLLaMA) is studied in this work, which makes it unclear whether the same conclusions would generalize to other code-specific models (e.g., StarCoder) or general purpose-models (e.g., LLaMA);  \nW2. I found it quite strange that all of the results are reported in the form of figures, and there are no concrete numbers in such figures. For reproducibility and comparison purposes, I would suggest at least putting the concrete numbers in the appendix;  \nW3. There are some key experiment details that are missing, see Q5 below.", "questions_to_authors": "Q1. Do you plan to add the results for at least another set of models, either code-specific (e.g., StarCoder) or general-purpose (e.g., LLaMA)?   \nQ2. Have you tried other reranking-based methods for code, such as MBR-Exec (Shi et al., 2022) and LEVER (Ni et al., 2023)? Also note that there are a couple of other heuristic-based reranking methods described in Ni et al., 2023. If not, can you comment on what the potential performance implications will be?  \nQ3. Can you comment on how methods like quantization and flash-attention would affect the FLOPs and wall-time calculated in Table 1?   \nQ4. For larger models, have different model parallelism methods been tried to ensure that maximal efficiency is achieved given the same compute infrastructure?  \nQ5. How are the tests being used for the results reported in Section 4? For example, for MBPP there are 3 test cases, do you use all three to filter the incorrect samples, or just 1?", "extra_scores": {}}, {"review_id": "LyWmeGad00", "reviewer": "Reviewer_52pF", "summary": "The paper investigates whether smaller LLMs can outperform or match the performance of larger models when constrained by the same computational budget. The authors present a approach that utilizes smaller models multiple times within a given budget and selects the best output using a ranking method. This study is situated within the broader discourse on the efficiency of model scaling and the practical constraints of deploying large LLMs.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their meaningful review. We are encouraged to hear that the reviewer found that our findings “have practical implications, especially for applications with limited computational resources”.\n\nAddressing specific reviewer comments:\n\n1. Regarding practical applications, we agree that it would be beneficial to study ways to improve performance across various compute budgets. However, we also like to emphasize that having a fixed budget (e.g., as a maximum cost per interaction) and a fixed model (e.g., using an open-source LLM checkpoint or API) is a very practical scenario. We also note that all our results are across various (fixed) compute budgets, and that our improvements are across the board. As a result, we believe our study provides valuable insights for the use of available resources.\n\n2. We agree with the reviewer that ranking LLMs outputs is an interesting research area to explore. Although this is not the main contribution of our paper, we explore the use of LLMs as verifiers using their NLL score for ranking (and find that it is insufficient). \nAs suggested by P86J, we tried using the recent LEVER [1] ranking approach over the generations produced by the Code Llama 7B model for the MBPP test set (using the trained model from [1]). The LEVER approach uses the LM probability over the generation combined with a probability given by a trained ranker.*\nResults are shown in Figure 3 in tinyurl.com/ykbappvj, and show a very similar trend to our experiments with NLL scores (Figure 8 in the paper). We will add this result to the paper.\nMoreover, our work sets the environment for further research in the field, both by providing code for rank-score@k metric and by releasing (upon publication) more than 1M generation examples of CodeLlama-7B for the HumanEval and MBPP datasets.\n\n* We didn’t use execution feedback, as in our **ranking** setting we assume tests are not available.\n\n3. Thank you for your comment about discussing other methods for reducing costs. We will ensure that this topic is covered in the next version of our paper.\n\n[1] arxiv.org/abs/2302.08468", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper reveals that smaller LLMs can outperform or match the performance of larger models when constrained by the same computational budget. The findings have practical implications, especially for applications with limited computational resources.", "reasons_to_reject": "1. In practical applications, developers aim to reduce computational costs while enhancing performance. However, the paper only presents results under a constant computational budget. It would be beneficial for the study to explore ways to improve performance across various computational budgets.\n2. The reliability of using large language models as rankers, as proposed in section 5.2, is questionable. The authors should conduct further analysis on this aspect.\n3. In the related work section, the authors should also discuss the direction for enhancing generation speed and quality with smaller models (such as distillation).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8olQbbrLal", "reviewer": "Reviewer_NH43", "summary": "The paper compares large and smaller LMs for the code generation under the same computation budget (FLOPs or wall-time). There are two setups in the paper depending on the availability of unit tests. When the unit tests are available, generating more outputs using smaller LLMs increases the likelihood of getting a correct answer. When the unit tests are unavailable, greedy decoding with large LMs seems to be a good choice.\n\nThe writing of the paper is clear and self-contained. As someone who is not familiar with code generation, I find the paper is easy to follow. The idea of turning the evaluation from pass@k to pass@flop and pass@time is simple but well-motivated given the current methodology of evaluation with pass@k. The paper executes the simple idea well.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We appreciate the reviewer for thinking that our evaluation metrics are “well-motivated”, noting that we execute our ideas well and acknowledging that the paper is “self-contained” and “easy to follow”.\n\nRegarding the absolute performance of the Code Llama models family, we agree that better results would be advantageous. However, we would like to point out that despite this, current code generation models have already shown practical utility, as evidenced by the wide adoption of tools like co-pilot, the cursor IDE, etc. by software developers. As a result, reducing their computational costs is of great practical value.\n\nRegarding scaling as a means to improve performance, we certainly acknowledge the reviewer's point and wish to clarify that we do not claim that scaling is the sole method for enhancing LLMs. Our statement was intended to highlight that scaling up is a common approach, though not the only one, currently used to improve models. We will make this distinction clearer.\n\nRegarding other ways to improve generation speed/quality, we recognize that there are various strategies to improve the performance of LLMs. However, we would like to emphasize that our paper presents a unique perspective by focusing on the efficient use of LLMs within specific budget constraints. This approach is orthogonal to other generation speed/quality methods, and we believe it contributes uniquely to the field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper could be interesting for code generation community\n- The work is incremental but solid", "reasons_to_reject": "- While we see the relative improvement of smaller LMs in pass@flop and pass@time, the absolute value of pass@flop (and also pass@k) seems to indicate that current Code Llama is far from usable. Thus I felt that the recommendation of using smaller LLMs to generate multiple outputs is not as critical as improving code generation LLMs itself.\n- The paper mentioned the common practice of scaling LLMs as a sole drive for improving performance. This is not true. In the past years the community has advocated for better pretraining/fine-tuning data. Thus I feel that the research in the paper is not far ahead looking.\n- In practice, there are multiple ways to improve generation speed/quality. For example, applying speculative decoding to LLM or distilling large ones to smaller ones. Those common practices are not studied in the paper thus it makes the paper have less practical value.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper explores whether smaller language models can perform as well as, or better than, larger models under constrained computational budgets, specifically within the context of code generation tasks. The study compares the performance of CodeLLaMA models ranging from 7B to 70B parameters, utilizing benchmarks such as HumanEval, MBPP, and APPS. The results indicate that while smaller models can achieve consistent improvements through unit tests, they fall short in ranking-based scenarios without test cases.

This paper investigates an interesting question in the field of NLP: the efficiency and performance trade-offs between smaller and larger models. The experiments are well-designed, and the findings are useful and have insights. Overall, this paper presents a well-executed study on an important topic with practical implications and I recommend acceptance. Addressing the noted weaknesses and expanding the scope of experiments would significantly enhance its contributions to the field.
Decision: Accept