The Larger the Better? Improved LLM Code-Generation via Budget Reallocation

Authors: michael  hassid
  tal remez
jonas  gehring
roy  schwartz
yossi  adi
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-20 14:00:32

Abstract: It is a common belief that large language models (LLMs) are better than smaller-sized ones. However, larger models also require significantly more time and compute during inference. This begs the question: what happens when both models operate under the same budget? (e.g., compute, run-time). To address this question, we analyze code generation LLMs of various sizes and make comparisons such as running a 70B model once vs. generating five outputs from a 13B model. We consider a standard unit-test setup, which can be used to select the correct output from the smaller model. Our findings reveal that the repeated use of smaller models can yield consistent improvements, with gains of up to 15% across five tasks. On the other hand, in scenarios where unit-tests are unavailable, a ranking-based selection of candidates from the smaller model falls short of the performance of a single output from larger ones. Our results highlight the potential of using smaller models instead of larger ones, and the importance of studying approaches for ranking LLM outputs.

Main paper: 
 introduction: 
 A common wisdom in deep learning, and language modeling in particular, is that investing more compute leads to improved performance (Kaplan et al., 2020) . The standard way of implementing this principle is training larger models. A simpler, yet often overlooked way to increase compute budget is to run a smaller model multiple times, and select the best output using some metric (Chen et al., 2021) . In this work we systematically compare between these two approaches: we ask whether, given a fixed compute budget, it is best to run a large model once, or a smaller model multiple times (Figure 1 ). Our results show that, perhaps surprisingly, given the same compute budget, running 7B or 13B models can not only match the performance of a 70B model, but also substantially surpass it.Addressing our research question requires a method for selecting the best LLM output from a given set of candidates. In this work we focus on execution based code-generation tasks, which assume the availability of unit-tests (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021) . We consider the widely-used pass@k metric (Kulal et al., 2019) , which evaluates a model's performance on code generation problems by generating k outputs and assigning a point if any of them passes all tests. To adapt this metric for our purposes, we take models of different sizes, and for each generate as many outputs as possible given a fixed compute budget, e.g., floating point operations (FLOPs) or wall-time.We apply this setup to evaluate the Code Llama (Roziere et al., 2023) model family (7B, 13B, 34B, and 70B) across five tasks: HumanEval (Chen et al., 2021) , MBPP (Austin et al., 2021) , and the three splits of APPS (Hendrycks et al., 2021) . For the HumanEval and MBPP benchmarks, we additionally use the recent Llama-3 (AI@Meta, 2024) model family (8B and 70B). Surprisingly, we find that for the two popular tasks, HumanEval and MBPP, the Figure 1 : Different ways to improve LLM performance by increasing compute budget. Top: the standard approach of increasing model size, while generating a single output. Bottom: our approach-using a small model to generate multiple outputs, and select the best one. smaller models (7B, 8B and 13B) outperform the larger ones (34B and 70B) by a margin of up to 15%. Importantly, this is observed using both budget types (FLOPs and wall-time) and across all computation budgets. When considering the challenging APPS benchmark, we find that the 13B model performs best across almost all budgets, with a consistent margin of 5% when considering the hardest split-competition.We then proceed to examine the scenario where unit-tests are unavailable, such as in an IDE code-completion setup. In such cases, an efficient policy is required to select a single solution from all generated ones. We consider a simple LLM-based policy, which ranks solutions based on the negative log likelihood of the LLM. We also augment this policy with a variant of a recent ranking approach-LEVER (Ni et al., 2023) . We experiment with the 7B model, and rank its outputs using each of the models. Our results show that, as expected, ranking-based selection improves with the increase in compute budget, and with the size of the ranking LLM. Nonetheless, this procedure still falls short of the performance achieved by running the larger model independently with the same budget.Our results highlight the potential of using smaller models instead of larger ones, a practice that has many benefits. First, small models are far computationally cheaper to pre-train.2 Further, at inference time, they are considerably more hardware-friendly: a 13B model can be accommodated on a single A100 GPU, a feat unachievable for a 70B model (Dettmers et al., 2022) . Finally, as we have shown, when controlling for the compute budget, smaller models may actually outperform larger ones.Our findings also emphasize the importance of developing effective ranking approaches for LLM outputs. This is especially important in cases where no unit-tests or other verification methods are available (Zou et al., 2021; Uesato et al., 2022; Sun et al., 2023) . To support such research direction, we release 2, 000 Code Llama 7B outputs for each example in HumanEval and MBPP-a total of more than 1M outputs. 1 
evaluation under compute restrictions: 
 To study our main research question-what is the optimal way of using a given LLM compute budget-we consider a code-generation setup with unit-tests (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021) . Below we discuss our methodology for code generation evaluation under computational restrictions. We begin by describing pass@k (Kulal et al., 2019) , the current main approach for evaluating code generation tasks (Section 2.1). We then transition to describe our variant of code generation metrics under computational restrictions (Section 2.2).To evaluate LLM code-generation abilities, a common setup assumes a set of coding questions, each with a set of unit-tests. The LLM is fed with each question, and a fixed number of output generations (labelled k) are sampled. The evaluation protocol considers each question for which at least one output passes all unit-tests as correct. To estimate the performance of a model that generates k outputs, it is common to generate a larger number of outputs n (> k) and compute:EQUATIONwhere c ≤ n is the number of examples that pass the unit-tests. The above mentioned metric results in an unbiased estimator as was shown by Chen et al. (2021) .Our goal is to compare between LLMs of different sizes under a fixed compute budget. To do so, we allow smaller models, which consume fewer resources, to generate more outputs. This results in models of different sizes requiring roughly the same amount of compute.We consider two types of compute budgets: the number of FLOPs and wall-time. For each type, a specific resource limit is set (e.g., 10k Tera-FLOPs or 8 seconds), and the model generates examples up to the point where the compute limit is reached. That is:pass flops @ f := pass@k where:EQUATIONpass time @t := pass@k where:EQUATIONwhere flops(k) and time(k) are functions that return the FLOPs/wall-time usage of a given model that generates k outputs. Notably, the FLOPs restriction is a more theoretical computational restriction, as it assumes perfect utilization of the hardware. On the other hand, the wall-time restriction is more realistic, but is hardware specific, and thus not directly comparable across different machines. 
experimental setup: 
 In this section we describe our experimental setup, focusing on the code benchmarks used (Section 3.1), our metrics (Section 3.2), and our experiments (Section 3.3).We experiment with three python code benchmarks: HumanEval (Chen et al., 2021) , MBPP (Austin et al., 2021) and APPS (Hendrycks et al., 2021) . The HumanEval benchmark consists of 164 function declarations alongside their documentation. The Code-LLM's task is to complete the function according to the provided documentation. MBPP consists of 500 test examples, each one is an instruction for a code function. Here, the Code-LLM is required to generate the full function. Lastly, the test subset of APPS is composed of 5k programming problems at various levels of difficulty: introductory (1k), interview (3k) and competition (1k). In the APPS tasks, the Code-LLM is required to generate the complete python file, which includes import declarations, class definitions, and so on.Computing the pass flops @ f and pass time @t metrics requires an estimation of the flops(k) and time(k) functions from Equations ( 2) and (3). To estimate FLOPs usage, we use the calflops library (xiaoju ye, 2023), with input sequence length of 128. We measure wall-time while 1 , for readability we also report the normalized factor with respect to the 7B model.3 We experiment with the Code Llama family (Roziere et al., 2023) , a finetuned version of Llama (Touvron et al., 2023) . Code Llama comes in various sizes, which we use for our experiments: 7B, 13B, 34B and 70B. For the smaller benchmarks, HumanEval and MBPP, we also consider the Llama-3 family (8B and 70B).We follow Roziere et al. ( 2023), and use a zero-shot setting for HumanEval, a 3-shot prompting strategy for MBPP and 2-shot prompts for APPS, and limit the generation length to 512/256/256 tokens for HumanEval/MBPP/APPS. For the sampling process, we use nucleus sampling (Holtzman et al., 2019) with top-p = 0.95 and a temperature of 0.8/0.8/0.6 for HumanEval/MBPP/APPS, with all models sizes (Roziere et al., 2023) . Finally, we also report, the pass@1 results using a greedy decoding method for all models.To compare models in varying sizes, we select the maximal number of generations for each model with respect to the values in Table 1 . Specifically, for the smaller benchmarks, HumanEval and MBPP, we generate n = 2, 000/1, 000/400/200 answers for the 7-8B/13B/34B/70B models, respectively. For the larger benchmarks, the three splits of APPS, we use n = 1, 000/500/200/100. To get a robust estimation of these measures, we follow Chen et al. (2021) and Roziere et al. (2023) , and report for all benchmarks a maximal value of k = n 2 for the pass@k metric, while using all available unit-tests. 
small models outperform large ones under a fixed compute budget: 
 Results for HumanEval and MBPP using the Code Llama models are presented in Figures 2 and 3 , respectively. 4 The corresponding results for the Llama-3 models can be found in Figures 10 and 11 (Appendix A ). We first note that, as expected, the pass@k metric improves both with model scale, and with the number of generations k (sub-figure (a) in all figures). However, perhaps surprisingly, when considering the pass flops @ f and pass time @t metrics (sub-figures (b) and (c)), we see a different trend-given a fixed compute budget, smaller models yield better results than larger ones. Specifically, the 7B/8B/13B models outperform the larger models across all compute budgets. Particularly, in the small budget regime (up to 32 normalized FLOPs units and 64 wall-time units) the performance gap reaches 5-15%.Another way of looking at our results is by observing that smaller models match the performance of larger ones using substantially lower budgets. For instance, in HumanEval, the Code Llama 7B and 13B models achieve a score of 60% using one quarter of the time it takes the larger models to reach that score. This efficiency gap further increases with the Figure 3 : Code Llama performance vs. compute for the MBPP benchmark. As in Hu-manEval (Figure 2 ), larger models perform better as a function of k (Figure 3a ), but worse under a fixed compute budget (Figures 3b and 3c ). Llama-3 models (Figure 10c ). Finally, we compare small models to greedy decoding with larger models, which generally performs better than sampling. We observe that even in this setup, using the smaller models several times is equivalent or preferable in all cases.We next turn to discuss the Code Llama results over the three splits of the APPS benchmark (Figures 4 to 6 ). 5 We first consider the 13B model, and observe the same trends as in HumanEval and MBPP: this model achieves the best performance in almost all fixed compute budgets. Specifically for the competition split (Figures 6b and 6c ), the most challenging APPS split, the 13B model outperforms all other models in all compute budgets, with a consistent margin of ≈5% from the 70B model when considering the wall-time budget. We further observe that the 13B model achieves similar or better performance as the greedy approach of all models in all three splits. Finally, when fixing the performance, the 13B model is 2-4 times more efficient than the 70B model (both for FLOPs and wall-time).We next observe that the 7B model is also competitive with larger models in small budget regimes (up to 8 normalized FLOPs units and 16 wall-time units). Nonetheless, it slightly underperforms the other models on larger budgets. This can be attributed to the 7B model's inability to generate a sufficient number of correct answers for the task, and may suggest that there is a minimum size requirement for a certain level of task difficulty.Our results indicate that small models can match or even outperform large ones under a fixed compute budget, assuming the availability of unit-tests. An intriguing aspect of our research question is what happens when unit-tests are unavailable, and a single selection among several generations must be made. We delve into this topic in the following section. 
evaluating code generation without unit-tests: 
 We examine the scenario where unit-tests are not available (e.g., IDE code-completion setup). In this case, an efficient selection policy strategy may be used to select one answer from the model's generations. In the previous cases (Section 2), unit-tests served as this policy.Here we investigate using ranking as a selection policy. In Section 5.1 we show how to estimate the performance of a model given such a strategy, and in Section 5.2 we analyze the performance of larger models as rankers for a small model. We assume a model that generates k outputs, and a policy that ranks them. To estimate the performance of such setup, we count the number of groups containing k generations where the highest-ranked generation within them is a correct one. That is:EQUATION)where n(>k) is the number of answers generated for the estimation, and [pass 1 , pass 2 , . . . , pass n ] ∈ {0, 1} n are the pass scores sorted according to the ranking policy. That is, pass i is 1 if the example ranked i according to the policy is correct, and 0 otherwise. See Figure 7 for a python implementation of rank-score@k.Similarly to Equations ( 2) and (3), we also define:rank-score flops @ f := rank-score@k where:EQUATION)rank-score time @t := rank-score@k where:EQUATION)where flops(k) and time(k) are the same functions as in Section 2.2. Next, we evaluate the performance of large models as rankers using the above metrics.We examine the usage of LLMs as rankers. To produce a ranking order over a set of generations, we use the averaged Negative Log Likelihood (NLL) the LLM assigns to each generation (excluding the prompt), and rank the generations according to that score. It should be noted that extracting the NLL of a model over a given generation can be done in a parallel manner (i.e., non-autoregressively), which is substantially faster than traditional token-by-token generation. The score given by a model to a generation G = (w 1 , . . . , w l ) given a prompt P is:EQUATION)To study the performance of LLMs as rankers we use the HumanEval and MBPP benchmarks. We use 2, 000 generations produced by Code Llama 7B as described in Section 3.3. As rankers we use all four Code Llama model sizes. We discard any generation that fails to complete, i.e. reached the maximal number of generated tokens without producing an end-of-sequence token. We also report the performance of running each model independently with one generation budget (both greedy and sampling). Figure 8 : rank-score time @t as a function of wall-time for HumanEval (left) and MBPP (right), using different rankers (different lines). Greedy sampling is marked as a star, and top-p sampling as a circle. While ranking results improve with the size of the ranker and with compute budget, they still fall short of greedy decoding with larger models.Our results are presented in Figure 8 . As can be seen, using LLMs as rankers over generations obtained from smaller models improves performance. Interestingly, we observe that using a 7B model as a ranker for itself can enhance its generation even further than the greedy approach, albeit with the cost generating several outputs. We also find that using larger models as rankers results in better perfomance. When considering a fixed compute budget, we find that it is sometimes comparable to use LLMs as rankers instead of sampling from them, as can be seen with the 13B and 34B models. However, this is not the case for the greedy approach which consistently outperforms ranking multiple generations from a smaller model given a fixed compute budget. To further check the use of external verifiers, we integrate the LEVER verifier model (Ni et al., 2023) with the Code Llama models. The LEVER approach aims to enhance code generation by learning to verify generated programs. The full LEVER pipeline involves using the NLL produced by the code generation model, error pruning based on execution, and a verifier trained on code generations with execution results. However, since we assume that no tests are available in our setting, execution pruning and execution results cannot be used. LEVER released a trained verifier over the MBPP benchmark, which we use along with the NLL scores of each model. As shown in Figure 9 , the LEVER verifier does not improve the results in the test-less setting, which is expected given that one of the main components of the approach relies on execution over unit-tests.In summary, there remains a gap to bridge between using LLMs as rankers for smaller models and using them as generators. To further promote this line of research, we release the 2, 000 generations per example produced by the 7B model for both HumanEval and MBPP (a total of 1, 328, 000 generations).6 Related Work 
model scaling: 
 Model scaling was found to be one of the key elements in the success of LLMs (Dehghani et al., 2023; Gu et al., 2023; Hassid et al., 2023; Rae et al., 2021; Chowdhery et al., 2023; Touvron et al., 2023) , with Wei et al. (2022) demonstrating how specific abilities emerge mainly after reaching a specific scale. The way language models behave when they are scaled up and their ability to adjust have been a significant factor in the creation of LLMs (Hernandez et al., 2021) . Kaplan et al. (2020) investigated the optimal model size to train for a given compute budget, while Hoffmann et al. (2022) demonstrated how scaling both model and dataset sizes improves performance across various tasks. Clark et al. (2022) analyzed the scaling properties of mixture-of-experts models, showing that scaling with the number of experts diminishes as model size increases. Recently, Gadre et al. ( 2024) provided a scaling law analysis considering downstream tasks rather than next-token prediction loss. They related the perplexity of a language model to its downstream task performance via a power law and used it to predict the top-1 error averaged over the evaluated downstream tasks.Our work differs from all of the above, as we do not claim to provide new scaling laws but rather suggest that when fixing the budget, smaller models can provide comparable or superior results to larger ones.Recent studies by Shi et al. (2024) and Mei et al. (2024) have demonstrated that under constrained compute budgets, smaller vision models can surpass their larger counterparts. Specifically, Shi et al. (2024) found advantages in using multiple image scales, whereas Mei et al. (2024) observed that smaller diffusion models perform better than larger ones when the compute budget is fixed. Our approach, which generates multiple text outputs from a small model, aligns with these findings.LLM verifiers and rankers is a growing trend, which leverages LLMs to verify and rank generations obtained from weaker and smaller models (Cobbe et al., 2021b; Uesato et al., 2022; Saha et al., 2024; Havrilla et al., 2024) . Both Cobbe et al. (2021b) and Uesato et al. (2022) leveraged an external classifier to rank LLM outputs. Specifically, in both setups the authors proposed to generate many candidate solutions and select the one ranked highest by the verifier. The authors demonstrated the applicability of using such verifiers in solving math word problems (Cobbe et al., 2021a) . Qin et al. (2023) demonstrated that LLMs can serve as efficient text rankers when considering pairwise ranking.Another line of work leveraged LLMs to evaluate the quality of smaller models (Saha et al., 2024; Dubois et al., 2023; Zheng et al., 2023; Oren et al., 2024) . Although providing a promising alternative, such evaluation suffers from biases in the larger model (Zheng et al., 2023) and reliance on hand-designed evaluation plans that impact the method's ability to generalize (Liu et al., 2023) . Large models also serve as verifiers of small ones in a speculative decoding setup, with the goal of speeding-up LLM generation (Leviathan et al., 2023; Kim et al., 2023; Chen et al., 2023) . It is also common to distill knowledge from a large model into a smaller one in order to improve efficiency (Hinton et al., 2015; Sanh et al., 2019; Xu et al., 2024) , see Treviso et al. (2023) for a survey on efficient methods in NLP.In this work, we explore the potential of LLMs as selectors of the best output of a smaller model in a fixed budget setup. Similarly to ours, Li et al. (2024) found that smaller sized LMs (7B parameters) already exhibit strong mathematical abilities when selecting the best response from k different generations. When considering code generation models, Al-phaCode Team (2023) presented impressive results on challenging coding contests tasks while generating 1M samples, and later on filtering and ranking them using Gemini-Pro LLM (Team et al., 2023) . Dou et al. (2024) proposed a method to improve code-generation models by learning a policy model using reinforcement learning methods. Lastly, Shi et al.  
discussion & limitations: 
 Our results show that using smaller models with the same amount of compute can improve LLM code-generation performance. An interesting question we do not fully address is whether, given enough compute, the larger models will overtake the smaller ones, or perhaps they will all saturate at a similar performance level at some point. Our HumanEval and MBPP results seem to slightly support the latter hypothesis (as all models begin to saturate, see Figures 2 and 3 ). However, unfortunately, due to compute constraints, our setting is restricted to exploring only a limited number of generations per model. 6 We note that despite this limitation, in practice, due to these costs our conclusions apply to most practical use-cases. We defer more expensive experiments to future work. 
conclusion: 
 In this work, we compared large language models with smaller-sized models under fixed budget constraints (i.e., FLOPs and wall-time). We evaluated the models using executionbased code-generation tasks, which provide access to unit-tests. Our findings reveal that generating multiple outputs from a 13B model may lead to gains of up to 15% over a single generation from a 70B model across five tasks. This highlights the potential of using smaller models instead of larger ones. In scenarios where unit tests or other solution verifiers are unavailable, we explored a simple ranking-based approach for candidate selection. We found the proposed ranking approach falls short in performance compared to a single output from the larger model. Our findings emphasize the importance of studying approaches for ranking LLM outputs, which hold great potential to not only improve model performance but also improve budget allocation. To further enhance this research direction we release over 1M samples from the Code Llama 7B models spanning both HumanEval and MBPP benchmarks. Figure 11 : Llama-3 performance vs. compute for the MBPP benchmark. As in Hu-manEval (Figure 10 ), larger models perform better as a function of k (Figure 11a ), but worse under a fixed compute budget (Figures 11b and 11c ). 
 
 Appendix: We present Llama-3 results for the HumanEval and MBPP benchmarks in Figures 10 and 11 , respectively.In Tables 2 to 4 presents precise pass@k results for the datasets examined (HumanEval, MBPP and APPS, respectively). Due to the infeasibility of reporting results for all k, we provide results for selected k values. Nevertheless, it is important to note that all relevant k values were calculated and used in the computation of the figures. E.g., Llama-2 7B was ≈10X faster to pre-train compared to the 70B variant(Touvron et al., 2023).Llama-3 8B/70B presents similar usage to Code Llama 7B/70B, with a difference of up to 7%.Tables2 and 3in Appendix B presents detailed results.Table 4 in Appendix B presents detailed results.For instance, generating 1, 000 answers for the 5, 000 examples of the APPS benchmark with a7B model takes about 20 days using a node of 8 A100 GPUs. 


Decision: The paper explores whether smaller language models can perform as well as, or better than, larger models under constrained computational budgets, specifically within the context of code generation tasks. The study compares the performance of CodeLLaMA models ranging from 7B to 70B parameters, utilizing benchmarks such as HumanEval, MBPP, and APPS. The results indicate that while smaller models can achieve consistent improvements through unit tests, they fall short in ranking-based scenarios without test cases.

This paper investigates an interesting question in the field of NLP: the efficiency and performance trade-offs between smaller and larger models. The experiments are well-designed, and the findings are useful and have insights. Overall, this paper presents a well-executed study on an important topic with practical implications and I recommend acceptance. Addressing the noted weaknesses and expanding the scope of experiments would significantly enhance its contributions to the field.

Reviews: review: paper_summary: paper_summary: The paper explores the hypothesis of whether larger models are better than smaller models even in limited computational budget scenarios. In other words, the question the paper explores is: can we generate more outputs from a smaller model so that it matches the performance of bigger models under a certain computational budget? The analysis focuses on the task of code generation. The models are large language models (LLMs) of different sizes and the smallest size is 7B parameters while the largest is 70B parameters. The results of the paper show that repeated use of smaller models can yield consistent improvements with gains of up to 15% across different tasks. In scenarios in which a candidate generation needs to be ranked and then selected as the best one, the smaller model fall short of the performance of a single greedy decoding output from a larger model. It is unclear from the paper the reason for this result.
strength_weakness: strength_weakness: Mainly related to the generalization of the results across different generation tasks or all types of tasks. The paper focuses solely on one task, code generation, for which there is a big interest in the software engineering community. However, it is difficult to generalize the conclusions obtained for code generation for every other generation task. The main reason is because the evaluation metrics change according to the task.
 
It is not completely clear and focused on the paper why the ranked-selection-based task (for code generation cases without unit tests) do not work well for smaller models. Previous work [1], [2] explored re-ranking approaches to other NLP problems such as machine translation in which the the ranker is an LLM trained with human-labeled supervision about the translation quality of translated outputs. In this scenario, re-ranking with this specialized ranker usually works well but no analysis of different LLMs sizes has been done. Perhaps this is an interesting research direction that complements the work here.

Overall the paper could use from experiments on other tasks besides code generation to be able to claim that smaller models are better than larger ones. 

[1] https://arxiv.org/abs/2205.00978
[2] https://aclanthology.org/2023.emnlp-main.733/
paper_summary: paper_summary: This paper studies an interesting questions of given the same budget of compute or wall-time, will the smaller models outperform larger models given their lower inference cost. Experiments are conducted with the CodeLLaMA models ranging from 7B to 70B, on three function-level code generation benchmarks, HumanEval, MBPP and APPS. Results show that when test cases are available, smaller CodeLLaMA models consistently outperforms larger models. However, when test cases are not available, larger models still perform better even when likelihood-based ranking methods are used.
strength_weakness: strength_weakness: W1. The major weakness of this paper is that only one set of models (i.e., CodeLLaMA) is studied in this work, which makes it unclear whether the same conclusions would generalize to other code-specific models (e.g., StarCoder) or general purpose-models (e.g., LLaMA);  
W2. I found it quite strange that all of the results are reported in the form of figures, and there are no concrete numbers in such figures. For reproducibility and comparison purposes, I would suggest at least putting the concrete numbers in the appendix;  
W3. There are some key experiment details that are missing, see Q5 below.
questions: questions: Q1. Do you plan to add the results for at least another set of models, either code-specific (e.g., StarCoder) or general-purpose (e.g., LLaMA)?   
Q2. Have you tried other reranking-based methods for code, such as MBR-Exec (Shi et al., 2022) and LEVER (Ni et al., 2023)? Also note that there are a couple of other heuristic-based reranking methods described in Ni et al., 2023. If not, can you comment on what the potential performance implications will be?  
Q3. Can you comment on how methods like quantization and flash-attention would affect the FLOPs and wall-time calculated in Table 1?   
Q4. For larger models, have different model parallelism methods been tried to ensure that maximal efficiency is achieved given the same compute infrastructure?  
Q5. How are the tests being used for the results reported in Section 4? For example, for MBPP there are 3 test cases, do you use all three to filter the incorrect samples, or just 1?
paper_summary: paper_summary: The paper investigates whether smaller LLMs can outperform or match the performance of larger models when constrained by the same computational budget. The authors present a approach that utilizes smaller models multiple times within a given budget and selects the best output using a ranking method. This study is situated within the broader discourse on the efficiency of model scaling and the practical constraints of deploying large LLMs.
strength_weakness: strength_weakness: 1. In practical applications, developers aim to reduce computational costs while enhancing performance. However, the paper only presents results under a constant computational budget. It would be beneficial for the study to explore ways to improve performance across various computational budgets.
2. The reliability of using large language models as rankers, as proposed in section 5.2, is questionable. The authors should conduct further analysis on this aspect.
3. In the related work section, the authors should also discuss the direction for enhancing generation speed and quality with smaller models (such as distillation).
paper_summary: paper_summary: The paper compares large and smaller LMs for the code generation under the same computation budget (FLOPs or wall-time). There are two setups in the paper depending on the availability of unit tests. When the unit tests are available, generating more outputs using smaller LLMs increases the likelihood of getting a correct answer. When the unit tests are unavailable, greedy decoding with large LMs seems to be a good choice.

The writing of the paper is clear and self-contained. As someone who is not familiar with code generation, I find the paper is easy to follow. The idea of turning the evaluation from pass@k to pass@flop and pass@time is simple but well-motivated given the current methodology of evaluation with pass@k. The paper executes the simple idea well.
strength_weakness: strength_weakness: - While we see the relative improvement of smaller LMs in pass@flop and pass@time, the absolute value of pass@flop (and also pass@k) seems to indicate that current Code Llama is far from usable. Thus I felt that the recommendation of using smaller LLMs to generate multiple outputs is not as critical as improving code generation LLMs itself.
- The paper mentioned the common practice of scaling LLMs as a sole drive for improving performance. This is not true. In the past years the community has advocated for better pretraining/fine-tuning data. Thus I feel that the research in the paper is not far ahead looking.
- In practice, there are multiple ways to improve generation speed/quality. For example, applying speculative decoding to LLM or distilling large ones to smaller ones. Those common practices are not studied in the paper thus it makes the paper have less practical value.

Comments: title: Thanks for the response
comment: Thank you for the response, I'm keeping the scores as is.
Thanks for your response. I maintain the same scoring, and look forward to your revised version.
We would like to thank you once again for taking the time to review our manuscript! Your comments and feedback are highly appreciated.

The discussion period is about to end, and we wanted to ask whether there are any more details we can clarify about our work.
We would like to thank you once again for taking the time to review our manuscript! Your comments and feedback are highly appreciated.

The discussion period is about to end, and we wanted to ask whether there are any more details we can clarify about our work.
We would like to thank you once again for taking the time to review our manuscript! Your comments and feedback are highly appreciated.

The discussion period is about to end, and we wanted to ask whether there are any more details we can clarify about our work.
We thank the reviewer for the positive feedback and constructive comments. We will incorporate the suggested discussions in the paper.
I would like to thank the authors for the response.

I think all my concerns are adequately addressed, adding the results for LLaMA-3 will definitely make the paper stronger to show that the conclusions generalize to non-code-specific LLMs.

Regarding W2 and Q3+Q4, please include these discussions in the next version of the paper as I think those would make the paper more holistic.

I'm increasing my score based on the response, good luck!