Evaluating Language Models for Efficient Code Generation

Authors: jiawei  liu
songrun  xie
junhao  wang
yuxiang  wei
yifeng  ding
lingming  zhang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-16 22:57:35

Abstract: We introduce Differential Performance Evaluation (DPE), a framework designed to reliably evaluate Large Language Models (LLMs) for efficient code generation. Traditional coding benchmarks often fail to provide reliable insights into code efficiency, due to their reliance on simplistic test inputs and the absence of effective compound metrics. DPE addresses these issues by focusing on efficiency-demanding programming tasks and establishing an insightful compound metric for performance evaluation. DPE operates in two phases: To curate efficiency datasets, it selects efficiency-demanding tasks from existing coding benchmarks and generates computationally expensive inputs to stress the efficiency of LLM solutions. To assess the code efficiency, DPE profiles the new solution and compares it globally against a set of reference solutions that exhibit distinct efficiency levels, where the matched level defines its efficiency score. As a proof of concept, we use DPE to create EvalPerf, a benchmark with 121 performance-challenging coding tasks. Our comprehensive evaluation draws interesting findings on the efficiency impact of model sizes, instruction tuning, and prompting. For example, while the scaling law fails to account for code efficiency, general instruction tuning benefits both code correctness and efficiency. We also evaluate the evaluation by examining the effectiveness of DPE, showing that EvalPerf is reliable and convenient to use even across platforms.

Main paper: 
 introduction: 
 With the increasing usage (GitHub, 2023; Amazon Web Services, 2023) of Large Language Models (LLMs) for code generation, comprehensively evaluating these LLMs is crucial for finding the next advancements. As such, the functional correctness in code generation (Chen et al., 2021; Austin et al., 2021) has been well-studied, where given a coding instruction in natural language, LLMs produce solutions whose correctness is assessed through test execution.While code correctness ensures the program performs its intended behaviors accurately, code efficiency is equally crucial for building high-quality software. With the massive 1. Dimension: While the correctness evaluation of code generation has been well studied, we deliver a new and important aspect to the community by studying the data curation and assessment for the efficiency evaluation of LLM-generated code. 2. Technique: We propose Differential Performance Evaluation (DPE) for effective efficiency evaluation. DPE curates performance-demanding coding tasks by sampling synthesized test input generators and using filters to ensure evaluator quality. A solution's efficiency is then globally assessed by referencing representative solutions. 3. Benchmark: Using DPE we create EVALPERF, including 121 performance-exercising programming tasks and test inputs. We also fully open-source and maintain the data curation pipeline and evaluator at github.com/evalplus/evalplus as part of EvalPlus. 4. Study: We extensively study the code efficiency of popular LLMs and draw interesting findings regarding the efficiency impact of model sizes, instruction tuning, and promptings. We also show that DPE can create inputs that are more performance-exercising than prior art by 4.8× and EVALPERF can lead to consistent performance evaluation even across various platforms. 
differential performance evaluation: 
 Figure 1 illustrates the overview of Differential Performance Evaluation (DPE), including (i) how to create a performance evaluation dataset to differentiate code performance ( § §2.1 to 2.4); and (ii) how to evaluate new code solutions using the created dataset ( §2.5).At the high level, the input to the dataset creation phase is a set of programming tasks, e.g., from HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) . As output, DPE produces a subset of performance-exercising tasks equipped with challenging test inputs. Specifically, we follow the steps below to transform and select performance-exercising tasks:1. Valid solution curation: Given a programming task, we collect a rich set of correct solutions by sampling various LLMs and test execution. 2. Performance-exercising input generation: We maximize the performance difficulty of a coding task by synthesizing a test generator aiming to produce costly test inputs. 3. Performance-exercising task selection: We profile validated solutions using performanceexercising inputs and filter out tasks using various quality criteria. 4. Performance clustering: Based on the profiled performance, solutions for each task are partitioned into several clusters for performance reference at evaluation time.During evaluation, if passing the correctness tests, the new solutions are profiled to compare against the reference solutions. Specifically, from slow to fast, the cumulative ratio of the cluster that includes the matched reference solution is the efficiency score of the evaluated solution. For example, in Figure 1 the new solution (the green box) matches the efficiency of the representative solution of the "100%" cluster, leading to a score of 100 (%) for this task.To start with, DPE takes a set of programming tasks as inputs and assumes such tasks are equipped with task descriptions (i.e., base prompts for the LLMs) and correctness tests. For example, these tasks can come from existing coding benchmarks such as HumanEval and MBPP. Next, we sample plausible solutions from diverse code LLMs and validate these solutions via test execution. Because correctness is the prerequisite for performance, we comprehensively validate plausible solutions using the rigorous tests from EvalPlus (Liu et al., 2023a) .While performance-exercising inputs are crucial, automating their creation can be challenging, as it is not always as simple as producing large integers. This is because these programming problems can define "large" inputs differently and require various structural and semantic constraints. For instance, when testing is prime(n), a randomly large n often leads to a quick path of False since most numbers are divisible by smaller numbers (e.g., half of the integers are divisible by two). Instead, the desirable test inputs are large prime numbers.Generate function `perf_input_gen(scale: int)` to produce a "large" input to exercise the efficiency of the `prime_num` function:" To this end, we propose Synthesizing a Synthesizer (SAS) to automatically produce performance-exercising inputs of different programming tasks by prompting powerful code LLMs to generate test generators. Further-more, the generator function is controllable through a scale factor, allowing for tuning the complexity of generated test inputs.Generating input generator. SAS applies few-shot prompting with Chain of Thoughts (CoT) for input generator synthesis, exemplified in Figure 2 . Specifically, the end goal of prompting is to obtain the generator function (i.e., perf input gen) illustrated in the "Input Generator" block (at the bottom). The generator function takes a scale factor as input and outputs performance-exercising test inputs according to the scale. The generator is expected to respect monotonicity over the scale factor, i.e., a larger scale factor should lead the generator to produce a more challenging input. The context for generating the generator includes three parts: (i) an instruction clarifying the goal of code generation; (ii) a ground-truth solution helping the LLM understand the overall semantic and complexity; and (iii) a few question-answer pairs to activate CoT reasoning of the task complexity. By initializing the prompt using such few-shot samples, we further load the "instruction" and "ground-truth solution" block for a programming task under test synthesis and let a generative LLM follow the few-shot demonstration and produce a test input sampler.For each coding task, we sample performance-exercising inputs by running the generator function (i.e., perf input gen) using different scale factors (i.e., scale). Specifically, we start by setting the scale factor to 1 and sample test inputs by doubling the factor round by round. The sampled test inputs are evaluated through execution, and we stop generation when an input hits a time or memory limit on any validated solution ( §2.1). By expanding the scale factor exponentially, we obtain the most performance-exercising input within our computational limits. Meanwhile, we use test execution to drop ill-formed generators and retry another sample of input generators at failure.Prior work (Liu et al., 2023a; Li et al., 2022) also prompts LLMs to generate test inputs directly. However, such an approach is inapplicable for generating challenging inputs whose text representation can be huge, blowing up the context limits of LLMs. Meanwhile, it is also hard for LLMs to strictly follow the structural requirements and diversify the inputs during long-context generation. Hence, we highlight the indirect generation of complex test inputs via input generator synthesis. Recently, Zhang et al. (2023) proposed ALGO which uses ChatGPT Code Interpreter to create input generators for exhaustive validation. Different than ALGO, SAS aims for performance-exercising inputs via few-shot CoT and scale tuning, which does not rely on the powerful ChatGPT Code Interpreter.Even with the most challenging test input, a programming task might not meet the requirement of performance diversity and runtime variation (e.g., add two numbers). Consequently, we propose filtering strategies to drop undesired programming tasks. For every programming task, we profile all valid solutions {s 1 ,s 2 ,•••,s n } curated in §2.1 multiple times. Therefore, for solution s i we profile its execution for k times and obtain a list of execution time1 T i = [t 1 ,•••,t k ].As such, a selected programming task must meet the following criteria:1. Sufficient computation: A performance-exercising task must experience a reasonably long execution. As such, we require min{mean(T i ) | i ∈ [1,n]} > t thresh, meaning that the execution of any solution must run longer than t thresh . 2. Low performance variation: We requireP 99 {CV(T i ) | i ∈ [1, n]} < CV thresh , where CV(T i ) = std(T i ) /mean(T i ), i.e., coefficient of variation, and P 99 is the 99% largest variation. 3. Performance diversity: We apply a clustering method (to be discussed in §2.4) to adaptively cluster the solutions into several groups at different levels of efficiency. Therefore, we require the number of output clusters to be greater than K where K > 1.Given a performance-exercising task, we cluster its reference solutions by their performance characteristics and use them to differentiate new solutions at evaluation time. . Following this, in Line 7, the algorithm employs an adaptive thresholding mechanism to determine the segmentation points over the sorted T. Considering Figure 6 , which indicates that smaller executions often exhibit larger variations, Line 7 calls the adaptive thresholding function in Line 1 which produces larger thresholds for smaller runtime. Specifically, to build a segmentation point at t i (left inclusive), the algorithm mandates that δ ti > bias+ √ w / ti , where bias and w are hyper-parameters of the threshold minima and scale, respectively. Therefore, in Line 8, the algorithm returns the runtime clusters for each task by splitting them over the provided segmentation points.In the final stage of dataset curation, we retain the slowest solution per cluster along with its cumulative percentage in each task. These solutions serve as benchmarks for efficiency scoring. During evaluation, both the reference solutions and the new solution are profiled in the same environment for performance assessment. Notably, we opt for a single representative solution per cluster as profiling all solutions during evaluation would be extremely time-consuming. Differential Performance Score. For each task, we denote the reference solutions from the m clusters as [s 1 ,s 2 ,•••,s m ] and their corresponding cumulative ratios [r 1 ,r 2 ,•••,r m ] (r 1 > 0 and r m = 100%). Meanwhile, we profile each reference point multiple times and obtain its mean runtime ti . Given a new and validated solution s * to evaluate, we use the same profiling procedure and obtain a mean runtime of t * . As such, we compute a Differential Performance Score (DPS) for s * as max {0}∪{r i | ti > t * } i∈[1,m] , i.e., the cumulative ratio of the reference that is immediately slower than s * . In addition, we also provide a normalized version of DPS by ablating the volume of solutions at each level, i.e., DPSnorm = max {0}∪{ i m | ti > t * } i∈[1,m]. The above discusses how to compute the performance score for one coding task, to compute the dataset-wise performance score we simply average over all passed tasks. Exemplification. For example, a DPS of 80% implies that overall the LLM generates code whose efficiency can match or improve 80% of all LLM-generated solutions. Similarly, a DPS norm of 80% indicates that the code efficiency of the LLM can overall match or improve 80% of performance clusters, by ignoring the size of each cluster. Lastly, DPS is not free as it requires to collect reference solutions of diverse efficiency levels. 
evalperf: a benchmark for code efficiency evaluation: 
 Based on Differential Performance Evaluation, we build EVALPERF, a new dataset with 121 performance-exercising programming tasks, for effective evaluation of code efficiency.We put HumanEval+ (164 tasks) and MBPP+ (399 tasks) together as the initial tasks to DPE, given their more rigorous tests (Liu et al., 2023a) to safeguard correctness. To echo §2.1, we sample and test code solutions from 21 open LLMs that achieve over a pass@1 score of 50 on the EvalPlus leaderboard (Xie et al., 2024) , where we sample 50 solutions for each model at a temperature of 0.8 for diversified generation. Next, we generate test input samplers ( §2.2) using two few-shot samples and we start the generation at the "Chain of Thoughts" block in Figure 2 right after providing the ground-truth solution. Specifically, we use AWQ-quantized (Lin et al., 2024) DeepSeekCoder-instruct-33B (DeepSeek AI, 2023) as the generative LLM to produce 16 input generator samples for each task at a temperature of 0.8. We then sample concrete test inputs from these generators, starting with a scale factor of 2 1 and increasing it exponentially until hitting the 20-second time wall or the 16GB memory wall. Of course, some generators could be broken and we filter them out via the running itself and its generated tests.Task selection and clustering require profiling of these solutions. Specifically, we use the number of executed assembly instructions as the profile metric. This profile can be easily and natively obtained by querying the Performance Monitoring Units (PMU) of modern CPUs through system calls, such as perf event (Weaver, 2013a) in Linux, which is pervasively available on most platforms. Compared to physical runtime, the #instruction is much more stable, resulting in negligible variation. Compared to software profilers such as architecture simulators, hardware counters provide low overhead (Wikipedia, 2024) and are easy to use through simple system calls. As such, we profile the #instructions for each solution over the performance-exercising input and filter the tasks using the criteria in §2.3. Specifically, we filter out tasks whose solutions can finish in t thresh = 10k instructions which is the scale of instructions for printing "hello world". We omit the variation criterion as we use the hardware performance counters for cost measurement. For clustering ( §2.4), we set the base threshold (i.e., bias) as 20% and the weight (i.e., w) as 10k instructions for the adaptive threshold function in Figure 3 , and for diversity require each task to have at least K = 4 performance clusters. As such, we build EVALPERF, a dataset with 121 performance-exercising coding tasks, equipped with computationally challenging inputs and solutions for performance reference.Lastly, our future efforts will continuously extend EVALPERF using more coding tasks. 
evaluation: 
 In §4.1 we study the code efficiency of recent code LLMs on EVALPERF and in §4.2 we evaluate the effectiveness of Differential Performance Evaluation.Setup. Following recent work (Lozhkov et al., 2024) , we evaluate the performance as well as the correctness of programs synthesized by a series of open model families, including CODEL-LAMA (Rozière et al., 2023) , DeepSeekCoder (DeepSeek AI, 2023), StarCoder (Li et al., 2023) , and StarCoder2 (Lozhkov et al., 2024) . For proprietary models, we evaluate GPT-4 Turbo (Ope-nAI, 2023) (i.e., gpt-4-0125-preview) which is to date the leading model on the EvalPlus leaderboard (Xie et al., 2024) . Specifically, we use up to four variants for each model type:1. base: the base pre-trained model without instruction tuning. 2. instruct: the instruction-tuned model using its specialized instruction format. 3. perf-instruct: the instruction-tuned model using a prompt asking the model to "solve the programming task efficiently by writing a fast implementation". 4. perf-CoT: the instruction-tuned model with a zero-shot chain-of-thought prompt (Kojima et al., 2022) by adding "Think step by step" before the perf-instruct prompt.By default, we generate 50 samples at a temperature of 0.2 for each programming task following Lozhkov et al. (2024) . Yet, for cost mitigation, we limit it to 10 samples for GPT-4 Turbo. This approach is justified by the GPT-4 Turbo's higher accuracy in generating correct code. For correctness evaluation, we compute a comprehensive pass@1 on the sum of 164 HumanEval+ tasks and 399 MBPP+ tasks. For code efficiency analysis, for each task, we compute the average DPS and DPS norm values for the first 10 correct solutions in the 50 samples and report the score averaged across all tasks. Notably, we compare models in a pairwise fashion and compute the efficiency scores over the common set of passing solutions to eliminate correctness inconsistency.Impact of instruction tuning. Code instruction tuning finetunes the base model over high-quality code which can significantly improve the correctness in code generation (Wei et al., 2024; DeepSeek AI, 2023) . Surprisingly, as is suggested by Figure 4 , correct code generated by instruction-tuned models also tends to be more efficient than that of the base model (except for StarCoder2-15B). For example, instruction-tuned DeepSeekCoder-6.7B improves the base model by 19% regarding DPS. This interesting finding implies that general instruction tuning methods can improve multiple code quality aspects beyond correctness, even if the existing instruction tuning methods were not designed to optimize code efficiency.For instruction-following models, besides the general chat template (i.e., instruct), we also use two performance-encouraging zero-shot prompting methods (i.e., perf-instruct and perf-CoT). Overall the performance-encouraging prompts neither consistently nor noticeably improve the code efficiency compared to using the basic prompting method. This shows that existing models are still weak in following such instructions, calling for future work to improve the instruction-following abilities of code LLMs. In the Appendix, Table 3 also shows that performance-encouraging prompts commonly lead to correctness degradation in code generation.Impact of model sizes. It has been a general conclusion that larger models within a model family can often generate more accurate code. Instruct improves the DPS of the 1.3B version by 14% and there is an 8.7% improvement from StarCoder2 7B to 15B. However, performance degradation with > 1% loss also happens 4 times, e.g., in the worst case, there is a 6% degradation between the 3B and 7B versions of StarCoder2 base models. This underscores a new finding -the scaling law (Kaplan et al., 2020) persists for code correctness but does not seem explicit for code efficiency, calling for future research in modeling and data curation to improve efficiency in code generation models.Published as a conference paper at COLM 2024 Lastly, we defer more evaluation details in Appendix A.3.SAS vs. EvalPlus. EvalPlus (Liu et al., 2023a) generates an abundance of test cases, some of which could be already performance-exercising. Therefore, we use EvalPlus as a baseline to evaluate the performance difficulty of test inputs generated by SAS. Specifically, we use the filtering criteria in §2.3 as the evaluator and compare the number of retained tasks using the most challenging inputs of both methods, i.e., more is better.Table 1 shows the results. We start with a total of 563 tasks from Hu-manEval+ and MBPP+. Of these, for 342 tasks, we are able to obtain at least 10 validated solutions ( §3).Using the criteria of "enough computation" which asks for test execution of over 10k instructions, the most performance-challenging EvalPlus inputs pass 204 out of 342 tasks (i.e., Table 1 : Retained tasks after different filtering phases using inputs from EvalPlus and SAS. C ≥ 10 refers to tasks with at least 10 correct solutions from sampling. 60%), whereas SAS improves it by 1.3× for enabling 271 tasks (i.e., 79%) with inputs that exhibit larger computation. Meanwhile, since our cost measurement is based on hardware counters ( §3), the variation-related criterion does not apply here. The clustering criterion allows tasks with at least 4 clusters of different performance characteristics. By applying this, EvalPlus-enabled tasks further reduce to 25 out of 204 (i.e., 88% drop), whereas SAS stands out by passing 121 tasks, resulting in a relative improvement of 4.8×.A usable and reliable benchmark must easily run on various platforms and draw consistent conclusions. As such, we study the result consistency of EVALPERF over different test beds. Table 2 lists the DPS and DPS norm of three instruction-tuned models over 4 test-beds, covering a wide range of configurations covering desktop-, workstation-, and server-scenarios. With the emergence of heterogeneous CPUs, we also include a desktop using a heterogeneous architecture, i.e., i9-12900K with 8 performance cores and 8 efficiency cores. All of these test beds are equipped with hardware counters (which are widely available), allowing for efficient profiling of #instructions. Specifically, Table 2 demonstrates that EVALPERF overall leads to very consistent conclusions, with a maximum coefficient of variation at 0.4%. Meanwhile, the evaluation takes approximately no more than 15 minutes for most evaluated models (i.e., up to 10 profiled solutions for each of 121 tasks). This highlights the reliability and usability of the EVALPERF dataset as well as the DPE methodology. The low cross-platform variation comes from two major design choices: (i) Differential evaluation: in DPE the score is determined by the relative position compared against reference solutions which differentiate each other significantly; and (ii) Hardware performance counters: using hardware counters we can efficiently obtain reliable #instructions of profiled execution despite system noises.  
related work: 
 The correctness of general code generation is one of the most studied evaluation aspects. APPS (Hendrycks et al., 2021) and MBPP (Austin et al., 2021) curate Python problems with tests from coding websites. Meanwhile, HumanEval (Chen et al., 2021) includes 164 Python programming tasks manually designed from scratch. Yet, Liu et al. (2023a) shows that existing benchmarks have limited tests and proposes to extend the test coverage using automated test generation, creating HumanEval+ and MBPP+. Meanwhile, these Python tasks are translated to other languages for multilingual evaluation (Cassano et al., 2022; Athiwaratkun et al., 2022; Zheng et al., 2023) . Furthermore, benchmarks also cover important domains such as data science (Lai et al., 2022; Yin et al., 2022; Zan et al., 2022) , repository-level generation (Ding et al., 2023; Liu et al., 2023b) , software development (Jimenez et al., 2023) , security (Pearce et al., 2022) , open-domain generation (Wang et al., 2023; Zhuo et al., 2024) , and code understanding (Gu et al., 2024; Muennighoff et al., 2023; Liu et al., 2024) . It is also important to avoid contamination in evaluation (Jain et al., 2024) .In terms of code efficiency, recent work PIE (Shypula et al., 2023) creates a benchmark to evaluate the program optimization capability of LLMs given base C++ programs. Furthermore, PIE (Shypula et al., 2023) employs CPU simulators to profile code execution to address the reproducibility concern. As a general evaluation mechanism, DPE (ours) as a meta technique can be applied to evaluating code generation and optimization, and also use CPU simulators for measurements. More concretely, our EVALPERF dataset based on DPE focuses on evaluating the code efficiency of the setting of direct code generation, which is more realistic in daily software development where oftentimes a base reference program is not available. Meanwhile, for measurement of computational cost, EVALPERF uses hardware performance counters that are low-overhead, reliable, and easy to use, explained in Appendix A.2.At the time when the paper is accepted, there have been emerging sibling benchmarks for evaluating efficiency in code generation (Huang et al., 2024; Qiu et al., 2024; Waghjale et al., 2024) . While these benchmarks consider additional source tasks (e.g., those from LeetCode) and efficiency aspects (e.g., memory usage), they still suffer from limitations including test computation insufficiency and relying on variation-sensitive compound metrics like average speedups. DPE as a meta-evaluation framework complements these new works and addresses these limitations by automatically creating performance-exercising test inputs and a stable compound metric mechanism for code efficiency. In addition, we suggest ablating impacts of the correctness dimension in efficiency evaluation, by comparing different models over a common set of passing tasks rather than the unaligned whole test suite (Huang et al., 2024) . 
conclusion: 
 This paper presents Differential Performance Evaluation (DPE), a novel framework to effectively assess the efficiency of code generated by Large Language Models (LLMs). By improving the computational complexity and metric mechanism of existing program synthesis benchmarks, DPE provides a general and robust approach to reasonably evaluate code efficiency. DPE includes two phases: (i) making a performance-exercising benchmark; and (ii) evaluating the global performance of new solutions. In the data curation phase, DPE transforms tasks into challenges that rigorously test code efficiency. In the evaluation phase, DPE profiles new solutions against reference solutions with representative performance characteristics. DPE is general, and based on it we create EVALPERF, a benchmark with 121 performance-challenging coding tasks.Our evaluation based on EVALPERF reveals new insights into the impact of instruction tuning, promptings, and model sizes on code efficiency. Meanwhile, the evaluation of DPE itself showcases its effectiveness, reliability, and simplicity in performance benchmarking, even across various platforms.hardware and software settings. However, because EVALPERF follows DPE to determine performance ranking by referencing representative solutions, such inconsistency becomes invisible when computing DPS on different test beds, shown in Table 2 . Last but not least, it does not mean that hardware counter can always replace simulators given that one of its major use cases is to mimic the behaviors of certain CPUs without having them physically.Relative speedup is also a commonly used compound metric that is agnostic to the measurements mentioned above. Although straightforward for direct comparisons between two programs, its applicability becomes less clear across a broad range of tasks. This is because the degree of speedup varies significantly across different tasks, with the average speedup often skewed by tasks that allow for substantial improvements, such as optimizing an iterative solution over a recursive one for the n-th Fibonacci number calculation. This variability can lead to confusion, making it challenging to discern whether observed efficiency gains reflect a holistic improvement or are merely the result of optimizations for a narrow set of highly improvable tasks.To overcome these limitations, Differential Performance Evaluation (DPE) defines a new metric, termed Differential Performance Score (DPS). Specifically, DPS for each task has a consistent range from 0 to 100% which stands for its empirical global performance position. This metric allows for an immediate understanding of a solution's effectiveness and the distance to optimal performance by comparison with benchmark solutions, providing a more nuanced and scalable approach to evaluating program efficiency improvements. 
 
 Appendix: In this section, we complement §4.1 to show more result details in the code efficiency evaluation of LLMs. Specifically, Table 3 lists the detailed correctness on three sets of tasks and the performance scores of both DPS and DPS norm on EVALPERF. The sample-wise performance scores are consolidated by calculating the average, maximum, and minimum scores within the initial 10 samples of each task and then we aggregate the task-wise score by computing the average. Notably, different than the main evaluations in §4.1, efficiency scores in this section are computed over all passing tasks within each model for global comparison, i.e., the correctness can impact the set of tasks that are used to compute the efficiency scores.Moreover, Figure 8 offers a side-by-side comparison with Figure 7 , illustrating DPS norm against DPS. Compared to DPS, DPS norm on EVALPERF tasks tend to be lower, indicating that the distribution of correct LLM samples is skewed to slower implementation. This is expected, as crafting efficient code often presents a greater challenge. Interestingly, while GPT-4 Turbo achieves the best DPS in Figure 7 , using DPS norm the best-performing model becomes DeepSeekCoder-6.7B-instruct. This indicates that GPT-4 Turbo can generate code that is faster than the majority while DeepSeekCoder-6.7B-instruct tends to generate code that is on top of various efficiency levels.Additionally, Figure 9 and Figure 10 show the correctness scores by computing pass@1 for the 121 EvalPerf tasks. Interestingly, we see a trend of reversed scaling law on CODELLAMA model family that smaller CODELLAMA models even achieve better pass rate. Despite this, other findings are overall consistent with that of Figure 7 and Figure 8 . For clarity, in this section, we use "execution time" or "runtime" to refer to the execution cost, which in practice can be generalized to other metrics such as the number of instructions. 


Decision: The paper introduces Differential Performance Evaluation (DPE), a novel framework for assessing the efficiency of code generated by large language models (LLMs). Unlike traditional benchmarks, DPE emphasizes real-world efficiency-demanding tasks, offering a reliable metric for evaluation. The authors transform coding tasks and develop the EVALPERF benchmark, comprising 121 performance-exercising programming tasks and test inputs. Their evaluation highlights the impact of model sizes, instruction tuning, and prompting on code efficiency, showcasing DPE's simplicity, speed, and cross-platform reliability.

This paper tackles a significant and underexplored problem in the evaluation of LLMs: the efficiency of generated code. The proposed Differential Performance Evaluation (DPE) framework and the accompanying EVALPERF benchmark are innovative contributions that offer valuable insights into code performance across various LLMs. The comprehensive evaluation, including recent high-performing models, underscores the potential of DPE to advance the field. However, several areas require improvement. Reviewer 2P7H and Reviewer TCT1 suggest expanding the benchmark to include a more diverse set of tasks and models, while Reviewer 6zda highlights the need for methodological clarity and rigorous comparison across models. Addressing these concerns, particularly by providing reproducibility through released code and enhancing methodological transparency, would strengthen the paper. Despite these limitations, the paper's contributions are significant, and with revisions, it holds promise for substantial impact in the field.

Reviews: review: paper_summary: paper_summary: This paper introduces Differential Performance Evaluation (DPE), a framework for assessing the efficiency of code generated by LLMs. Unlike traditional benchmarks, DPE focuses on real-world efficiency-demanding tasks and provides a reliable metric for evaluation. By transforming coding tasks and creating the EVALPERF benchmark, the authors highlight insights into the impact of model sizes, instruction tuning, and prompting on code efficiency. The paper's evaluation underscores DPE's simplicity, speed, and cross-platform reliability, marking a significant advancement in code generation evaluation.
strength_weakness: strength_weakness: 1. The paper could potentially benefit from a more extensive comparison of EVALPERF Benchmarks across different open-source LLMs.

2. To enhance the paper, releasing the code used to reproduce the results would be beneficial.
questions: questions: n/a
paper_summary: paper_summary: This paper attempts to create a benchmark for performance evaluation of generated programs by utilizing existing sets of programs and seeing how well the model performs with respect to these. Overall, I definitely liked the paper and think that it could potentially be a good contribution to the conference.
strength_weakness: strength_weakness: I still like this paper and think it should be accepted, but I did have a few concerns.

--- Major comments

One major concern that I had with the methodology, however, is that it seems that performance is only measured on programs where the answer is correct? If tht is the case, doesn't that introduce a major confounding factor that weaker models may generate answers to only the simpler programming problems, which are also easy to generate efficient solutions for? It seems that in order to rigorously compare two models with respect to their performance, the models would have to be compared across the same set of programs (so probably take the intersection of the sets of correct programs generated by the two models). I wonder if the authors 

Another concern is that I found the methodology complex and pretty difficult to follow -- I hope that the clarity of the paper could be improved before it is finally published. Specifically, I found sections 2.4 and 2.5 pretty hard to follow due to their completxity and brief description. I was able to mostly understand them after a careful reading, but I would definitely encourage the authors to re-read these sections and try to make them clearer. Specific comments:
- Adding an illustrative example (e.g. to Figure 3) of the time1d, rdiff, and splitters values would be helpful to better understand.
- "mean execution time" is referenced in 2.4 before it is explained in 2.5, which made it confusing.
- Is "base" a typo for "bias"?
- Unless I missed it, it seems "cumulative ratio" is not defined anywhere? This is an important concept, and it was a bit hard for me to understand the methodology without knowing what this means.

--- Minor comments

But one possible issue with it is that it tries to generate inputs that would test the efficiency of the ground-truth solution (since the ground-truth solution is provided in the prompt). Programs other than the ground truth solution may be slow in different ways, and it is not guaranteed that this method would find inputs that exercise these other programs. Did you consider also generating programs that exercise the LLM-generated solutions instead?

One limitation of this work is that it only uses HumanEval and MBPP as inputs, which use the Python standard library only. It is not clear if this framework would scale up to testing more complex programs that rely on external libraries (like numpy or or PyTorch). It'd be good to at least mention this fact in the paper.
questions: questions: --- Small typo fix suggestions

which is pervasively available on most platforms
-> which is available on most platforms

the new solution (in the green box) matchs
-> the new solution (in the green box) matches

improving the the worst case scenario
-> improving the worst case scenario

There is a place that says "Compared to software profilers such as architecture simulators, hardware counters provides low overhead (Wikipedia, 2024)", but I would suggest a reference other than Wikipedia, as Wikipedia is generally a secondary source rather than a primary authoritative source.

There is a place where you say "It is worth noting that our scoring mechanism is agnostic to different efficiency measurements." in section 5. It is not clear what this means -- don't the results of your method vary depending on the architecture used for measurment (in section 4.2)? Maybe instead of "agnostic", "robust" would be a more appropriate word?
paper_summary: paper_summary: This paper proposes a new framework to evaluate the efficiency of code that large language models generate. I believe the authors' vision towards improving the efficiency of LLM-generated code is good and important: as LLMs generate more code, they should generate efficient code rather than slower code. In addition, the authors evaluate almost all the recent high-performing code LMs and give a very comprehensive evaluation of these models. 

On the other hand, the domain of problems considered in this paper is relatively simple Python natural language to code programs. There are many more domains of programs that are optimized in real-world software engineering (such as code in different languages and performance-critical programs) that are not covered in this benchmark, which somewhat limits the scope of the work. 

The paper, however, is relatively well-written and easy to follow. I would be more inclined to recommend acceptance of this work if it covered a wider variety of performance improvements and/or a more detailed analysis that revealed more insights of performance features of LLM generated code.
strength_weakness: strength_weakness: - The size of EvalPerf is small, and there is no analysis of the variance of model scores on the benchmark. For example, it is hard to determine whether differences in the benchmark are statistically significant.
- While the paper uses prompt-based methods to try and evaluate the abilities of current language models, a more thorough attempt at asking models to generate efficient code would have been helpful.
- Scores on the benchmark are relatively high (e.g. in the 90s). The authors, however, note that DPS is a general technique that can be applied to other problems. However, the authors do not show any application of DPS to problems other than HumanEval and MBPP, which are relatively simple and already mostly saturated by the best models.
- There is no qualitative analysis
questions: questions: - To me, there are two types of efficiency: algorithmic efficiency (e.g. O(n^2) to O(n) optimizations) and smaller, code-level optimizations (e.g. changing the order of for loops in matrix multiplication). The first is more focused from an algorithmic level, while the second is generally done after writing a piece of code and profiling it. Why do you decide to unify both of these under one benchmark?
- Can you more clearly differentiate the contributions of your benchmark from those of PIE (Shypula et al. 2023)? What insights do you believe your benchmark reveals that theirs does not?
paper_summary: paper_summary: This work proposes a methodology for evaluating efficiency of a given LLM-based
code generation technique (model + prompting). It takes a selection of Python
tasks from HumanEval and MBPP datasets along with a sample of reference
solutions from different LLMs, and builds a benchmark by clustering these
solutions on LLM-generated inputs. Crucially, the inputs to elicit different
complexity clusters are generated by task-specific _programs_, themselves
LLM-generated for every task. The performance score of a code-generation
technique is then the average percentile of performance of its generated
solutions among the clustered reference solutions for each task. The authors
evaluate five families of open-weight and proprietary LLMs and demonstrate that
high code correctness does not necessarily correlate with performance.
strength_weakness: strength_weakness: # Weaknesses and Questions

Most importantly, the proposed benchmark does not show a scaling signal, which
is often a red flag for its predictive power. When scaling the model (e.g.,
among the tiers of CodeLlama) one would expect the efficiency of generated code
to increase **if explicitly instructed to do so** (i.e., in "perf-prompted"
settings). However, it actually drops according to the paper's metric even as
code correctness increases - and that can be seen for almost all LLM families.
The paper provides no rationale or qualitative evaluation for these
observations, which in turn makes it difficult to attribute the misalignment to
the evaluated models or to the benchmark methodology itself.

I would expect some qualitative evaluation of examples to analyze when more
powerful models generate slower code. The simple tasks of HumanEval/MBPP make
such analysis possible.

The input-generating technique is useful and novel, thus should be evaluated as
an independent component.
1. Does the `scale` parameter actually correlate with "challenge" of the
   generated inputs? Do bigger values of `scale` lead to slower reference times
   across all tasks?
2. How do the methodology change if we simply asked for test generators
   parameterized with a random `seed`? Might it still elicit a useful
   distribution of challenging inputs?
questions: questions: # Minor

p2, line 1: "tp perform" => "performs"

Comments: title: Thank you for the reply
comment: Thank you for response! We will include these details. Let us know if you have more questions or concerns!
Thank you for the reply, I appreciate the additional details and think adding them to the paper could be useful.
We deeply appreciate all the reviewers for their insightful feedback and suggestions for our work!

In our responses below, we address each **primary question** (denoted as **Q**) or **comment** (denoted as **C**) raised by the individual reviewers. Additionally, we will revise our paper to incorporate editorial suggestions and add more comprehensive result analysis. 

Should there be any misunderstandings of the questions, please kindly let us know. We are happy to communicate with all the reviewers throughout the discussion period.