Evaluating Language Models for Efficient Code Generation
Authors: Jiawei Liu
Songrun Xie
Junhao Wang
Yuxiang Wei
Yifeng Ding
LINGMING ZHANG
Venue: COLM (2024)
Keywords: Code Generation, Evaluation, Code Efficiency
TL;DR: We propose a new methodology and dataset to effectively evaluate the efficiency of LLM produced code.

Abstract: We introduce Differential Performance Evaluation (DPE), a framework designed to reliably evaluate Large Language Models (LLMs) for efficient code generation. Traditional coding benchmarks often fail to provide reliable insights into code efficiency, due to their reliance on simplistic test inputs and the absence of effective compound metrics. DPE addresses these issues by focusing on efficiency-demanding programming tasks and establishing an insightful compound metric for performance evaluation. DPE operates in two phases: To curate efficiency datasets, it selects efficiency-demanding tasks from existing coding benchmarks and generates computationally expensive inputs to stress the efficiency of LLM solutions. To assess the code efficiency, DPE profiles the new solution and compares it globally against a set of reference solutions that exhibit distinct efficiency levels, where the matched level defines its efficiency score. As a proof of concept, we use DPE to create EvalPerf, a benchmark with 121 performance-challenging coding tasks. Our comprehensive evaluation draws interesting findings on the efficiency impact of model sizes, instruction tuning, and prompting. For example, while the scaling law fails to account for code efficiency, general instruction tuning benefits both code correctness and efficiency. We also evaluate the evaluation by examining the effectiveness of DPE, showing that EvalPerf is reliable and convenient to use even across platforms.

Published as conference paper at COLM 2024
Evaluating Language Models for Efficient Code Generation
Jiawei Liu
Songrun Xie
Junhao Wang
Yuxiang Wei
Yifeng Ding
Lingming Zhang
University Illinois Urbana-Champaign [jiawei6, LingmingJeilling edu
Tongji University
Abstract
We introduce Differential Performancc Evaluation (DPE), framework de- signed reliably evaluate Large Language Models (LLMs) for efficient code generation. Traditional coding benchmarks often fail to provide reliable insights into code efficiency, due to their reliance on simplistictsstinuuss and the absence of effective compound metrics. DPE addresses these issues by focusing on seeiiiinncyddmmnnne progrdannngggginngggg tasks and establishing an in- sightful compound metric for performance evaluation. DPE operates in two phases: To curate efficiency datasets, selects from existing coding benchmarks and generates computationally expensive inputs to stress the efficiency of LLM solutions. To assess the code efficiency, DPE profiles the new solution and compares it globally against set of ref- erence solutions that exhibit distinct efficiency levels, where the matched level defines its efficiency score. As proof of concept, we use DPE to create EVALPERF, a benchmark with 121 performance-challene
test
tasks
coding tasks.
Our comprehensive evaluation draws interesting findings on the efficiency impact of model sizes, instruction tuning, and prompting. For example, while the scaling law fails to account for code efficiency, general instruction tuning benefits both code correctness and efficiency. We also evaluate the evaluation by examining the effectiveness of DPE, showing that EVALPERE is reliable and convenient to use even across platforms.
1 Introduction
With the increasing usage (GitHub, 2023; Amazon Web Services, 2023) of Large Language Models (LLMs)) for code generation, comprehensively evaluating these LLMs is crucial for finding the next adyancements. As such, the functional correctness in code generation (Chen et al., 2021; Austin al., 2021) has been well-studied, where given coding instruction nat- ural language, LLMs produce solutions whose correctness is assessed through test execution.
While code correctness ensures the program performs its intended behaviors accurately, code efficiency is equally crucial for building high-quality software. With the massive
Synthesizing Synthesizer Perf-Exercising b 8, 1
Profiling O
RM Task Prompt
def fib(n):
Inputs
for in range (n): a, b b, return
"n"Return n-th Fibonacci nunber >0> fib = 55
(1) 1
New Solution
Perf Matching
{ 1924
Validated Solutions
38%
58%
88%
100%
Eerf-Exxrcising Task Selection
if & 1:
X
return
return fib(n-1) fib(n-2)
Profiling
Enough comuuee
old Solutions
Perforsance diversity
Performance Clustering
Figure 1: Overview of Differential Performance Evaluation

---

Published as a conference paper at COLM 2024
deployment of coding copilots, these assistants can help developers write low-latency, scalable, and cost-effective code by suggesting efficient algorithms, data structures, and coding patterns. More importantly, code execution can be a bottleneck for running the emerging Program-aided Language Models (Gao et al., 2023) (e.g., GPT-44,, motivating the generation of efficient code toward a smooth user experience.
Reasonably evaluating the code efficiency is important yet challenging. A naive evaluation approach may simply record the execution runtime of validated solutions from existing bench- marks. However, such a strategy fails to provide reliable performance insights for two reasons:
Limitation #1: Light computation. Existing coding tasks commonly involve minimal computation, caused by small test inputs and simplistic control flow (e.g,, adding two numbers). However, is not evaluation-friendl. regarding code efficiency because lighter computation can incur larger result flakiness at orders of magnitude (Appendix A. due to its sensitivity to system noises. Meanwhile, code efficiency becomes less important a tiny scale since all complexities are "equal" when N is small. For example, a recursive implementation can be no slower than an efficient iterative solution when computing the first few Fibonacci numbers. Consequently, it is more meaningful to study code efficiency over large-scale data.
Limitation #2: Inadequate metric. Runtime speedup has been the de facto compound metric in the literature (Mendis et al., 2019; Zheng et al., 2020; Baghdadi et al., 2021) of efficiency optimization. While speedup is straightforward when studyingg single optimization subject, averaging speedups over multiple tasks is confusing when interpreting the overall code efficiency of an LLM. For example, assuming model A is slower than B by 2x on 99 tasks while outperforming B on the only task by 100x, the average speedup says code from A is generally faster than that of by 0.5x99+100x1 11.455x, which may not align with general 1000
user perception. While we defer detailed discussions in Appendix A.2, the misperception comes from the huge scale variation of speedups across different tasks, calling for a more insightful compound metric for code efficiency.
To this end, we propose Differential Performance Evaluation (DPE), general framework to curate cerformaanc--exeriii evaluation. From the data and metric perspective, DPE argues that effective performance benchmarking requires: (i) efficiency-char effficienc-cchallenginng programming tasks to differentiate code solutions, and (ii) an unsightly metric to tell how far an LLM is to generate empirically optimal code. At dataset curation time, DPE takes set of coding tasks input and fransforms them into tasks worth practicing for efficiency. Specifically, DPE generates performance-exerrising test inputs for each task by Synthesizing Synthesizer (SAS). SAS prompts an LLM with Chain-of-Thouu (Wei et al., 2022) (CoT) few-shot learning to produce canle-controllable test input sampler. Next, we tune the sampler to generate challenging yet computable inputs via exponential input sampling. Furthermore, we design filtering strategies to pick reliable tasks for performance evaluation. For each performance-exercisinn of valid solutions and clusters them by performance characteristics. At evaluation time, DPE profiles given new solution along with reference solutions and the ranking of the matched performance cluster determines its score. Below summarizes the contributions of this paper:
programming challenges and perform effective code efficiency
task, DPE samples rich set
Dimension: While the correctness evaluation of code generation has been well studied, we deliver a new and important aspect to the community by studying the data curation and assessment for the efficiency evaluation of LLM-generated code.
2. Technique: We propose Differential Performance Evaluation (DPE) for effective efficiency evaluation. DPE curates performance-demandii
coding tasks by sampling synthesized test input generators and using filters to ensure evaluator quality. A solution's efficiency is then globally assessed by referencing representative solutions.
3. Benchmark: Using DPE we create EVALPERE, including 121 performance-xxrrcsingg programming tasks and test inputs. We also fully open-source and maintain the data curation pipeline
pipelien and evaluator at github com/ conn/valllls paalplu/evaalluu evalplus as part of EvalPlus.
4. Study: We extensively study the code efficiency of popular LLMs and draw interesting findings regarding the efficiency impact of model sizes, instruction tuning, and prompt- ings. We also show that DPE can create inputs that are more performance--xerriiing than prior art by 4.8x and EVALPERF can lead to consistent performance evaluation even across various platforms.
2

---

Published as a conference paper at COLM 2024
2 Differential Performance Evaluation
Figure illustrates the overview of Differential Performance Evaluation (DPE), including (i) how to create performance evaluation dataset to differentiate code performance ($52.1
to 2.4); and (ii) how to evaluate new code solutions using the created dataset ($2.5).
At the high level, the input the dataset creation phase a set of programming tasks, e-g., from HumanE'val (Chen et al., 2021) and MBPP (Austin et al., 2021), As output, DPE produces subset of arnfformannc-exerrisiiin tasks equipped with challenging test inputs. Specifically, we follow the steps below to transform and select performance--xerrisii
tasks:
Valid solution curation: Given a programming task, we collect rich set of correct solutions by sampling various LLMs and test execution.
Performante--xerciiingg input generation: We maximize the performance difficulty of coding task by synthesizing test generator aiming to produce costly test inputs. Performance-eeerrisin task selection: We profile validated solutions using performance- exercising inputs and filter out tasks using various quality criteria.
Performance clustering: Based on the profiled performance, solutions for each task are partitioned into several clusters for performance reference at evaluation time.
During evaluation, if passing the correctness tests, the new solutions are profiled to compare against the reference solutions. Specifically, from slow to fast, the cumulative ratio of the cluster that includes the matched reference solution is the efficiency score of the evaluated solution. For example, in Figure 1 the new solution (the green box) matches the efficiency of the representative solution of the "100%" cluster, leading to a score of 100 (%) for this task.
2.1 Valid Solution Curation
To start with, DPE takes a set of programming tasks as inputs and assumes such tasks are equipped with task descriptions (i.e., base prompts for the LLMs) and correctness tests. For ex- ample, these tasks can come from existing coding benchmarks such as HumanEval and MBPP. Next, we sample plausible solutions from diverse code LLMs and validate these solutions via test execution. Because correctness is the prerequisite for performance, we comprehensively validate plausible solutions using the rigorous tests from EvalPlus (Liu al., 2023a).
2.2 Synthesizing Synthesizer: Performance-Bxerriiingg Input Generation
While performance-xerrising inputs are crucial, automating their creation can be challenging, as is not always as simple producing large integers.
Instruction
Genarate Function 'parf_ genssale int)" produce "large" input exarcise afficiency of 'prine nusa function;
Ground Fround-truth truth Solution
This is because these programming problems define "large" inputs differently and require various struc- tural and semantic constraints. For instance, when testing is is_pr war.ppmmme im (n),
"*"Mrite function to check if number is prine"nu inport
def primme_ pr-ime_nun(nun) (nun):
return
for i in range(2, math.isart
nun
return False
return True
randomly large often leads to
s
Chain of Thoughts
quick path of False since most num- bers are divisible by smaller numbers (e-g., half of the integers are divisible by two). Instead, the desirable test inputs are large prime numbers.
Anslysis:
format: inteage 'n'
Time complexity: 0(n) complexity: 9(3)
perf? Large
A Input Generator
can reuse the prine_nus" nue function # Larger "sale' neans larger input # use case: prise__un(sport aaau(cporf_inott_geeeeea geput__gn(ssatee) def perf_ Linput_gen(scalll for in range(scale, 2, -1):
To this end, we propose Synthesizing a Synthesizer (SAS) to automatically produce cetoommance-xxecisimm in-
int):
if prine_nun(i): return (i,)
puts of different programming tasks by prompting powerful code LLMs to generate test generators. Further-
return (2,)
Figure 2: Exemplifying Synthesizing a Synthesizer.
3

---

Published as a conference paper at COLM 2024
more, the generator function is controllable through scale factor, allowing for tuning the complexity of generated test inputs.
Generating input generator. SAS applies few-shot prompting with Chain of Thoughts (CoT) for input generator synthesis, exemplified in Figure 2. Specifically, the end goal of prompting is to obtain the generator function (i.e., ptwf.inputt.ee.. gen) illustrated in the "Input Generator" block (at the bottom). The generator function takes a scale factor as input and outputs performance-exerrisin test inputs according to the scale. The generator is expected to respect monotonicity over the scale factor, i.e., a larger scale factor should lead the generator to produce a more challenging input. The context for generating the generator includes three parts: (i) an instruction clarifying the goal of code generation; (ii) ground-tru -uund solution helping the LLM understand the overall semantic and complexity; and (iii) few question-answe pairs to activate CoT reasoning of the task complexity. By initializing the prompt using such few-shot samples, we further load the "instruction" " and "ground-trun solution" block for programming task under test synthesis and let a generative LLM follow the few-shot demonstration and produce test input sampler.
Exponential sampling For each coding task, we we sample performance- by running the generator function (i.e., serf-input.gen) using different scale factors (i.e., scale). Specifically, we start by setting the scale factor to 1 and sample test inputs by doubling the factor round by round. The sampled test inputs are evaluated through execution, and we stop generation when an input hits a time or memory limit on any validated solution (52.1). By expanding the scale factor exponentially, we obtain the most ienoorraaanneeeecssiiiiii input within our computational limits. Meanwhile, we use test execution drop ill-formed generators and retry another sample of input generators at failure.
inputs
Insight. Prior work (Liu et al., 2023a; et 2022) also prompts LLMs to generate test inputs directly. However, such an approach is inapplicable for generating challenging inputs whose text represenntatinn can be huge, blowing up the context limits of LLMs. Meanwhile, is also hard for LLMs to strictly follow the structural requirements and diversify the inputs during long-context generation. Hence, we highlight the indirect generation of complex test inputs via input generator synthesis. Recently, Zhang et al. (2023) proposed ALGO which uses ChatGPT Code Interpreter to create input generators for exhaustive validation. Different than ALGO, SAS aims for performancce-xxrrciistg inputs via few-shot CoT and scale tuning, which does not rely on the powerful ChatGPT Code Interpreter.
2.3
ieftormannc-Exercissi Task Selection
Even with the most challenging test input, a programming task might not meet the requirement of performance diversity and runtime variation (e.g. add two numbers). Con- sequently, we propose filtering strategies to drop undesired programming tasks. For every programming task, we profile all valid solutions (s ) curated in $2.1 multiple times. Therefore, for solution s; we profile its execution for & times and obtain a list of execution time- [ At]. As such, a selected programming task must meet the following criteria:
1. Sufficient computation: A performance-exercising task must experience a reasonably long execution. As such, we require min(mean meaan (7,) iE [1,x]] > fireesh meaning that the execution of any solution must run longer than fthresh.
2. Low performance variation: We require Poo (C)) | E [1, < CVthreshr where CV(T)) mmeemt// i.e., coefficient of variation, and Pog is the 99% largest variation.
3. Performance diversity: We apply clustering method (to be discussed in $2.4) to adaptively cluster the solutions into several groups at different levels of efficiency. Therefore, we require the number of output clusters to be greater than K where K>1.
2.4 Adaptive Performance Clustering
Given performance-exerrisigg task, we cluster its reference solutions by their performance characteristics and use them to differentiate new solutions at evaluation time.
For clarity, in this section, we use "execution time" or "runtime" to refer to the execution cost, which in practice can be generalized to other metrics such as the number of instructions.
4

---

Published as a conference paper at COLM 2024
1: def t # smaller runtime = higher variation, needs larger thresh
2: return BIAS nath.sqrt t)
3:
4: def clusterid(timeld) List[float]) - List[List[fLoat])
5: timeld sort(tmmee)))
slow fast
6: rdiff = -np. (pp.dif((timedd) / timeld[:-1] relative drop in 0-100%
7: splitters [i+1 for i, in enumerate (rdiff) if thresh(timeld(11)
8: return np. spiimed splitters)
# return list of clusters
segmentsolutions Figure The algorithm to adaptively segment solutions for each task based on their efficiencc
Figure 3 elucidates our adaptive clustering algorithm. Commencing from Line the cluster- ing algorithm takes a list of mean execution time (i.e,, 1-dimension) input and dynamically partitions them into clusters based on their relative scale. Subsequently, we sort the execution time from slow to fast in Line 5. Denoting the sorted time 1d as = [F,,2, -x),, in Line 6, we
compute the relative difference in percentage, i.., ó1, 1-
Following this, in Line 7, the al- gorithm employs adaptive thresholding mechanism to determine the segmentation points over the sorted T. Considering Figure 6, which indicates that smaller executions often exhibit larger variations, Line calls the adaptive thresholding function in Line 1 which produces larger thresholds for smaller runtime. Specifically, to build a segmentation point f; (left inclu- sive), the algorithm mandates that of; > bias + w/F, where bias and w are hyper-parameters of the threshold minima and scale, respectively. Therefore, Line 8, the algorithm returns the runtime clusters for each task by splitting them over the provided segmentation points.
2.5 Efficiency Scoring by References
In the final stage of dataset curation, we retain the slowest solttion per cluster along with cu- mulative percentage in each task. These solutions serve as benchmarks for efficiency scoring. During evaluation, both the reference solutions and the new solution are profiled in the same environment for performance assessment. Notably, we opt for single tattees per cluster as profiling all solutions during evaluation would be extremely time-consuming.
Differential Performance Score. For each task, we denote the reference solutions from the clusters as (5. and their corresponding cumulative ratios ["1,",,"-,,' (r1 >> and Im =00%). Meanwhile, runtime f. Given a new and validated solution s* to evaluate, we use the same profiling pro- cedure and obtain a mean runtime of f*. As such, we compute Differential Performance Score
profile each reference point multiple times and obtain its mean
(DPS) for s* max (r (orn(&>>>etppp )e((mm) i.e., the cumulative ratio of the reference that is immediately slower than s*. In addition, we also provide a normalized version of DPS by ablating the volume of solutions at each level, i.e., DPS = max (0) / 1,>1** "i>>") (.m]m)))
The above discusses how to compute the performance score for one coding task, to compute the dataset-wise performance score we simply average over all passed tasks.
Exemplification. For example, a DPS of 80% implies that overall the LLM generates code whose efficiency can match improve 80% of all LLM-generated solutions. Similarly, DPSporm of 80% indicates that the code efficiency of the LLM can overall match or improve 80% of performance clusters, by ignoring the size of each cluster.
Lastly, DPS is not free as it requires to collect reference solutions of diverse efficiency levels.
3 EVALPERF: A Benchmark for Code Efficiency Evaluation
Based on Differential Performance Evaluation, we build EVALPERE, a new dataset with 121 performance-exerrising programming tasks, for effective evaluation of code efficiency.

---

Published as a conference paper at COLM 2024
We put HumanEval+ (164 tasks) and MBPP+ (399 tasks) together as the initial tasks to DPE, given their more rigorous tests (Liu et al., 2023a) to safeguard correctness. To echo $2.1, we sample and test code solutions from 21 open LLMs achieve over a pass@1 score of 50 on the EvalPlus leaderboard (Xie al., 2024), where we sample 50 solutions for each model at a tem- perature of 0.8 for diversified generation. Next, we generate test input samplers ($2.2) using two few-shot samples and we start the generation at the "Chain of Thoughts block in Figure 2 right after providing the ground-truth solution. Specifically, we use AWQ-quantized (Lin (DeepSeek AI, 2023) as the generative LLM to produce 16 input generator samples for each task at a temperature of 0.8. We then sample concrete test inputs from these generators, starting with a scale factor of 2 and increasing exponentially until hitting the 20-second time wall the 16GB memory wall. Of course, some generators could be broken and we filter them out via the running itself and its generated tests.
et al., 2024) CrepSeekCoder-intrruce
Task selection and clustering require profiling of these solutions. Specifically, we use the number of executed assembly instructions as the profile metric. This profile can be easily and natively obtained by querying the Performance Monitoring Units (PMU) of modern CPUs through system calls, available most platforms. Compared to physical runtime, the #instruction is much more stable, resulting in negligible variation. Compared to software profilers such as architecture simulators, hardware counters provide low overhead (Wikipedia, 2024) and are easy use through simple system calls. As such, we profile the #instructions for each solution over the performance-exerriiing input and filter the tasks using the criteria in $2.3. Specifically, we filter out tasks whose solutions can finish in firesh = 10k instructions which is the scale of instructions for printing "hello world". omit the variation criterion as we use the hardware performance counters for cost measurement. For clustering ($2.4), we set the base threshold (i.e., bias) as 20% and the weight (i.e., w) as 10k instructions for the adaptive threshold function in Figure 3, and for diversity require each task to have at least K= performance clusters. As such, we build EVALPERF, a dataset with 121 iefformanne-exerrisiii coding tasks, equipped with computationally challenging inputs and solutions for performance reference.
as perf event (Weaver, 2013a) in Linux, which is pervasively
Lastly, our future efforts will continuously extend EVALPERF using more coding tasks.
4
Evaluation
In $4.1 we study the code efficiency of recent code LLMs on EVALPERF and in $4.2 we evaluate the effectiveness of Differential Performance Evaluation.
4.1 Evaluating Code Efficiency
Setup. Following recent work (Lozhkov et al., 2024), we evaluate the performance as well as the correctness of programs synthesized by a series of open model families, including CODEL- LAMA (Rozière et al., 2023), DeepSeekCoder (DeepSeek AI, 2023), StarCoder (Li et al., 2023), and Starodere kLozhkov et 20.,)) For proprietary models, we evaluate GPT-4 Turbo (Ope- nAI, 2023) (i.e., gpt-4-2125-previer which is to date the leading model on the EvalPlus leaderboard (Xie et al., 2024). Specifically, we use up four variants for each model type:
1. base: the base pre-trained model without instruction tuning.
2. instruct: the instruction-tuned model using its specialized instruction format.
3. perf-instruct: the instruction-tunee model using prompt asking the model to "solve the programming task efficiently by writing fast implementation".
4. perf-CoT: the instruction-tuned model with zero-shot chain-of-thought prompt (Kojima at 2022) by adding "Think step by step" before the perf-instruct prompt.
By default, we generate 50 samples at temperature of 0.2 for each programming task following Lozhkov et al. (2024). Yet, for cost mitigation, we limit it to 10 samples for GPT-4 Turbo. This approach is justified by the GPT-4 Turbo's is higher accuracy in generating correct code. For correctness evaluation, we compute a comprehensive pass@1 on the sum of 164 HumanEval+ tasks and 399 MBPP+ tasks. For code efficiency analysis, for each task, we compute the average DPS and DPS values for the first 10 correct solutions in the 50 samples and report the score averaged across all tasks. Notably, we compare models in
6

---

Published as conference paper at COLM 2024
periCoT
pert linernett
intrest
Pert-oT
Per inrfinstrect
intrest
Perl.CT
Pert Perfinstreet
imtrest
Perf-CT
per Prriiiereet
intrest
Perl-CT
Per Fostrred
imtrrst
biot
80.8
82.2
85,8
B077
78.0 75.4
80.3
82.0 78.0
67.0
79.4
perf CoT
80.2
78.8
perf instruct
81.7 77.0
B1.4
85.1
/6.4
79.4
73.4
81.8
82.4
85.5
86.5
79.
77.5
inen
82.8
95.4
81.6
80.b
86.
CodeLiama (13B)
CodeLiama (34B)
CodeLiama (70B)
CodeLiama
CodeQwen1.5 (70)
Per--CoT
20.7 76.6
77.6
83.0
81.9
83.0
82.1
03.4
82.5
91.7 91.7
83.3 01.3 82.5
70.5
74.7
85.7
78
89.1
89.7
81.5 83.7
perf-instuct
83.0
Instrutt
194
86
80.9
CospSnek-Coder
DeapSank-Codeer
CoepSmik-Coder
GPT 4 Turbo
SteCoder2 (15B)
Figure Pairwise comparison of DPS with model variant pairs. Each pair of variants is com- pared over the common set of passing solutions. Within each block, the bottom-left number comes from the corresponding variant the y-axis and the top-right number for the x-axis.
pairwise fashion and compute the efficiency scores over the common set of passing solutions to eliminate correctness inconsistency.
Impact of instruction tuning. Code instruction tuning finetunes the base model over high-quality code which can significantly improve the correctness in code generation (Wei et al., 2024; DeepSeek AI, 2023). Surprisingly, as is suggested by Figure 4, correct code generated by instruction-tuned models also tends to be more efficient than that of the base model (except for StarCoder2-15DD For example, instruction-tuned improves the base model by 19% regarding DPS. This interesting finding implies that general instruction tuning methods can improve multiple code quality aspects beyond correctness, even if the existing instruction tuning methods were not designed to optimize code efficiency.
DeepSeekCoder-e
Impact of prompting. For instruction-following models, besides the general chat template (i.e., instruct), we also use two (i.e., perf-instrus and perf-CoT). Overall the performance-encourage consistently nor noticeably improve the code efficiency compared to using the basic prompting method. This shows that existing models are still weak in following such instructions, calling for future work to improve the instruction-followree abilities of code LLMs. In the Appendix, Table 3 also shows that performance-encouraee prompts commonly lead correctness degradation code generation.
zero-shot prompting methods prompts neither
Impact of model sizes. It has been a general conclusion that larger models within model family can often generate more accurate code. Figure 5 explores how parameter sizes within the same model family impact the overall code efficiency. Within the 12 pairs in Figure there are seven cases where a larger model in the family outperforms the smaller one regarding code ef- ficiency, e.g., DeepSeekCoder-6. Instruct improves the DPS of the 1.3B version by 14% and there is an 8.7% improvement from
768
9413
138
79
1.88
158
834
8016
01
33R
158
340
86.8
83.7
82.3
6.7
78
78
1.88
38
CodeLlama (instruct)
DeepSock (instruct)
StarCodor2 (base)
Figure 5: Pairwise DPS comparison with models of different parameter sizes.
StarCoder2 7B to 15B. However, performance degradation with > 1% loss also happens 4 times, e.g., in the worst case, there is a 6% degradation between the 3B and 7B versions of StarCoder2 base models. This underscores new finding the scaling law (Kaplan et al., 2020) persists for code correctness but does not seem explicit for code efficiency, calling for future research in modeling and data curation improve efficiency in code generation models.
7

---

Published as conference paper at COLM 2024
Lastly, we defer more evaluation details in Appendix A.3.
4.2 Evaluating the Evaluation
SAS vs. EvalPlus. EvalPlus (Liu al., 2023a) generates an abundance of test cases, some of which could be already performance-exerrisii
Therefore, we use EvalPlus as baseline to evaluate the performance difficulty test inputs generated by SAS. Specifically, we use the filtering criteria in $2.3 the evaluator and compare the number of retained tasks using the most challenging inputs of both methods, i.e, more is better.
Table 1 shows the results. We start with a total of 563 tasks from Hu- manEval+ and MBPP+. Of these, for 342 tasks, we are able to obtain at least 10 validated solutions ($3). Using the criteria of "enough com- putation" which asks for test execu- tion of over 10k instructions, the most performance-challenniig. EvalPlus in- puts pass 204 out of 342 tasks (i.e.,
Total
Filtering
(C 10)
> 10k instr.
+ +# Cluster
>
EvalPlus
204
25
563 (342)
SAS (Ours)
271
121
Table Retained tasks after different filtering phases using inputs from EvalPlus and SAS. C> 10 refers to tasks with at least 10 correct solutions from sampling. 60%), whereas SAS improves by 1.3x for enabling 271 tasks (i.e., 79%) with inputs that exhibit larger computation. Meanwhile, since our cost measurement is based on hardware counters ($3), the variation-related criterion does not apply here. The clustering criterion allows tasks with at least 4 clusters of different performance characteristics. By applying this, EvallPus-enabled tasks further reduce to out of 204 (i.e., 88% drop), whereas SAS stands out by passing 121 tasks, resulting in relative improvement of 4.8x.
Cross-pl ploss-platffrm variation. A usable and reliable benchmark must easily run on various plat- forms and draw consistent conclusions. As such, westudy the result consistency of EVAL PERP over different test beds. Table 2 lists the DPS and DPSnorm of three instruction-turee over 4 test-beds, covering wide ring range of configurations covering desktop-, workstation- and server-scenarios. With the emergence heerooggneouuu CPUs, we also include a desktop using heterogeneous architecture, i.e., 19-12900K with 8 performance cores and 8 efficiency cores. All of these test testbeds beds are equipped with hardware counters (which are widely available), allow- ing for efficient profiling of #instructions. Specifically, Table demonstrates that EVALPERF overall leads to very consistent conclusions, with maximum coefficient of variation at 0.4%. Mianwhile, the evaluation takes approximately no more than 15 minutes for most evaluated models (i.e., up to 10 profiled solutions for each of 121 tasks). This highlights the reliability and usability of the EVAL. PERF dataset as well as the DPE methodology. The low cross gross-platform variation comes from two major design choices: (i) Differential evaluation: in DPE the score is determined by the relative position compared against reference solutions which differentiate each other significantly; and (ii) Hardware performance counters: using hardware counters we can efficiently obtain reliable #instructions of profiled execution despite system noises.
models
Desktop
Desktop 19-12900K 8P & Cores 64GB RAM
Workstation Pro 5975WX
Server Xeon 6442Y
17-10700K Cores 64GB RAM
CV
32 Cores
48 Cores
(%)
256GB RAM
512GB RAM
CODELLAMA-700 DPS
79.2
79.4
79.4
78.8
0.3
instruct
DPSpumm
75.4
75.9
75.2
75.1
DeepSeekCoder-33B instruct
DPS
85.4
85.6
85.5
85.4
0.1
DPS
78.6
78.5
78.6
78.4
DPS
90.5
90.0
90.7
89.9
0.4
GPT-4 instruct
DPSparm
79.9
79.8
79.9
79.9
0.1
Table 2: Cross-platform variation of the mean Differential Performance Score.
8

---

Published as a conference paper at COLM 2024
5 Related Work
The correctness of general code generation is one of the most studied evaluation aspects. APPS (Hendrycks et al., 2021) and MBPP (Austin et al., 2021) curate Python problems with tests from coding websites. Meanwhile, HumanEval (Chen et al., 2021) includes 164 Python programming tasks manually designed from scratch. Yet, Liu et al. (2023a) shows that existing benchmarks have limited tests and proposes to extend the test coverage using automated test generation, creating HumanEval+ and MBPP+. Meanwhile, these Python tasks are translated to other languages for multilingual evaluation (Cassano et al., 2022; Athiwaratkun et al., 2022; Zheng et al., 2023). Furthermore, benchmarks also cover important domains such as data science (Lai et al., 2022; Yin et.,, 2022; Zan et al., 2022), repository-level generation (Ding et al., 2023; Liu et al., 2023b), software development (Jimenez et al., 2023), security (Pearce et al., 2022), open-domain generation (Wang et al., 2023; Zhuo et al., 2024), and code understanding (Gu et al., 2024; Muennighoff et al., 2023; Liu et 2024). It is also important to avoid contamination in evaluation (Jain et al., 2024).
In terms of code efficiency, recent work PIE (Shypula et 2023) creates a benchmark to eval- uate the program optimization capability of LLMs given base C++ programs. Furthermore, PIE (Shypula et al., 2023) employs CPU simulators to profile code execution to address the reproducibile concern. As a general evaluation mechanism, DPE (ours) as a meta technique can be applied evaluating code generation and optimization, and also use CPU simulators for measurements. More concretely, our EVALPERR dataset based on DPE focuses on evaluating the code efficiency of the setting of direct code generation, which is more realistic in daily software development where oftentimes a base reference program is not available. Meanwhile, for measurement of computational cost, EVALPERR uses hardware performance counters that are low-overhead, reliable, and easy to use, explained in Appendix A.2.
At the time when the paper is accepted, there have been emerging sibling benchmarks for eval- uating efficiency in code generation (Huang et al., 2024; Qiu et al., 2024; Waghjale et al., 2024). While these benchmarks consider additional source tasks (e.g,, those from LeetCode) and effi- ciency aspects (e.g,, memory usage), they still suffer from limitations including test computa- tion insufficiency and relying on suriation-sensittee compound metrics like average speedups. DPE as meta-evaluation framework complements these new works and addresses these limitations by automatically creating performance-exercising test inputs and a stable com- pound metric mechanism for code efficiency. In addition, we suggest ablating impacts of the correctness dimension in efficiency evaluation, by comparing different models over common set of passing tasks rather than the unaligned whole test suite (Huang et al., 2024).
6 Conclusion
This paper presents Differential Performance Evaluation (DPE), a novel framework to effectively assess the efficiency of code generated by Large Language Models (LLMs). By improving the computational complexity and metric mechanism of existing program synthesis benchmarks, DPE provides a general and robust approach to reasonably evaluate code efficiency. DPE includes two phases: (i) making a performance-exeriisiii and (ii) evaluating the global performance of new solutions. In the data curation phase, DPE transforms tasks into challenges that rigorously test code efficiency. In the evaluation phase, DPE profiles new solutions against reference solutions with representative performance characteristics. DPE is general, and based on it we create EVALPERF, a benchmark with 121 performanc-challlenginn coding tasks.
benchmark;
Our evaluation based on EVALPERE reveals new insights into the impact of instruction tuning, promptings, and model sizes on code efficiency. Meanwhile, the evaluation of DPE itself showsases its effectiveness, reliability, and simplicity in performance benchmarking, even across various platforms.

---

Published as conference paper at COLM 2024
Acknowledgment
This work was partially supported by NSF grant CCF-2131943 and Kwai Inc, as well as API credits from the OpenAl Researcher Access Program.
References
Amazon Web Services. AT Code Generator Amazon CodeWhispener AWS. https://aws. amazon. ctezoon.co//coeeeee http:////ccooeesieeeeeerrrreeeee 2023.
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations, 2022.
2022..
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732 2021. URL
https://arxiv org/abs//2 atwaaa///......... 0t/.//118.0070 07732.
Riyadh Baghdadi, Massinissa Merouani, Mohamed-Hicham Leghettas, Kamel Abdous, Taha Arbaoui, Karima Benatchba, et al. A deep learning based cost model for automatic code optimization. Proceedings of Machine Learning and Systems, 3:181-193, 2021.
Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K Reinhardt, Ali Saidi, Arkaprava Basu, Hestness, Derek R Hower, Tushar Krishna, Somayeh Sardashti, et al. The gem5 simulator. ACM SIGARCH computer architecture news, 39()) 2011
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchhr Balaji, Shan- tanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan L Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
DeepSeek AI. Deepseek coder: Let the code write itself. cetps://githin com dewps//eeeee--eeeeeeeeeeeeeeeeee ai/ DeepSeek- Cooder, 2023.
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Paarminder Bhatia, Dan Roth, and Bing Xiang. Crosscodeeval: diverse and multilingual benchmark for cross-file code completion. In seirth- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: /ttp///ppppe/eeeee//eeepeeeeeeee htt///orummidwwgDccbsssst
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Frogram-aided language models. In International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023.
GitHub. GitHub Copilot Your Al pair programmer. copilot, 2023.
10

---

Published as a conference paper at COLM 2024
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-ezzma, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arv preprint ar Xi::::2:11100000 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurlPS, 2021.
Dong Huang, Jie M Zhang, Yuhao Qing, and Heming Cui. Effibench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXit:2402.020337 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamina- tion free evaluatioo of large language models for code. ar arv preprint ar Xtt::44033009774 2024.
Carlos F. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar Xio:2000.00861, 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systemss, 35:22199-222931 222133, 2022.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: natural and reliable benchmark for data science code generation, 2022.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozh- skii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmny, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023.
Mare
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbi, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378 (6924))1092-1097, December 2022. ISSN 1095-9203. doi: 10.1126 11116//cience.abg11555 URL http://dx. doi. orgoorgg///oo 10. 10e1112///seee 11226//cienceabban abq 158.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantizationnn for Ilm compression and acceleration. In MLSys, 2024.
Changshu Liu, Shizhuo Dylan Zhang, and Reyhaneh Jabbarvand. Codemind: A framework to challenge large language models for code reasoning. ar arXio preprint ar xix::22402.0966 2024.
11

---

Published as a conference paper at COLM 2024
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL wtw.htttts//openeeiiiiiii net/forumiid=1gg 10u7..
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint ar axXtt:2306.03097 2023b.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. Xrv preprint Xio:2402.19173, 2024.
Charith Mendis, Cambridge Yang, Yewen Pu, Saman Amarasinghe, and Michael Carbin. Compiler auto-vectorization with imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
Azalia Mirhoseini, Hieu Pham, Quoc Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. Device placement optimization with reinforcement learning. In International conference on machine learning, pp. 2430-2439. PMLR, 2017.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
OpenAL. Gpt-4 technical report,, 2023.
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of github copilot's ss code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 754-768. IEEE, 2022.
Ruizhong Qiu, Weiliang Will Zeng, Hanghang Tong, James Ezick, and Christopher Lott. How efficient is llm-generated code? a rigorous & high-standard benchmark. arXiv preprint arXiv:2406.0666 2024.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2023.
Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improviinn code edits. Xiv preprint ar arX::2332..07867 2023.
Siddhant Waghjale, Vishruth Veerendranath, Zora Zhiruo Wang, and Daniel Fried. Ecco: Can we improve model-generated code efficiency without sacrificing functional correctness? ar Xiv preprint ar Xiv:2407. 14044, 2024.
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-based evaluation for open-domain code generation. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Vincent M Weaver.
Linux perf event features and overhead. In The 2nd international workshop on performance analysis of workload optimized systems, FastPath, volume 13, pp. 5, 3,2011a. 3a.
Vincent M Weaver. Linux perf event features and overhead. In The 2nd international workshop on performance analysis of workload optimized systems, FastPath, volume 13, pp- 5, 2013b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of -ftthugght prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
12

---

Published as conference paper at COLM 2024
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang, Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning, 2024.
Wikipedia. Hardware performance counter wikipedia, 2024.
URL
https:
//en. weeipwiiipddi.comm wwkipedia..rg/wwkii/ieii........
Songrun Xie, Junhao Wang, and Jiawei Liu. Evalplus leaderboard, 2024.
URL
https://evalplus.ee www.. wtt.tub..oo/eadebbbaaaaaaaaa html.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, al. Natural language to code generation in interactive data science notebooks. arXiv preprint arXiv:2212.092243 2022.
Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. Cert: continual pre-training on sketches for library-oriented code generation. ar Xrviv preprint ar Stli:22060068886 2022.
Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li AL.GO: Synthesizing algorithmic programs with generated oracle verifiers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://ppenreviee..et..t http://wwwnn//oorriiiooooorrot/m
Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, et al. Ansor: Generating (High- Performance] tensor programs for deep learning. In 14th USENIX symposium on operating systems design and implementation (OSDI 20), pp. 863-879, 2020.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint ar Xixo:2303.17566, 2023.
Yanqi Zhou, Sudip Roy, Amirali Abdolrashidi, Daniel Wong, Peter Ma, Qiumin Xu, Hanxiao Shen Liu, Phitchaya Phothilimtha, Shen Wang, Anna Goldie, et al. Transferable graph optimizers for ml compilers. Advances in Neural Information Processing Systems, 33:13844-138
2020.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. ar preprint 2tt.:22406.187777 2024.
13

---

Published as conference paper at COLM 2024
A Appendix
A.1 Runtime Variation at Different Runtime Scales
We profile each validated solution sample from $3 times and draw the relation of its runtime variation and mean runtime in Figure 6, the 17-10700K desktop (Table without any other mandatory processes running. Specifically, the X-axis presents the mean runtime and the Y-axis presents the Coefficient of Variation (CV). Each blue dot in Figure corresponds to data point of one profiled solution and the purple bars draw the mean variation for each runtime magnitude at the scale of 10x. Figure 6 shows that in practice smaller executions are more sensitive to system noise despite using a clean test bed. This motivates the necessity of using large computations for robust and meaningful performance benchmarking.
Coefficient of Variation (CV)
109
Mean CV
10-1
10-2
10-1
10-4
10 2
100
10a
Mean Runtime in Seconds
Figure 6: Distribution of runtime variation over the runtime scale.
A.2 Discussion of Performance Measurement and Metrics
In this section, we discuss the pros and cons of different performance measurement methods as well as the commonly used metric of relative speedup.
Measurements. There are three primary measurements the context of machine learning for code performance.
Physical runtime the most commonly used metric for performance comparison (Mendis at 2019; Zheng al. 2020; Zhou et al., 2020; Baghdadi et al., 2021; Mirhoseini et al., 2017). While physical runtime easy to measure and can reflect physical differences, it can incur high variation in repetitive executions, especially in noisy environments (e.g. shared cloud platforms that run various applications). As it difficult control the system noise when running performance benchmarking in the wild, it oftentimes challenges reproducibility Architecture simulator used by Shypula et al. (2023) to address the cross-ple cross-platform reproducibility issue. Architecture simulators such Gem5 (Binkert et al., 2011) use software to simulate the CPU behaviors when executing a program and therefore its reproducibility is guaranteed by software. However, also because it is software-based approach, simulating the program execution is oftentimes much slower than native execution and does not necessarily reflect the physical performance. Meanwhile, it is also not user-friendly to set up and run such simulators for interpreter-based programming languages such as the commonly used Python.
3. Hardware performance counter (Wikipedia, 2024) is used by EVALPERF (ours) to natively record the number of executed instructions between two program points. There are several benefits of using hard counters: (i) Usability: performance counters are easy to use that one can simply query system calls between two program points to obtain the profile; (ii) Efficiency: performance counters only incur light-weight overheads (Weaver, 2013b); (ūi) Reproducibility: the reported #instructions are highly reproducible that the observed variance of repeated executions at most few hundred instructions (e.g., caused by context switch), whereas most computations in our benchmark can use billions of instructions. Note that #instructions might vary across platforms due to different
14

---

Published as a conference paper at COLM 2024
hardware and software settings. However, because EVALPERF follows DPE to determine performance ranking by referencing representative solutions, such inconsistency becomes invisible when computing DPS different test beds, shown in Table 2. Last but not least, does not mean that hardware counter can always replace simulators given that one of its major use cases is to mimic the behaviors of certain CPUs without having them physically.
Relative speedup is also a commonly used compound metric that is agnostic to the measurements mentioned above. Although straightforward for direct comparisons between two programs, its applicability becomes less clear across a broad range of tasks. This is because the degree of speedup varies significantly across different tasks, with the average speedup often skewed by tasks that allow for substantial improvements, such as optimizing an iterative solution over recursive one for the n-th Fibonacci number calculation. This variability can lead to confusion, making it challenging to discern whether observed efficiency gains reflect holistic improvement or are merely the result of optimizations for narrow set of highly improvable tasks.
To overcome these limitations, Differential Performance Evaluation (DPE) defines a new metric, termed Differential Performance Score (DPS). Specifically, DPS for each task has consistent range from to 100% which stands for its empirical global performance position. This metric allows for an immediate understanding of solution's effectiveness and the distance to optimal performance by comparison with benchmark solutions, providing a more nuanced and scalable approach to evaluating program efficiency improvements.
A.3 Extended Results of Code Correctness and Efficiency
In this section, complement $4.1 to show more result details in the code efficiency evaluation of LLMs. Specifically, Table 3 lists the detailed correctness on three sets of tasks and the performance scores of both DPS and DPS performance scores are consolidated by calculating the average, maximum, and minimum scores within the initial 10 samples of each task and then we aggregate the task-wise score by computing the average. Notably, different than the main evaluations in $4.1, efficiency scores in this section are computed over all passing tasks within each model for global comparison, i.., the correctness can impact the set of tasks that are used to compute the efficiency scores.
EVALPERF. The sample-wise
Moreover, Figure offers a side-by-side comparison with Figure 7, illustrating DP'Soorm against DPS. Compared to DPS, DPSnorm on EVALPERF tasks tend to be lower, indicating that the distribution of correct LLM samples is skewed to slower implementation. This is expected, as crafting efficient code often presents greater challenge. Interestingly, while GPT-4 Turbo achieves the best DPS Figure 7, using DPSmom the best-performing model -ereeeCooor---.7--innee
becomes
This indicates that GPT-4 Turbo can generate code that is faster than the majority while DeepSeekCoderr-.. -.w.-idrrr----s-stuuuceee tends to generate code that is top of various efficiency levels.
Additionally, Figure 9 and Figure 10 show the correctness scores by computing pass@1 for the 121 EvalPerf tasks. Interestingly, we see trend of reversed scaling law on CODELLAMA model family that smaller CODE LLAMA models even achieve better pass rate. Despite this, other findings are overall consistent with that of Figure 7 and Figure
15

---

Published as conference paper at COLM 2024
Correctness (pass@1)
DPS
DPSnor
HE+
MBPP+
EVALPERP
Avg
Max
Min
Avg
Max
Min
CODELLAMA-7B
35.9
46.4
72.8
79.3
85.8
71.4
75.6
80.2
69.5
instruct
36.8
44.5
86.9
80.6
80.7
80.2
77.3
77.5
77.1
perf-instruct
33.0
44.1
90.5
80.1
80.2
79.8
77.3
77.3
77.0
perf-CoT
32.5
44.1
91.3
77.8
78.3
77.4
74.5
74.9
74.3
CODELLAMA-IBR base
38.7
50.4
76.7
75.6
80.6
67.7
74.5
77.2
70.0
instruct
40.2
50.6
88.9
80.3
80.7
78.7
75.8
76.3
75.2
perf-instruct
37.4
49.4
82.0
80.9
82.2
79.4
76.5
77.3
75.7
perf-CoT
36.4
49.1
85.9
78.7
80.9
75.5
75.5
76.8
73.5
30DELLAMA-34B
base
44.0
53.4
71.6
73.0
80.6
64.2
72.0
76.0
67.7
instruct
45.1
53.0
88.1
81.9
82.6
79.5
78.6
79.0
77.7
serf-instruct
43.3
50.7
87. 1
79.8
82.0
78.3
78.5
79.2
77.8
perf-CoT
41.7
49.7
85.1
81.8
83.8
80.2
79.6
81.2
78.7
CODELLAMA-70B
base
50.2
52.9
75.5
75.9
84.8
65.8
73.8
79.3
68.6
instruct
62.5
60.2
81.7
79.2
85.4
71.2
75.4
79.3
70.1
perf-instruct
59.3
59.0
77. 1
79.5
82.6
72.2
76.4
78.1
72.4
perf-CoT
60.2
58.6
78.8
76.9
82.2
71.3
74.6
78.6
70.8
1.er base
25.6
45,9
74.3
74.7
79.4
69.6
74.6
78.0
71.1
instruct
59.0
50.7
84.5
80.0
83.6
76.2
75.5
77.9
73.9
perf-instruct
59.4
49.5
84.2
82.0
84.5
79.2
76.7
77.9
75.6
perf-CoT
60.3
47.0
75.4
77.1
82.6
69.5
74.1
77.7
69.9
DeepScekCCdder-6.75 base
36.3
55.2
71.3
73.3
80.9
64.2
70.6
74.8
65.9
instruct
74.9
63.2
92.9
89.9
90.7
88.5
81.4
81.6
80.9
perf-instruct
74.4
60.7
87.2
86.3
87.9
84.2
79.3
80.2
78.0
perf-CoT
70.5
59.9
83.4
81.8
88.5
74.5
78.1
82.0
74.6
DeepScekCoder-33B base
42.3
60.6
80.5
79.2
84.2
72.1
74.2
77.6
70.2
instruct
74.6
67.5
92.2
85.4
87.4
82.3
78.6
79.7
76.7
perf-instruct
71.6
66.5
85.3
86.3
87.9
84.3
79.1
80.3
77.9
perf-CoT
73.2
64.4
84.2
82.4
86.0
76.2
78.6
81.8
75.0
StarCoder-3B
17.1
35.4
59.3
80.3
86.1
71.5
77.9
82.3
72.4
StarCoder-7B
base
24.0
40.5
63.3
83.2
87.0
77.1
78.3
80.7
74.8
StarCoder-15B
28.7
48.1
66.7
80.4
85.0
76.3
75.6
78.3
73.4
StarCoder2-3B
base
26.1
45.7
69.4
80.5
84.1
76.5
77.6
80.1
74.9
StarCoder2-7B
base
29.4
45.4
68.3
73.8
81.6
65.3
73.6
77.9
69.5
StarCoder2-15B
base
37.8
54.2
74.0
82.1
88.2
73.8
77.2
80.7
73.2
GPT-4 Turbo
instruct
81.7
73.0
97.1
88.5
89.8
86.3
76.3
78.2
74.6
perf-instruct
82.3
71.2
94.9
90.5
92.2
87.5
79.9
81.3
77.4
perf-CoT
83.5
71.2
96.7
91.5
94.6
90.3
79.7
82.6
77.7
Table 3: Overall correctness and efficiency using 10 samples with temperature-0.2. Scores within one point of the best score per model are highlighted.
16

---

Published as conference paper at COLM 2024
100
Cadeellumu
Stmkherr
Coder2
80
70
60
perf-CoT
base
perf-instruct
Best
80-
instruct
Figure 7: DPS on EVAL. u.s. pass@1 on HumanEval+ and MBPP+
GPT-4 Turbo
Codellama
CeepSeek
Star Coder
Star Code22
00
DP5
70
20
40
Pae001
perf-CoT
base
perf-instruct
Best of
80
instruct
Figure 8: DPS on EVALPERR u.s. pass@1 on HumanEval+ and MBPP+.
100
GPT-4 Turbe
Cade
DeepSeekCodd
StreCader
StaCnder2
DP5mm
138
348
70B
J.36
5.76
338
.88
198
7B
158
80
20
50
perf-CoT
base Best of All
80
perf-instruct instruct
90
Figure 9: DPS EVALPERF a.s. pass@1 on 121 EVALPERF tasks.
17

---

Published as conference paper at COLM 2024
GPT4 Turbq
CodeLlama
SrepSrekCnder
Star Coderr
Star Stde22
90
138
346
70B
1.38
28
898
.88
2B
158
28
158
80
60
70
perf-CoT perf-instruct
base
Best of
instruct
100
Figure 10: DPS on EVAL PERF u.s. pass@1 on 121 EVALPERF tasks.
18


Reviews:
[{"review_id": "t3W7NAHS03", "reviewer": "Reviewer_2P7H", "summary": "This paper introduces Differential Performance Evaluation (DPE), a framework for assessing the efficiency of code generated by LLMs. Unlike traditional benchmarks, DPE focuses on real-world efficiency-demanding tasks and provides a reliable metric for evaluation. By transforming coding tasks and creating the EVALPERF benchmark, the authors highlight insights into the impact of model sizes, instruction tuning, and prompting on code efficiency. The paper's evaluation underscores DPE's simplicity, speed, and cross-platform reliability, marking a significant advancement in code generation evaluation.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> C1: ... a more extensive comparison ... across different open-source LLMs.\n\nThanks for the suggestion! For a more extensive comparison, in our rebuttal, we include evaluations of some recent open-source code models:\n\n* **CodeQwen1.5-7B**: DPS: 72.1 (%) / DPS_norm: 65.6 (%) / Pass@1: 68.2 (%)\n* **CodeQwen1.5-7B-Chat**: \n    * Instruct: DPS: 78.9 (%) / DPS_norm: 71.8 (%) / Pass@1: 78.8 (%)\n    * Perf-Instruct: DPS: 79.3 (%) / DPS_norm: 73.0 (%) / Pass@1: 75.2 (%)\n    * Perf-Cot: DPS: 74.2 (%) / DPS_norm: 69.3 (%) / Pass@1: 72.2 (%)\n* **starcoder2-15b-instruct-v0.1**:  \n    * Instruct: DPS: 66.9 (%) / DPS_norm: 63.2 (%) / Pass@1: 71.9 (%)\n    * Perf-Instruct: DPS: 69.2 (%) / DPS_norm: 65.9 (%) / Pass@1: 72.5 (%)\n    * Perf-Cot: DPS: 67.5 (%) / DPS_norm: 62.8 (%) / Pass@1: 72.5 (%)\n\nMeanwhile, we include more experimental insights (e.g., pairwise comparison) for these models in C1 for Reviewer 6zda. In our next revision, we will try to include more if new code models are coming out.\n\n\n> C2: ... releasing the code used to reproduce the results would be beneficial.\n\nThanks for the suggestion! We are enthusiastic about open research and have already open-sourced and packaged the code and the dataset. However, for review anonymity, we cannot disclose the pointers now but will include them in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper considers an interesting and important problem.\n\n2. The paper proposes a novel framework for assessing the efficiency of code generated by LLMs. \n\n3. The paper presents EVALPERF, including 121 performance-exercising programming tasks and test inputs.", "reasons_to_reject": "1. The paper could potentially benefit from a more extensive comparison of EVALPERF Benchmarks across different open-source LLMs.\n\n2. To enhance the paper, releasing the code used to reproduce the results would be beneficial.", "questions_to_authors": "n/a", "extra_scores": {}}, {"review_id": "ZvCL06Dd6q", "reviewer": "Reviewer_6zda", "summary": "This paper attempts to create a benchmark for performance evaluation of generated programs by utilizing existing sets of programs and seeing how well the model performs with respect to these. Overall, I definitely liked the paper and think that it could potentially be a good contribution to the conference.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> C1: ... models would have to be compared across the same set of programs...\n\nThanks for the great suggestion! Indeed, it is important to compare model pairs over the same set of validated solutions. So we draw heatmaps in the following anonymous links:\n\n* Heatmap of $DPS$ comparison:  https://ibb.co/TBQvFV0\n* Heatmap of $DPS_{norm}$ comparison: https://ibb.co/5TkQbfY\n\nSpecifically, we sort the models by their global scores. Each block in the figure shows the performance score of the Y-axis model and its improvement against the X-axis model (if the difference > 1pp). Overall, most higher-ranking models still outperform lower-ranking ones on the same set of solvable tasks. We will discuss more in our revision. \n\n\n> C2: ..hard to follow due to their complexity and brief description..\n\nThanks for the suggestion! Indeed we should improve the methodology section by adding more explanation and examples. Regarding the detailed questions, \"base\" is a typo for \"bias\" and we should explain \"cumulative ratio\" in its definition. We will fix these in our revision!\n\n\n> C3: ..consider also generating programs that exercise the LLM-generated solutions\n\nThanks for the question! We did study a handful of cases where we saw both ground truth and slower implementation can lead to similar test generators. \n\nLet's case-study HumanEval/39 which computes the n-th prime Fibonacci number. The ground truth tests primality using the Miller-Rabin algorithm with O(k logN), while LLMs can use a vanilla method with O(N) or O(sqrt(N)). Using our prompts, all of the three lead to the same input sampler:\n\n```python\ndef perf_input_gen(scale: int):\n  return (scale,)\n```\n\nThis might be reasonable as LLMs mainly use the reference solutions to ground code semantic understanding and their complexity might not lead to significantly different generators.\n\n\n> C4: ..testing more complex programs that rely on external libraries\n\nThanks for the suggestions! We initially applied DPE on close-domain benchmarks as proof of concepts. We believe our technique can be applied to open-domain benchmarks such as DS-1000. We will include this in our future work.\n\n> Q1: ..results of your method vary depending on the architecture..\n\nYes, hardware profilers are architecture-dependent. We were saying our \"scoring mechanism\", i.e., the metric of \"differential performance score\" (in S2.5), is agnostic to measurements (physical time, #instructions, etc.). We will fix the confusion in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is tackling a problem that has not gotten enough attention in the literature.\n\nThe \"Synthesizing a Synthesizer\" framework is clever, I like it!\n\nOverall, the methodology and evaluation both are pretty solid. This was definitely the most well-done paper in my batch of reviews at COLM in this regard.", "reasons_to_reject": "I still like this paper and think it should be accepted, but I did have a few concerns.\n\n--- Major comments\n\nOne major concern that I had with the methodology, however, is that it seems that performance is only measured on programs where the answer is correct? If tht is the case, doesn't that introduce a major confounding factor that weaker models may generate answers to only the simpler programming problems, which are also easy to generate efficient solutions for? It seems that in order to rigorously compare two models with respect to their performance, the models would have to be compared across the same set of programs (so probably take the intersection of the sets of correct programs generated by the two models). I wonder if the authors \n\nAnother concern is that I found the methodology complex and pretty difficult to follow -- I hope that the clarity of the paper could be improved before it is finally published. Specifically, I found sections 2.4 and 2.5 pretty hard to follow due to their completxity and brief description. I was able to mostly understand them after a careful reading, but I would definitely encourage the authors to re-read these sections and try to make them clearer. Specific comments:\n- Adding an illustrative example (e.g. to Figure 3) of the time1d, rdiff, and splitters values would be helpful to better understand.\n- \"mean execution time\" is referenced in 2.4 before it is explained in 2.5, which made it confusing.\n- Is \"base\" a typo for \"bias\"?\n- Unless I missed it, it seems \"cumulative ratio\" is not defined anywhere? This is an important concept, and it was a bit hard for me to understand the methodology without knowing what this means.\n\n--- Minor comments\n\nBut one possible issue with it is that it tries to generate inputs that would test the efficiency of the ground-truth solution (since the ground-truth solution is provided in the prompt). Programs other than the ground truth solution may be slow in different ways, and it is not guaranteed that this method would find inputs that exercise these other programs. Did you consider also generating programs that exercise the LLM-generated solutions instead?\n\nOne limitation of this work is that it only uses HumanEval and MBPP as inputs, which use the Python standard library only. It is not clear if this framework would scale up to testing more complex programs that rely on external libraries (like numpy or or PyTorch). It'd be good to at least mention this fact in the paper.", "questions_to_authors": "--- Small typo fix suggestions\n\nwhich is pervasively available on most platforms\n-> which is available on most platforms\n\nthe new solution (in the green box) matchs\n-> the new solution (in the green box) matches\n\nimproving the the worst case scenario\n-> improving the worst case scenario\n\nThere is a place that says \"Compared to software profilers such as architecture simulators, hardware counters provides low overhead (Wikipedia, 2024)\", but I would suggest a reference other than Wikipedia, as Wikipedia is generally a secondary source rather than a primary authoritative source.\n\nThere is a place where you say \"It is worth noting that our scoring mechanism is agnostic to different efficiency measurements.\" in section 5. It is not clear what this means -- don't the results of your method vary depending on the architecture used for measurment (in section 4.2)? Maybe instead of \"agnostic\", \"robust\" would be a more appropriate word?", "extra_scores": {}}, {"review_id": "4RFAEfmSgF", "reviewer": "Reviewer_TCT1", "summary": "This paper proposes a new framework to evaluate the efficiency of code that large language models generate. I believe the authors' vision towards improving the efficiency of LLM-generated code is good and important: as LLMs generate more code, they should generate efficient code rather than slower code. In addition, the authors evaluate almost all the recent high-performing code LMs and give a very comprehensive evaluation of these models. \n\nOn the other hand, the domain of problems considered in this paper is relatively simple Python natural language to code programs. There are many more domains of programs that are optimized in real-world software engineering (such as code in different languages and performance-critical programs) that are not covered in this benchmark, which somewhat limits the scope of the work. \n\nThe paper, however, is relatively well-written and easy to follow. I would be more inclined to recommend acceptance of this work if it covered a wider variety of performance improvements and/or a more detailed analysis that revealed more insights of performance features of LLM generated code.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> C1: ..there is no analysis of the variance..\n\nPlease kindly note that Table 2 shows a negligible variation even when replicating the experiments over four different test beds.\n\nTo study statistical significance, we rerun the experiment 40 times using the \"instruct\" and \"perf-instruct\" versions of DeepSeekCoder-6.7B (2 settings X 40 repeats X 50 samples X 121 tasks). Next, we use the Wilcoxon signed-rank test to check if the difference in the normalized DPS between the two settings is statistically significant, showing a P-value of 0.000379, which is much smaller than 0.05 (the most common significance level).\n\n> C2: ..application of DPS to problems other than HumanEval and MBPP..\n\nPlease refer to C4 for reviewer 6zda.\n\n> C3: ...no qualitative analysis\n\nPlease note that we did show a few qualitative conclusions, e.g., instruction tuning implicitly improves efficient code generating. One reason for limited qualitative analysis is that code efficiency is not optimized in training, thus we see more randomness than scaling laws, calling for research to optimize this direction. Nonetheless, Q1 for reviewer Jb72 includes some qualitative case studies.\n\n> Q1: Why ... unify both of these under one benchmark?\n\nEvalPerf evaluates end-to-end efficiency where both aspects can equally exhibit efficiency advantages.\n\nWhy not put code-level efficiency under a program rewriting setting? Besides post-implementation optimizations, experienced developers can often directly write code of efficient paradigms. For example, in Python, it's faster to run built-in APIs like filter/map/reduce than vanilla loops. In C++, memory pre-allocation is a common efficiency tip where calling `vector::reserve` before `vector::push_back` reduces costly runtime allocations.\n\n> Q2: ..contributions of your benchmark from those of PIE\n\nPIE treats the efficiency problem in a **program repair** setting where a base solution is available, while DPE targets the **direct generation** of efficient code, as is exemplified in Q1.\n\nThe unique contributions of DPE are to systematically create efficiency benchmarks and reliably evaluate them. As shown in \"Limitation #1\" (Section 1), prior work does not manage to use compute-costly tests and DPE specifically curates such tasks/tests to exhibit code efficiency. As shown in \"Limitation #2\", traditional speedup has unbound scales and we develop DPS to fix this issue. These can also be applied to the PIE setting. We will discuss more in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper focuses on a new angle in the evaluation of LLMs for code, namely program efficiency. Given that there are very few works looking at program efficiency, this benchmark could provide a valuable signal as more of these techniques appear.\n- The authors do a comprehensive study of a variety of models consisting of different sizes, both open and closed source models, which provides information on the effectiveness of different classes of models in generating optimized code.\n- The DPS technique is relatively general and can be applied to all sorts of coding problems.", "reasons_to_reject": "- The size of EvalPerf is small, and there is no analysis of the variance of model scores on the benchmark. For example, it is hard to determine whether differences in the benchmark are statistically significant.\n- While the paper uses prompt-based methods to try and evaluate the abilities of current language models, a more thorough attempt at asking models to generate efficient code would have been helpful.\n- Scores on the benchmark are relatively high (e.g. in the 90s). The authors, however, note that DPS is a general technique that can be applied to other problems. However, the authors do not show any application of DPS to problems other than HumanEval and MBPP, which are relatively simple and already mostly saturated by the best models.\n- There is no qualitative analysis", "questions_to_authors": "- To me, there are two types of efficiency: algorithmic efficiency (e.g. O(n^2) to O(n) optimizations) and smaller, code-level optimizations (e.g. changing the order of for loops in matrix multiplication). The first is more focused from an algorithmic level, while the second is generally done after writing a piece of code and profiling it. Why do you decide to unify both of these under one benchmark?\n- Can you more clearly differentiate the contributions of your benchmark from those of PIE (Shypula et al. 2023)? What insights do you believe your benchmark reveals that theirs does not?", "extra_scores": {}}, {"review_id": "74ys6inqzP", "reviewer": "Reviewer_Jb72", "summary": "This work proposes a methodology for evaluating efficiency of a given LLM-based\ncode generation technique (model + prompting). It takes a selection of Python\ntasks from HumanEval and MBPP datasets along with a sample of reference\nsolutions from different LLMs, and builds a benchmark by clustering these\nsolutions on LLM-generated inputs. Crucially, the inputs to elicit different\ncomplexity clusters are generated by task-specific _programs_, themselves\nLLM-generated for every task. The performance score of a code-generation\ntechnique is then the average percentile of performance of its generated\nsolutions among the clustered reference solutions for each task. The authors\nevaluate five families of open-weight and proprietary LLMs and demonstrate that\nhigh code correctness does not necessarily correlate with performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: ..examples to analyze when more powerful models generate slower code\n\nThanks for the question! We case study some a few samples where DeepSeekCoder-6.7B-Instruct (perf-instruct) outperforms DeepSeekCoder-33B-Instruct (perf-instruct):\n\nMbpp/732: Replace certain characters to \":\". \n```\n# The 6.7B model uses the simpler and faster \"replace\" function:\ndef replace_specialchar(s):\n    return s.replace(' ', ':').replace(',', ':').replace('.', ':')\n\n# whereas the 33B model tend to use \"re.sub\" which is more powerful but slower as parsing regular expressions is costly:\nimport re\ndef replace_specialchar(s):\n    return re.sub(r'[ ,.]', ':', s)\n```\n\nHumanEval/24: Finding largest divisor.\n```\n# The 6.7B model smartly cut half of the computation in the beginning\ndef largest_divisor(n: int) -> int:\n  i = n // 2\n  while i > 0:\n    if n % i == 0:\n      return i\n    i -= 1\n\n# The 33B model\ndef largest_divisor(n: int) -> int:\n  for i in range(n-1, 0, -1):\n    if n % i == 0:\n      return i\n```\n\n> Q2: ..Do bigger values of scale lead to slower reference times across all tasks?\n\nThanks for the great question! We did see cases where input samplers produce \"smaller\"/\"faster\" inputs given a larger scale factor, e.g., sometimes GPT-4-turbo ignores the monotonicity requirement and simply generates random samplers. Notably, when making EvalPerf we add monotonicity assertions to falsify samplers that give \"faster\" inputs over larger scale factors.\n\nTo study the frequency of monotonicity violation, we ran input samplers by setting the scale factor from 2 to 10 without monotonicity checking. Among the 563 initial tasks, samplers from 486 tasks can generate valid test inputs and 327 (67%) of them generate inputs that always respect computational monotonicity. We will add this in our revision.\n\n> Q3: ..if we simply asked for test generators parameterized with a random seed?\n\nAsking LLMs to produce random test generators has limitations in both theory and practice. \n\nIn theory, when random sampling the tests, the scale/complexity of the test input is bounded by the range of randomness hardcoded by the LLM, e.g., the \"1000\" in `return [randint(0, 1000) for _ randint(0, 1000)]`. Such hardcoded values can still be too small to fully exercise the code solutions or too large to run. Meanwhile, for finding the largest computable input within a user-given time limit, searching inputs uniformly and randomly is less targeted and thus slower than searching inputs in monotonic scales.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "# Strengths\n\n* Important property to study and quantify.\n* The proposed approach is general enough to apply across programming languages\n  and architectures, provided a dataset of tasks to instrument.\n* Exhaustive evaluation on many different LLMs, albeit with mixed results.\n* Using LLMs to generate input-exercising programs instead of test inputs\n  themselves is clever and novel. Similar techniques are used in programming\n  competitions to create challenging inputs for contestants' solutions, but to\n  my knowledge have not been used for exercising AI-generated code.", "reasons_to_reject": "# Weaknesses and Questions\n\nMost importantly, the proposed benchmark does not show a scaling signal, which\nis often a red flag for its predictive power. When scaling the model (e.g.,\namong the tiers of CodeLlama) one would expect the efficiency of generated code\nto increase **if explicitly instructed to do so** (i.e., in \"perf-prompted\"\nsettings). However, it actually drops according to the paper's metric even as\ncode correctness increases - and that can be seen for almost all LLM families.\nThe paper provides no rationale or qualitative evaluation for these\nobservations, which in turn makes it difficult to attribute the misalignment to\nthe evaluated models or to the benchmark methodology itself.\n\nI would expect some qualitative evaluation of examples to analyze when more\npowerful models generate slower code. The simple tasks of HumanEval/MBPP make\nsuch analysis possible.\n\nThe input-generating technique is useful and novel, thus should be evaluated as\nan independent component.\n1. Does the `scale` parameter actually correlate with \"challenge\" of the\n   generated inputs? Do bigger values of `scale` lead to slower reference times\n   across all tasks?\n2. How do the methodology change if we simply asked for test generators\n   parameterized with a random `seed`? Might it still elicit a useful\n   distribution of challenging inputs?", "questions_to_authors": "# Minor\n\np2, line 1: \"tp perform\" => \"performs\"", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Differential Performance Evaluation (DPE), a novel framework for assessing the efficiency of code generated by large language models (LLMs). Unlike traditional benchmarks, DPE emphasizes real-world efficiency-demanding tasks, offering a reliable metric for evaluation. The authors transform coding tasks and develop the EVALPERF benchmark, comprising 121 performance-exercising programming tasks and test inputs. Their evaluation highlights the impact of model sizes, instruction tuning, and prompting on code efficiency, showcasing DPE's simplicity, speed, and cross-platform reliability.

This paper tackles a significant and underexplored problem in the evaluation of LLMs: the efficiency of generated code. The proposed Differential Performance Evaluation (DPE) framework and the accompanying EVALPERF benchmark are innovative contributions that offer valuable insights into code performance across various LLMs. The comprehensive evaluation, including recent high-performing models, underscores the potential of DPE to advance the field. However, several areas require improvement. Reviewer 2P7H and Reviewer TCT1 suggest expanding the benchmark to include a more diverse set of tasks and models, while Reviewer 6zda highlights the need for methodological clarity and rigorous comparison across models. Addressing these concerns, particularly by providing reproducibility through released code and enhancing methodological transparency, would strengthen the paper. Despite these limitations, the paper's contributions are significant, and with revisions, it holds promise for substantial impact in the field.
Decision: Accept