EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Authors: Dong HUANG
Guangtao Zeng
Jianbo Dai
Meng Luo
Han Weng
Yuhao QING
Heming Cui
Zhijiang Guo
Jie Zhang
Venue: ICML 2025 poster (2025)
Keywords: LLM-generated code
efficiency
Primary area: deep_learning->large_language_models

Abstract: As large language models (LLMs) play an increasingly important role in code generation, enhancing both correctness and efficiency has become crucial. Current methods primarily focus on correctness, often overlooking efficiency. To address this gap, we introduce SWIFTCODE to improve both aspects by fine-tuning LLMs on a high-quality dataset comprising correct and efficient code samples. Our methodology involves leveraging multiple LLMs to generate diverse candidate code solutions for various tasks across different programming languages. We then evaluate these solutions by directly measuring their execution time and memory usage through local execution. The code solution with the lowest execution time and memory consumption is selected as the final output for each task. Experimental results demonstrate significant improvements when fine-tuning with SWIFTCODE. For instance, Qwen2.5-Coder-7B-Instruct's pass@1 score increases from 44.8\% to 57.7\%, while the average execution time for correct tasks decreases by 48.4\%. SWIFTCODE offers a scalable and effective solution for advancing AI-driven code generation, benefiting both software development and computational problem-solving.

EFFICODER:
Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Dong Huang 12
Guangtao Zeng" "3 Jianbo Dai Meng Luo? Han Weng Yuhao Qing' Zhijiang Guo 16 Jie M. Zhang?
Heming Cui
Abstract
2024; Meta, 2024), including code-related applications like code completion (Chen et al., 2021; Austin et al., 2021), debugging (Haque et al., 2022: Chen et al., 2023), and trams- lation (Rozière et al., 2020; Ahmad et al., 2023). Before deploying LLMs into integrated development environments, it is crucial ensure that the generated code meets the re- quired efficacy standards. To address this, researchers have explored various datasets to fine-tune LLMs, thereby im- proving the efficacy of LLM-generated code (Ouyang et al., 2022; Wei et al., 2022). For example, Code Alpaca (Chaud- hary, 2023) utilized the Self-Instruct framework (Wang et al., 2023) to synthesize data, while WizardCoder (Luo al., 2024) employed the Evol-Instruct technique (Xu et al., 2024) to generate heuristic prompts for diverse solutions. Addi- tionally, OSS- (Wei et al., 2024b) created new cod- ing problems using open-source snippets with LLMs, and Octopack (Muennighoff et al., 2024) focused on curating high-quality Git commit messages that resemble natural language instructions. These efforts have led to increased correctness in LLM-generated code.
As Large Language Models (LLMs) play an im- portant role in code generation, enhancing both correctness and efficiency has become crucial. Current methods primarily focus on correctness, often overlooking efficiency. To address this gap, we introduce EFFICODER to improve both aspects by fine-tuning LLMs on a high-quality dataset EFFIINSTRUCT comprising correct and efficient code samples. Our method involves leveraging multiple LLMs to generate diverse candidate code solutions for various tasks across different pro- gramming languages. We then evaluate these so- lutions by measuring their execution time and memory usage through local execution. The code solution with the lowest execution time and mem- ory consumption is selected as the final output for each task. Experimental results demonstrate sig- nificant improvements when fine-tuning with Er- FIINSTRUCT. For instance, Qwen2.5-Coderr. Instruct's s pass @1 score increases from 44.8% to 57.7%, while the average execution time for cor- rect tasks decreases by 48.4%. EFFICODER offers scalable and effective solution for advancing AI- driven code generation, benefiting both software development and computational problem-solving. Dataset and Code are available at https:// github. com/ hwom//haaggllaa 999fffiiooder.
However, existing works primarily focus on enhancing the correctness of LLM-ggenrated code while neglecting to op- timize its efficiency. As result, the efficiency of such code often falls short compared to canonical solutions written by human developers. Recent studies (Shi et al., 2024; Niu et al., 2024; Du et al., 2024; Huang et al., 2024a; Liu et al., 2024b) also point out that LLM-generated code typically ex- hibits lower efficiency in execution time and memory usage. For instance, on the EffiBench benchmark (Huang al., 2024b), even the most advanced LLMs, such as GPT-4- Turbo, produced less efficient code, with average and worst- case execution times being 1.69 and 45.49 times longer than those of canonical solutions, respectively. Efficiency crucial because inefficient code consumes more computa- tional resources, leading to higher energy consumption and increased operational costs. This is particularly important in the context of sustainability, as the demand for computing power continues to grow, and reducing the environmen- tal impact of large-scale computations becomes a pressing concern. Furthermore, inefficient code may impractical for use in resource-constrained environments, such mo- bile devices embedded systems, where both energy and
1. Introduction
Large Language Models (LLMs) have recently made signif- icant strides across various tasks (OpenAI, 2023; Anthropic,
'Equal contribution 'University of Hong Kong National Uni- versity of Singapore 'Singapore University of Technology and Design *University of Edinburgh 'Bijjing University of Posts and Telecommunications University Cambridge "King's College London. Correspondence to: Zhijiang Guo ezg283 @ cam.ac.uks.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
processing power are limited. This underscores the urgent need to develop new methods that can enhance both the correctness and efficiency of LLM-generated code.
code repositories and the need to boost developer produc- tivity (Sun et al., 2024). Initial efforts focused on training models specifically for coding tasks, such as CodeT5 (Wang et al., 2021), AlphaCode (Li et al., 2022), CodeGen (Ni- jkamp et al., 2023), InCoder (Fried et al., 2023), Star- Coder (Li et al., 2023a), SantaCoder (Allal et al., 2023), and DeepSeek-Code (DeepSeekAl, 2023). Contrastingly, mod- els such as Codex (Chen et al., 2021) and CodeLlama (Roz- ière et al., 2023) represent a subsequent stride, being fine- tuned from foundation models (Brown et al., 2020; Touvron et al., 2023). These code L.MSs have been applied to vari- ous tasks, including code generation (Chen et al., 2021; Dai et al., 2024), program repair (Haque et al., 2022; Jiang et al., 2023), automated testing (Lemieux et al., 2023; Deng et al.,
In this paper, we introduce EFFICODER, aimed at fine- tuning LLMs to improve both code efficiency and correct- ness. We begin by aggregating source code from existing open-source datasets. This is followed by rigorous prepro- cessing and cleaning process, coupled with the generation of test cases for each task to evaluate code efficiency. We leverage multiple LLMs to generate diverse candidate code solutions for various tasks across different programming languages. We then evaluate these solutions by directly measuring their execution time and memory usage through local execution. Code solutions with the lowest execution time and memory consumption are selected as the final output. The resulting optimized code, along with its asso- ciated metadata, forms EFFIINSTRUC,, which serves as high-quality resource for training LLMs.
2023), code translation (Rozière et al., 2020; Ahmad et al..
2023), type prediction (Mir et al., 2022: Wei al., 2023), and code summarization (Hasan et al., 2021; Ahmed & De- vanbu, 2022). While LLMs have achieved impressive results in code generation tasks like HumanEval (Chen et al., 2021) and MHPP (Dai et al., 2024), their efficiency has received less attention. Recent studies (Shi et al., 2024; Huang et al., 2024b; Niu et al., 2024) have shown that LLM L.M- generated code exhibits lower efficiency in terms of execution time and memory usage compared to canonical solutions. These findings highlight the need for further research and devel- opment to improve the efficiency of LLM -eM-egeerrtted code. In this work, we propose the first fine-tuning method that significantly improves both the efficiency and correctness of code generated by various LLMs.
Extensive experiments demonstrate that tine-tuning LLMs with EFFINSTRUCCT improves both correctness and effi- ciency. For example, the fine-tuned Qwen2.5-Coder-77h Instruct (Hui et al., 2024) increases the pass@1 from 44.8% and 76.2% to 57.7% and 78.0% on EffiBench and Hu- manEvalPlus, while also reducing the average execution time from 0.31 seconds to 0.16 seconds - representing 48.4% reduction in execution time overhead on EffiBench. Compared to PIE (Shypula et al., 2024), which increases the @ from 12.2% to 19.5% on HumanEEalPllus, the @ass@I 1 of CodeLlama-7B (Rozière et al., 2023) fine-tuned with EFAINSTRUCT further increases to 31.1 %. In addi- tion, EFFICODER decreases the execution time by 46.2% while PIE decreases it by 23.1% %. We will fully open- sourcsource EFFICODER and the source code to facilitate research. To conclude, this paper makes the contributions:
2.2. Instruction Tuning for Code
Instruction tuning has proven effective in enhancing the usability and overall performance of LLMs across various language tasks (Ouyang al., 2022: Wei et al., 2022: Zhao et al., 2024). This approach has been extended the do- main of code generation. The core challenge is the ac- quisition of high-quality instructional data, which is often labor-intensive. To address this, recent research has focused on developing methods to generate synthetic instruction data. Studies have shown that -uabiit synthetic data alone can improve a model's coding and reasoning capabilities (Gunasekarr al., 2023; Li al., 2023b). One early effort was Self-Instruct (Wang et al., 2023), which utilized LLMs to generate synthetic instruction pairs using carefully crafted prompts. The same LLM was then instruction-tuna
We propose versatile framework for constructing code generation datasets with efficient solutions, adapt- able to various programming languages and sources.
We construct EFFIINSTRUCT, to the best of our knowl- edge, it is the first instruction-tumin dataset designed to improve the efficiency of LLM --geerrrrted code, facil- itating fine-tuning for more efficient code generation.
We introduce EFFICODER by fine-tuning various widely used LLMs using EFFIlNSTRUCT, demonstrat- ing improvements in both correctness and efficiency.
on this synthetic data. Code AI- paca (Chaudhary, 2023) applied the Self-Instruct approach with GPT models, tailoring it specifically for code gener- ation, editing, and optimization tasks. Building upon this, WizardCoder (Luo et al., 2024) adapted the Evol-Instruct technique (Xu et al., 2024) to the coding domain by design- ing heuristic prompts to create more complex and diverse synthetic data. OSS- Ins-ruutt (Wei et al., 2024b) took a
2. Related Works
2.1. LLMS for Code
The increasing popularity of LLMs for code generation has coincided with the growing availability of open-source
2

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Time: Select
Mining
Destription
Sampling
Respanse
Mem: 200Mb
</>
Task Solution
Response
Time: 0.3s
Solutions
Mem: 2000Mb
Code
Candidate Tasks
Task Cases
LLMs
Response
Overhead
Time: 0.06s 300Ml
Effi-Instruct
Corpus
(uummetedd)
Code Tasks
Candidate Solutions
Profiles
Figure 1. Overview of the construction pipeline for EFFIINSTRUCT: We begin by collecting the initial EHFINSTRUCT from different open-source datasets. Starting with the original code, require multiple LLMs generate candidate solutions, using test profile execution overhead, and use the most efficient solution generated by LLMs as the solution for cach task. We then have our final fine-tuning dataset, EFFLINSRRUC,, which consists optimized code and rich metadata, designed to train models generating efficient code.
def sing LeNumberr (num)
def sing LeNumber((uun) a b = num:
def sing LeNumber (num); return reduce(operatono num)
for 1 in range Len (nm))
if num, count/ (num [i])
return num [1]
aa = (~a & b & -C)
(a & <b & nc)
Slow Exicution
High
b od, bb
Slow Exeoution
Low Merrory
Fast Execution
Low Merrory
Memory
Figure 2. Examples of code with varying efficiency levels: The first solution has high memory usage and long execution time. The second reduces memory usage still has long execution time. third optimized low memory usage and fast execution.
different approach by leveraging LLMs to automatically generate new coding problems inspired by random code snippets from open-source repositories. In contrast, Oc- topack (Muennighoff et al., 2024) focused on collecting and filtering high-quality Git commit messages that resemble natural language instructions. While these existing methods primarily emphasize generating correct code, EFFICODER explores the use of fine-tuning to improve code efficiency. Our method is orthogonal to existing synthetic techniques, offering the potential for combination to further enhance the coding capabilities of LLMs.
efficiencies. The results, presented in Figure 3, reveal strong positive correlations between the efficiency the training data and the efficiency of the generated code. Specifically, the correlation for ET is 0.972, for MU it is 0.950, and for TMU it is 0.986. These high correlation coefficients indicate that as the efficiency of the training data increases, so does the efficiency of the generated code. This study demon- strates that training LLMs on efficient code significantly enhances the efficiency of the generated code. The strong correlations across all three metrics support the hypothesis that the efficiency of the training data critical factor in improving the performance of LLL.-generaa code. These findings inspire further exploration of specific techniques for optimizing training datasets to maximize code efficiency.
3. EFFICODER: Sone-Tuning For Efficiency
3.1. Preliminary Study
3.2. Dataset Construction
We begin by investigating how the efficiency of training data influences the efficiency of code generated by LLMs. Fol- lowing prior works (Huang et al., 2024b), we evaluate code efficiency using three metrics: Execution Time (ET), Max Memory Usage (MU), and Total Memory Usage (TMU). We hypothesize that training LLMs efficient code will lead to the generation of more efficient code. To test this hypothesis, we synthesized multiple training datasets with varying levels of efficiency and used them to train differ- ent LLMss The efficiency of the generated code was then measured in a controlled environment using ET, MU, and TMU, The training datasets included both efficient and in- efficient code samples to ensure comprehensive range of
Figure 1 illustrates an overview of the process for constructing the EFFIlNSTRUCT dataset for fine- tuning. The first step involves collecting candidate code generation tasks from nine open-source datasets available on the HuggingFace platform!. For each task, we aim to construct more efficient solution compared to the initial solutions provided by the open-source datasets. Our ap- proach shares similarities with existing works (Du et al., 2024; Huang et al., 2024a), where researchers execute LLM- generated code locally and analyze the execution time and 'ht 'nttps://buggingtan co/ /wcs//cc///////////// dat
Curation Process
3

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
ET: Corr 0,972
MU:: Corr 0,950
TMU: Corr 0.986
1.8
Line
Line
Fit Line
1.6
Em Em-Code
1.4
MU Err- Cee-Cooe
244
TMU Elf Cor--CClle
1.2
30
1.0
0.8
22
0.6
0.4
0.2
0.64
0.66
0.70 ET E/fiBench
0.72
0.74
80.0 82.5
90.0 92.5 95.0 97.5
60
80
MU Eriieecc
TMU ElliBench
Figure 3. Correlation of the efficiency of the automatically generated code and the LLM code train set.
memory usage. However, construction pipeline differs in that it utilizes multiple LLMs (e.g., DeepSeek-Cote Coder and GPT-4o) to generate multiple candidate code solutions for each task in our candidate task set. We then directly calculate the execution time and memory usage for each generated code solution by executing them in local envi- ronments. The code with the lowest execution time and memory usage is selected as the final code for each task. For example, as shown in Figure 2, we directly select the code on the right as the final code.
verse range of coding challenges, ensuring a comprehensive coverage of various programming concepts and problem- solving techniques. Python has the highest representation in EFFILNSTRUCT, with 33,489 tasks sourced from all nine datasets. This extensive collection of Python tasks allows for effective fine-tuning of LLMs to generate efficient and optimized Python code. C++ and Java also have significant contributions, with 11,547 and 14,726 tasks, respectively, These tasks are primarily sourced from CodeFeed, Glaive, Evol-Ins, Dolphin, and Oss-Ins datasets, providing a robust foundation for fine-tuning LLMs in these popular program- ming languages. Rust and Go, although having relatively fewer tasks compared Python, C++, and Java, still have substantial presence in EFFIINSTRUCT. With 4,270 Rust tasks and 1,678 Go tasks, the dataset enables fine-tuning of L.MMs to generate efficient code in these modern and rapidly growing programming languages.
Data Sources We collect the candidate tasks from the open-source code LLM training sets, which in- clude SelfCodeAlign (SelfCodeAliin; Wei et al. 2024a), FiFFeecdbccc--llleerd---sttteeee
(CodeFeed; MAP 2023), Tested 1asteed--Skk-yython-AAlppp (Alpaca; Vezora 2023), Glaive- Code-Assistant (Glaive; Computer 2023), Magicoder-Evol- Instruct-110K (Evol-Ins; UIUC 2023a), Dolphin-Coo (Dolphin; Computations 2023), Magicoder- -nst-stue-- 75K IOss-Ins; UICC 2023b), SEff-OSS--nnsuue Seee--CCC-eeexeeee -xece Filter-50K (Self-Oss; BigCode 2023), and Apps (Hendrycks al., 2021). To collect the candidate tasks, all Python, C++, Java, Rust, and Go functions are extracted from the afore- mentioned open-source datasets. Following the filtering instructions of SelfCodeAlign (Wei et al., 2024a), a series of filtering rules are applied to ensure the code quality of the candidate tasks. After applying the filtering process, a total of 65k tasks were collected from an initial pool of about 790k candidate tasks?
Figure 4 illustrates the efficiency distribution of the dataset for three key metrics: execution time, memory usage, and max memory peak, which compares the distribution of these metries for both inefficient (canonical solutions provided by the nine datasets) and efficient solutions in the EFFIIN- STRUCT. For execution time, the inefficient solutions have a mean value of 1. 1.14s, while the efficient solutions have a significantly lower mean of 0.31s, which indicates that the optimization process has successfully reduced the execu- tion time of the code, resulting in more efficient solutions. Similarly, the memory usage and memory peak also show notable difference between inefficient and efficient solutions. For example, inefficient solutions have mean memory usage of 26.50 MBs, while efficient solutions have a much lower mean of 6.03 MBs.
3.3. Dataset Statistics
As shown in Table 1, coding problems in ESFINNTTUUCT have been collected from nine datasets, resulting in total of 65,710 tasks across five programming languages: Python, C++ Java, Rust, and Go. The dataset encompasses a di-
The efficiency distribution visualization highlights the ef- fectiveness of the optimization process creating more efficient solutions across all three metries. By carefully curating tasks through the multi-step cleaning process and applying SOAP optimization, we have created dataset that is valuable for training models to generate efficient
Analysis shows exact duplicates between training and eval- uation sets, with only 0.20% of evaluation samples having minimal vocabulary overlap (0%1.%).

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Table I. Distribution of tasks in the constructed EFRINSTRUCT for different programming languages.
Dataset
APPS
Alpaca
CodeFeee
Glaive
Evol-Ins
Dolphin
Oss-Ins
Self-Oss
SelfCode
Total
Python
1001
2920
1387
1250
1958
24038
33489
CPP
3
1257
2675
3439
1186
2985
2
11547
Java
1
2082
3278
4692
1746
2927
14726
Rust
26
187
467
500
3090
4270
Go
47
277
549
28
1678
Execution Time
Memory Usage
Max Meemrry Peak
Ineffiient Maam:
0.375
inoficcentt 26.09M8
Efficient Nean: 9.27s
kientt Meean: 5:33MB
Mean: 18M8
0.150
0.150
2.5
0.325
2.0
0.300
1.5
0.375
1.--
0.050
0.250
0.000
0.000
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
30
40
50
20
30
Execution Time
Memory Usage (MB
Max Memory Peak (MBB
Figure 4. Efficiency distribution of the Python subset collected from Hugging Face. The figure shows the distribution of execution time, memory usage, and max memory peak both inefficient (task-prove) pook--po solution) and efficient solutions EFFINSTRUCT. The inefficient solutions have higher overheads for all three metries than the efficient ones.
code. EFFIINSTRUCT provides diverse range of optimized coding problems, enabling researchers and practitioners to advance the field of code optimization using LLMs.
4.1. Evaluation of Python Code
To comprehensively demonstrate the efficiency and correct- ness of automatically generated code by LLMs with the SFT of EFFIINSTRUCT, we first provide the evaluation of the LLMss in generating Python code, where LLMs are asked to create Python code based on natural language or function signatures with docstrings.
4. Experiment
Datasets and Models We evaluate the efficiency and correctness of LLM-gennrated code on five code gener- ation benchmarks, i.e., EffiBench (Huang et al., 2024b), EvalPlus (HumanEvallllus and MBPPPlus) (Liu et al., 2024a), DS-1000 (Lai et al., 2023), EvoEval (Xia et al., 2024), and HumanEval-X (Zheng et al., 2023). We finetune eight open-source LLMs with EFFINNSTRUCT, including CodeLlama-77-bff, DeepSeek-Coders 6. base and instruct model (DeepSeekAI, 2023), Owen2.5-Coder-11 base and instruct model (Hui et al., 2024), and Qwen2.5-Coder (I.SB, 3B, and 14B).
EffiBench is benchmark used to measure the efficiency and correctness of LLM--gnerated code in LeetCode tasks. To ensure that the efficiency of LLM- code can be measured, the authors construct 100 tests for each task to provide enough testing time. As shown in Table 2, we can observe that for all LLMs, the efficiency of the LLM- generated code has been improved after fine-tuning with EFFLINSTRUCT. For example, the average execution time (ET) for the correct code generated by Qwen2.5-Coder-P7 Instruct and its EFAINNTRUCC fine-tuned version decreases from 0.31 (s) to 0.16 (s), reduction of 48.4%. Similarly, the total memory usage (TMU) of LLM-generated code also shows significant decreases. For instance, the TMU of DeepSeek- Cooer-6.7B-Innsttee (Mb*s) to 9.48 (Mb*s), a larger reduction than the decrease in average execution time for the correct code generated by the same model, which only reduces by 35.3% from
Fine-tuning Setup We use Llama-factory (Zheng et al., 2024) to fine-tune LLMs with fully supervised fine-tuning with the same setup and train the models using EFFIIN- STRUCT. The maximum sequence length set to 2048 tokens. We use batch size of 128 and set the learning rate 5e-6 with cosine learning rate scheduler and warmup ratio of 0.03. We fine-tune all LLMs for 4 epochs under the 6f16 data type.
decreases from 30.05
5

---

EFFICODER:
Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Table 2. Code efficiency and pass@1 of L. trained with EFRINSTRUCT on EftiBench using greedy decoding. The percentage in the
brackets indicates the extent of the reduction for each respective item. Overlap means the percentage of correct tasks addressed by both EFFUNNTRUUCT finetuned LLM and original LLM in total tasks of the dataset.
Model
ET (6)
NET 4
MU (Mb) J
NMU 4
TMU (Mb*s).. v
NTMU
Overlap (%)
Pass @ (%)
CodeL
0.24
1.88
141.91
4.78
149.23
73.82
8.2
15.0
+ EFFIINSTRUCT
0.20 (16.7%)
1.27 (14,2%)
80.86 (43.0%)
2.26 (52.7%)
87.09 (41.6%)
42.66 (42.2%%
8.2
17.6
0.37
2.62
36.94
1.04
17.26
2.74
47.1
54.4
+ EFPINSTRUCT
0.21 (43 2%)
1.42 (45.8%)
37.11 (-0.5%)
1.04 (0.05)
11.97 (30.6%)
2:10 (23345)
47.1
59.3
deepszek coder 6.7b instruct
0.34
2.56
47.26
1.45
30.05
9.97
36.0
44.4
+ FeFItNSTRUCT
0.22 (35 (35.3%)
1.71 (83.2%)
36.31 (23,255)
1.00 (31.0%)
9.48 (68.5%)
2 (78.8%)
36.0
51.7
Qwen2.5-Coder:-BBInstte
0.31
2,35
31,66
1,00
11.00
2,15
37.2
44,8
EFRINSTRUCT
(48..45))
1.12 (52.3%)
31.67 (-0.0))
1.00 (0.0%)
8.28 (24.7%)
1.18 (45.1%)
37.2
$7.7
Qwen2.5-Coder-1II1
0.40
2.95
35.34
1.03
15.68
3.51
27.1
39.6
EMINNTTRCC
0.22 (45.0%)
1.60 (45.8%%
34,58 (2.2%)
1.00 (2.9%)
9.09 (42.0%)
1.98 (33
27.1
41.7
Qwen2.5
0.43
2.73
48.73
1.00
33.88
2.59
16.9
31.2
EFFENSTRUCT
0.23 (46.55%)
1.60 (4%)
49. 18 (-0.9%)
1.00 (0.0%)
18.31 (46.0%)
1.98 (23.6%)
16.9
34.2
0.26
1.81
38.88
1,01
18.63
3.01
41.4
50.1
+ EFFINNSTRUCT
0.17 (34.6%)
1.23 (32.0%)
38.61 (0.7%)
1.00 (1.0%)
10.82 (41.9%)
1,32 (56.
41.4
57.3
Qwen2.5-Coder-145
0.36
2.73
32.41
1.00
12.59
2.57
50.1
57.5
+ EMTNNSTRUCT
0.15 (58 (58)
1.14 (58.2%)
32.41 (0.0%)
1.00 (0.0%)
6.80 (46.0%)
1.23 (52. 19)
50.1
63.6
Table 3. Code efficiency and pass@1 of LLMs trained with EFFIINSTRUCT on HumanEvalPlus and MBPPPlus.
Model
ET 6)
NET 4
MU (Mb)
NMU
TMU (Mb*s) 4.
NTMU 4
Overlap (%
P 1 (%)
HunanEvalPlus
codeeck--coder-777 coder-6.7b 2ee66.7b-bba busb
0.43
1.04
67.45
1.00
28.23
1,02
7.3
7.3
EMPINSTRUTT
(2.3%)
(3.8%)
67.37 (0.1%)
1.00 (0.0%)
(7.20
1.02 (0.0%)
64.6
deepseck -ooder-6.dd---oo 7ib-instuctt
0,54
2.27
61.64
0.98
20.18
2.30
42.1
47.6
EETINSTRUCT
0.37 (31.5%)
1.45 (36.19))
61.58
0.98 (0.0%)
16.48 (18.3%)
1.78 (22
42.1
72.6
Owen2,5-Codee-78 CB
0,35
1.23
61.77
0.98
15:25
1.39
36.6
40.2
EERIINTTUCC
(17.1%))
0.96 (22.0%)
61.70
0.98 (0.0%)
12.18 (20.1%)
0.96 (30.9%)
36.6
78.7
0.52
2.05
63.38
0.99
20.17
1.96
67.7
76.2
EFFIINSTRUCT
0.32 (38.5%)
1.08 (47.3%)
63.35 (0.0%)
0.99 (0.0%)
15.15 (24.9%)
1.16 (40.8%)
67.7
78.0
MBPPPlus
1.64
58.90
17.27
1.62
55.3
63.2
EPFINSTRUCT
0.31 (36.7%)
0.97 (40.9%)
58.99 (-0.2%)
1.00 (0.09)
10.27 (40.5%)
0.96 (40.7%)
55.3
65.9
59.03
14.39
59.0
65.3
EIMINSTRUCT
0.31 (27.9%)
1.01 (38 8%)
58.97 (0.)%)
1.00 (0.0%)
10.35 128.1%)
1.02 (38.2%)
59.0
67.5
Qwea2,5- 5-Coder-BB
1.70
58.89
17.00
59.5
60.1
EMVINSTRUCT
0.31 (35.4%)
0.96 (43 5%)
$8.98 (-0.2%)
0.99 (0.0%)
10.33 (39.2%)
0.94 (47.5%)
59.5
63.2
Qwea2.5 Son22.5-Con 7B-nstet7
64.90
23.99
63.2
68.0
BIVINSTRUCT
0.30 (34,8%)
0.96 (42,9%%
68,31 (-5.3%)
1.00 (0.0%)
16.91 (32.445
0.95 (42.8%%
63.2
70.6
0.34 (s) to 0.22 (s). This indicates that during code gen- eration, both the execution time and memory usage of the fine-tuned- LooooLLLL-ggeeeeeee code have been improved com- pared the code generated by the original models. Further- more, the memory usage/peak (MU) of PrepSeek-Coder Instrutt generated code decreases from 47.26 (Mb) to 36.31 (Mb), a reduction of 23.2 23.2%, ensuring that the LLM- generated code can be deployed in memory-constraing scenarios such as embedded systems or edge devices.
higher correctness after being fine-tuned with Take Hu- manEvalPPus as an example, the average execution time (ET) for correct code generated by (roon..--odde-7B--nnsee and its fine-tuned version decreases from 0.52 (s) 0.32 (s), reduction of 38.5%. These improvements demon- strate the effectiveness of in optimizing the efficiency of LLM-generated code. Moreover, the pass I of Qwen2.55 Co increases from 40.2% to 78.7%, an improvement of 38.5% after fine-tuning with This indicates that the fine-tuned models not only generate more efficient code but also produce correct code more frequently. Similar the results in HumanEvalPluss, the efficiency and correctness of fine-tuned L.LMs also improve in the MBPPPlus dataset. For instance, the average execution time for correct code generated by deepseek-coder-6.7 7-bbbase decreases by 36.7% %, and the total memory usage (TMU) decreases by 40.5% after fine-tuning.
Interestingly, we observe that compared to the MU, ET is more widely optimized across all models. This suggests that EFFIINSTRUCT fine-tuning has a more significant impact reducing the execution time of the generated code than on reducing its memory footprint. Nevertheless, the improve- ments in execution time and memory usage demonstrate the effectiveness of ESTINNTRUUC in enhancing the efficiency of L.L. -enerated code.
Evolval includes 828 programming problems created by prompting GPT-4 to evolve original HumanEval tasks across
Human EvaPPus and MBPPPlus As shown in Table 3, we observe that almost all LLMs achieve better efficiency and
6

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Table Code efficiency and pass @1 of Qwwem25--Codeer77BBlnnnee and deepseek--ooderr677---iiii fine-tuned using SFT with the
INFTRNTTUUC for the EvoEval dataset.
Model
ET
NET
MU (Mb)
NMU
TMU (Mb%))
NTMU
Overlap (%
Pase81 (%)*
EvoEval uso
Qwea2.5 S--Coode-----ntteee
1.80
59.11
1.00
33.0
43.0
EMTINSTRUCT
0.20 (42.9%)
0.98 (45 (6%)
59.08 (0.1%)
1.00 (0.0%8
6.55 (41.6%)
0.99 (43.18)
33.0
53.0
deepsock-coder-.77ee 7.bisttttt
74,88
31,51
44.0
54.0
EITINSTRUCT
0.24 (41.5%%
0.99 (45.9%%
74.77 (0.1%)
1.00 (0.0%)
26.50 (15.9%)
1,00 (42.5%)
44.0
55.0
EvoEvvll
Qwen2.5-Coder -B-rstrrett
0.40
1.49
70.46
1.00
29.15
1.46
48.0
55.0
EFRINSTRUCT
0.33 (17.5%%)
1.11 (25.59))
(40.0%)
1.00 (0.09)
28.30 (2.9%))
1.17
48.0
0.45
1.97
59.06
0.99
17.01
2.00
50.0
56.0
EFFIINSTUUCT
(33.33)))
1.32 (33.09)
58.93 (0.2%)
0.99 (0.05)
(34.29)
1.31 (34,5%)
50.0
EvoFvall creetivvee
Qwen2. Twea2.5-Codett 7-Cosse-7--Inntteett
0,51
2.16
62.71
1.00
24:21
2.39
32.0
44.0
BRWINSTRUCT
0.37 (27.55)
1.45 (32,95%)
62.69 (0.0%)
1.00 (0.0%)
21,10 (12.8%)
1.82 (23.8%)
32.0
44.0
deepseek coder- 6.7h insttinsttcct
61,88
15.91
27.0
EMVINSTRUCT
0.26 196)
0.99 142)
61,75 (0.25)
1.00 (0.0%%
10.86 (31.7%)
0.98 (39.5%)
27.0
41.0
Table Code efficiency and pass @1 of Qwen2.5- Codee-oo-ee--eeeeee- and (etpssee---coer--.7----i--
tunettune using SFT with the
INSTUNTT for the DS- 1000 dataset.
Model
ET (s)
NET 1
MU
NMU
TMU (MB*s) J
NTMU
Overlap (%)
Pas@1 (%)
Qwen2 C-Coeer-7--oosie
1,2668
1,0102
447,7101
1,01488
6,831 366
1,0175
9.30
17,00
EFFIINSTRUCT
1.2593 (0.8%)
1.0041 (1.0%)
447.07 (0.1%)
1.0134 (0.0%)
6.7694 (0.9%)
10081
9.30
boneeel--onder-67--insttt EFFIINTTRUCT
1,3154
1.0536
450.2792
1.0206
7,204722
1,0731
29.10
37.50
1,2587 (4.5%)
1.0082 (3.8%)
441,7553 (1.9%)
1,0013 (2.0%)
6.8022 (5.6%)
10132 (5.6%)
29.10
37,50
5 semantic- and 2 semantic- preservvig benchmarks, each of which has 100 problems. We conduct experiments on the _sol_ Subtle, and Creative benchmarks to eval- uate the performance of Qwen2.5-Coder-777 -nnsstrut and deepseek-coders cooder-6.7-inne 6.7b-insstuctt fine-tuned with.. As shown in Table 4, both models demonstrate significant improvements in code efficiency after fine-tuning with For the Tool_ Use benchmark, the average execution time (ET) for correct code generated by Qwen2.5-Coder-777 In-Intruct decreases by 42.9%, and the total memory usage (TMU) decreases by 41.6% after fine-tuning. Similarly, deepseek-coder-6. 7b- instruct achieves a 41.5% reduction in ET and a 15.9% reduction in TMU, In the Subtle benchmark, after fine- tuning, Qwen2.5- --c.5----ddrrBBB... ContructtInntrucct and deepseek-coder- 6. 6.7b-instruct exhibit 17.5% and 33.3% reductions in ET, respectively. after fine-tuning. The normalized total mem- ory usage (NTMU) also decreases by 19.9% and 34.5% for the two models. Moreover, the pass @1 rate improves increasing
4.2. Data Science Programming
DS-1000 is data science benchmark consisting of 1000 realistic challenges across 7 popular Python data science libraries. We evaluate the efficiency and pass 1 of LLM- generated code for the 1000 tasks using Qwen2.5-Coder 7B Instruct and deepseek original form and fine-tuned with As shown in Table 5, fine-tuning with leads to modest improvements in code ef- ficiency for both models. For the average execution time (ET) decreases by 0.8%, and the total memory usage (TMU) decreases by 0.9% after fine-tuning, while the pass@I I rate improves significantly from 17.00% to 35. 70%. For seepseek-coderr pooder-6.7b-ineee ib-instruc, fine-tuning results in a 4.5% reduction in ET and a 5.6% reduction in TMU, with the memory usage (MU) and nor- malized memory usage (NMU) also decreasing by 1.9% and 2.0%, respectively. although the pass @ rate remains the same at 37.50%. While the improvements in code efficiency for the DS01000 benchmark are less pronounced compared to other benchmarks, the results still demonstrate that fine- tuning with can enhance the efficiency of ALL--generatr code for data science tasks.
7bbiinstru,, both their
significantly, with Srven..5-Coder-7B-BB from 55.0% to 72.0% and deepseck-coder--.. -ot--eck--ooeer-.77----n------ in- creasing from 56.0% to 69.0%. For the Creative bench- mark, Qwen2.5-coder-75 --n-stttttttt instruct achieve 27.5% and 38.1% 1% reductions in ET, re- spectively, after fine-tuning. The NTMU also decreases by 23.8% and 39.5% for the two models. The pass@ rate at 44.0%
instructt and decpseek-coderr-..7 7b-
4.3. Difffernt Programming Language
remains the same for C-op:5---oor--B-BBet but improves from 31.0% to 41. .00 for deepseek-coder- instruct.
In addition evaluating the efficiency and pass@I of LLM- generated code for Python tasks, we also conduct exper- iments on the HumanEval-X dataset, where we focus on the C++ and Java subsets, to measure the performance of
7

---

EFFICODER:
Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Table Code efficiency and pass @1 of Srwenn25--oodr-7B---mmae and deepseck--coder-677---iiii fine-tuned using SFT with the
EFFUINSTRUCT for the HumanEval-X X (CPP and Java) dataset.
Model
ET (8)
NET 4
MU (Mb)
NMU4
TMU (MD*s)
NTMU
Overlap (%)
Pas 1 (%)
CPP
Qwen2 Owen25-Code-7BB Code-- EraUSSTTUUCT
0,0015
1,3973
1,5223
1,9489
0,000013
5,1950224
4.27
7.32
0.0009 (37.158)
10138 (27.4%)
0,8729 (42,615
1.0585 (15.7%)
0,000006 (53.8% (56)
2.784936 (46.45)
4.27
48.17
doepsesk-oode--..0 7.7-insrrttt
0.0015
1,2417
1,2991
1.0731
0,000000
2,0757900
4.27
BrnnssSTRUCT
0.0008 (45.2%)
0,5312 (57.256)
0.6496 (50.055)
0.4289 (60.0%)
0.000006 (40.0%
0,543047 (73.9%5
4.27
40.24
Qwen2.5 5-a.5-5--dee--- Ii-tustrett
EPHINSTRUCT
0,0062
0.2037
8.5926
0.1918
0.002996
0,1859
deepseck-coder-... 6.77--666bbsseee EPPISSTRUCT
0.00776
0,1921
8.1062
0,1789
0.002731
0.180034
6.71
14.63
0.0048 (36.598)
0,1231 (35.9%%
4.01/2 (50.516)
0.0908 (19.25))
0,001900 (20.458)
0.118972 (33.9%%)
6.71
57,93
Table 7. Efficiency comparison of different methods the HumanEvalPlus dataset. We use the tne-ttned CodeLlama-7b by PIE and Mercury as the baselines to measure the improvement of EFFLNNTRRUCT fine-tuned version.
Method
ET
NET
MU
NMU
TMU
NTMU
overlapped
pass I
Jama-7bhh
0.39
1,94
61.68
1.00
12.78
1.83
9.8
12.2
PIE (AIl)
0.30 (23.1 1%)
1.47 (24,2%)
61.39 (0.5%)
1.00 (0.0%)
11.28 (11.7%)
1.68 (8. (8.2%)
9.8
19.5
EPFLINSTRUCT
0.21 (46.2%)
1.03 (46.9%)
61.33 (0.6%)
1.00 (0.0%)
7.17 (43.9%)
1.04 (43.2%)
9.8
31.1
CodeLlama-7b-hl
0.39
1.94
61.69
1.00
12.78
1.83
4.3
12.2
Mercury
0.31 (20.5%)
1.51 (22.2%)
61.94 (-0.4%)
1.00 (0.0%)
10.24 (19.9%)
1.47 (19.7%)
4.3
9.1
EFFILNSTRUCT 0.21 (46.2%)
1.01 (47.9%)
61.73 (-0.1%)
1.00 (0.0%)
6.95 (45.6%)
0.98 (46.4%)
4.3
31.1
fine-tuned LLMs in a different-language setting. As shown in Table 6, both 1emn2..-Coder-7BB Instruct and deepseck- coder-6. 7b-iinsttttct demonstrate significant improvements in code efficiency and pass@1 rates after fine-tuning with for C++ tasks. Swee2.5-Coder. Bo-der-BB-Ing 7 Instructt achieves a 37.1% reduction in average execution time (ET), a 42.6% reduc- tion in memory usage (MU), and a 53.8% reduction in total memory usage (TMU), with the pass@1 rate improving from 7.32% to 48.17% Similarly, deepseck-coder-6.777 instruct exhibits a 45.2% reduction in ET, a 50.0% reduction in MU, and 40.0% reduction in TMU, with the pass @1 rate increasing from 14.63% to 40.24%. For Java tasks, achieves 36.5% reduction in ET, 50.5% reduction in MU, and 30.4% reduction in TMU after fine-tuning, with the pass 1 rate improv- ing from 14.63% to 57.93% matching the performance These re-
PIE required 0.30 (s) on average address each task, which is 23.1% % reduction in average execution time compared to the original the fine-tuned CodeLlama-7b-hf reduces the average execu- tion time by 46.2%, requiring only 0.21 (s) on average to address each correct task. Moreover, the fine-tuned model achieves a 46.9% reduction in NET, a 43.9% reduction in TMU, and a 43.2% reduction in NTMU compared to PIE. The pass @ also improves from 19.5% for PIE to 31. 1% for the fine-tuned model. Similarly, when compared to Mer- cury, the fine-tuned CodeLlama-7bb demonstrates a 46.2% reduction in average execution time, a 47.9% reduction in NET, a 45.6% reduction in TMU, and a 46.4% reduction in NTMU. The pass@1 rate improves significantly from 9. % for Mercury 31.1 %% for the fine-tuned model, with both methods having an overlapped percentage of 4.3%. The substantial improvements in efficiency and correctness achieved by the fine-tuned model demonstrate the effective- ness of this approach in optimizing L.LM-generated code for practical applications.
generated code. However,
reepseck-coder-667--innsee
of the fine-tuned Oween25-Cooder-7--nniiit sults demonstrate that fine-tuning with can significantly enhance the efficiency and correctness of L.MM-generated code across multiple programming languages.
EFFICDDER's exceptional performance can be attributed to several key factors. First, EFFICODER was designed with multilingual programming support in mind, covering diverse range of programming languages in its training process. This comprehensive language coverage enables the model to perform consistently well across various language- specific test sets, demonstrating its versatility and broad applicability in real-world programming scenarios. Second, EFFICODER'S superior efficiency stems from our meticu- lous data processing method. During the data preparation
4.4. Comparison with Baselines
To further demonstrate the efficiency of the code gener- ated by fine-tuned LLMs, we compare the performance of CodeLlama-7b-bh fine-tuned with two baselines: PIE (Shy- pula et al., 2024) and Mercury (Du et al., 2024). Similar with EFFICODER, PIE and Mercury are fine-tuned to improve the efficiency of LLM- code. The evaluation results the HumanEvalPlon dataset are presented in Table 7. We can observe that for the correct tasks addressed by both PIE and
8

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
phase, we implemented rigorous filtering operations that specifically retained only tasks demonstrating measurable improvements in code efficiency (as illustrated in our moti- vation figure). This efficiency-focuee focused data curation strategy directly aligns with our core objective of optimizing com- putational performance. Third, the scale of our training dataset significantly outperforms previous approaches, with over 70,000 training examples compared to Mercury's mere 1,800 samples. This substantial difference in training data volume allows our fine-tuned language models to develop more robust patterns for generating efficient code across diverse contexts. The combination of these factors, multi- lingual support, efficiency-orieeee oriented data filtering. and large- scale training data, collectively contribute to EFFICODER ability to consistently outperform bascline methods in both code correctness and execution efficiency metrics.
and models fosters transparency and accelerates re- search into sustainable Al systems, encouraging further innovations in code optimization.
Ethical Considerations:
The dataset is aggregated from pub- licly available open-source repositories on Hugging Face, ensuring compliance with licensing terms. How- ever, future work should continue to prioritize respon- sible data curation practices.
Data Provenance:
Bias and Generalization: While our framework sup- ports multilingual adaptability, biases in the source code (e.g.. language-specific optimizations cultural coding norms) may inadvertently propagate. Mitigat- ing such biases requires careful dataset design and validation.
5. Conclusion
Developer Dependency: Widespread adoption of op- timized LLM- --M-eeeerated code could influence coding practices. Ensuring human developers retain critical problem-solving skills remains important.
In this paper, our research addresses a critical gap in the effi- ciency of code generated by LLMs by introducing the EFFI- INSTrucT dataset, designed to enhance both the correctness and execution efficiency of LLM-generated code via fine- tuning. Through meticulous aggregation, preprocessing, and iterative optimization, we provide robust resource that significantly boosts the performance of open- LLMs like DeepSeek-Cooe Coder and Qwen. Our experiments reveal substantial improvements, with notable increases in pass rates and decreases in execution time, underscoring the potential of EFFICODER to advance the state of code generation in sssource-costrringe environments. By open- sourcing our model weights, training data, and source code, we aim to foster further research and innovation in this vital area of AI development tools.
Overall, this work aims to address a critical gap in LLM- generated code efficiency while maintaining correctness. We encourage future research to explore trade-offs between efficiency, maintainability, and fairness in automated code generation.
A Acknowledgment
The work is supported in part by National Key R&D Pro- gram of China (20222D0160201), HK RGC RIF (R7030- 22), HK RGC GRF (ref No.: 17208223 17204424), a Huawei flagship research grant 2023, SupernetAI, and the HKU-CAS Joint Laboratory for Intelligent System Software.
Impact Statement
This work is also supported by the National Research Foun- dation, Singapore under its Al Singapore Programme (AISG Award No: AISG2-PhD-200 1TnD-2021-08-000 1-880007) and ITEA Genius and ITEA GreenCode projects, funded by InnovateUK.
This paper presents work aimed at advancing the field of Machine Learning by improving the efficiency and correct- ness of code generated by LLMss. The societal benefits of this research include:
References
Sustainability: By reducing computational resource consumption (e.g., energy and memory usage), our work aligns with global efforts to mitigate the environ- mental impact of large-scale computing. Efficient code generation could lower operational costs and energy demands in industries reliant on software development.
Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., and Chang, K. AVATAR: A parallel corpus for java-python program translation. In Rogers, A., Boyd-Graber, Graber, J. L., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 2268-2281. Association for Com- putational Linguistics, 2023. doi: 10.18653/V1/2023, FINDINGS-ACLL145) URL. https: //doi. org/r0l 100 18653/v1/222
Environments: Enhanced effi- ciency enables broader adoption of L.LM-generated code in mobile, embedded, or edge computing systems, where energy and processing power are limited.
sesiings-cl..14
Ahmed, T. and Devanbu, P. T. Few-shot training 1lms for project-specific sode-summarization. In 37th JEEE/ACM
Research Advancement: Open-sourcing our dataset

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
International Conference on Automated Software En- gineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022, pp. 177:1-177:5. ACM, 2022. doi: 10. 114573551334.3355555 URL https:/ http::::/doo.og////// 10. 1 45/3551344554 .555355551549..5555555
D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. CoRR, abs/2107. 03374, 2021. URL httpps//aaxxii.rggabbb//////////
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferran- dis, C. M.. Muennighoff, N., Mishra, M., Gu, A., Dey, M.,
Umapathi, L. K., Anderson, C. J., Zi, Y., Lamy-Poirier, J., Schoelkopf, H., Troshin, S., Abulkhanov, D., Romero, M., Lappert, M., Toni, D., del Río, G., Liu, Q., Bose, S., Bhattacharyya, U. Zhuo, Y., Yu, I., Villegas, P., Zocca, M., Mangrulkar, S., Lansky, D., Nguyen, H., Contractor, D.. Villa, L., Li, J., Bahdanau, D., Jernite. Y., Hughes, S., Fried, D., Guha, A., de Vries, H., and von Werra, L. San- tacoder: don't t reach for the stars! CoRR, abs/2301 0.3988, 2023. doi: 10.48550AARXX2225000000200
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. CoRR, abs/2304.05 05128, 2023. doi: 10.48550/ARXXIV2234455522 URL https: //doi.org////ooo Ttw.:/..8550///8 2304. 05128.
URL ht tps: 2301. //doi. . wwg. (al88550/axiiii 2301 03988.
Computations, C.
Dolphin coder.
https:
//huggingface cttta/asttecacaaa/c/ccaaaasssttt comitttvecomppaaaiioosoooopppppp 2023.
Anthropic. Introducing the next generation of claude, 2024. URL https://w.antrropp.... com/ news/claude family. ly.
Computer, T.
Glaive-code-ussistant https:
/ facee dat/astasssssss.. Toogther/eee/eeeeeeeeeeeeeeeeeee Glaive Code Assistant, 2023.
Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.00732. 2021. URL https:
Dai, J., Lu, J., Feng, Y., Ruan, R., Cheng, M., Tan, H., and Guo, Z. MHPP: exploring the capabilities and limita- tions of language models beyond basic code generation. CoRR, abs/2405. 1 1430, 2024. doi: 10.48550/ARXIV. 2405.11430. URL https: /ttp///oiiiia/ /ttp:/iooggo..oo//0o/ #xXiv.2405.11430
//arxiv..rg/abb/22111
BigCode. Selfaassinttnntss---eeellll-l-- https:
//w.giiafacce.c//tttttttet////// w0wpeee0eeee-efee-e55500000 2023.
DeepSeekAL. Deepseek coder: Let the code write itself, 2023. URL https: http////eepeeeeeeerrrrre / . github. 10/.
Brown, T. B., Mann. B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert Vosser A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
Deng, Y., Xia, C. S., Yang, C., Zhang, D., Yang, S., and Zhang, L. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. CORR, abs/2304.02014, 2023. doi: 10.48550/ARXII, 2304.02014. URL https: httt:s://doi.or//////// wttp:/iiiooo....00///11 arXiv.2304. 02014.
J. Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Du, M., Luu. A. T. Ji, B., and Ng, S. Mercury: An efficiency benchmark for LLM code synthesis. CoRR, abs/2402.078444, 2024. doi: 10.48550/ARXIV. 2402.07844. URL https:ttt//.e... //t::///doiooo http://....o//./5555555333 arXiv. 2exiv.2402..0 2402. 07844.
Chaudhary, S. Code alpaca: An instruction Ilama model for code generation. https: / htt///gtttte terii.uub.... com/ sahi1280114 httpa/1///ooeeaaaaaeeea
2023.
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: / //operrvv ew net wwwp/eett...ieeeee http:/dfffddhhhh-bboo -innnuuw--1616.... 1MMEIL.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R.. Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, P., Cummings,
10

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T.,
Lai, Y., Li, C., Wang. Y., Zhang, T., Zhong, R., Zettle- moyer, L.., Yih, W.-t., Fried, D., Wang. S., and Yu, T. Ds-1000: A natural and reliable benchmark for data sci- ence code generation. In International Conference on Machine Learning, pp. 18319-18345, PMLR, 2023.
Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need. CoRR, abs/2306.1 1644, 2023. doi: 10.48550/ARXIV. 2306.11644, URL www.thtts///cm http://doo.ooggggo 48550/ D2xiv.2306 11644.
Lemieux, C., Inala, J. P., Lahiri, S. K., and Sen, S. Co- damosa: Escaping coverage plateaus in test genera- tion with pre-trained large language models. In 45th IEEE/ACM International Conference on Software En- gineering, ICSE 2023, Melbourne, Australia, May 14- 20, 2023, pp. 919-931. IEEE, 2023. doi: 10.1109/ ICSE48619.2023.00 00085. URL https://doi.oorg 10. 10.1109/TCCE4480 2004448611.222......22 00085.
Haque, M. M. A., Ahmad, W. U., Lourentzou, I., and Brown, C. Fixeval: Execution-based evaluation of
program fixes for competitive programming problems. CoRR, abs/2206.07796, 2022. doi: 10.48550/ARXIV. 2206.07796. URL https: http://oi....gg//////// arxiv. 220.22266.0 2206.07796.
Li, R., Allal, I., B., Zi, Y. Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskli, E., Zhuo T. Y., Wang. T., Dehaene, O., Davaadorj. M., Lamy-Poirier, J., Mon- teiro, J., Shliazhko, O., Gontier, N., Meade, N., Ze- baze, A., Yee, M., Umapathi, L. K., Zhu J., Lipkin, B., Oblokulov, M., Wang, Z.. V, R. M., Stillerman, J., Patel, S. S., Abulkhanov, D., Zoeca, M., Dey, M., Zhang, Z., Mhoustafa-Fahmy N., Bhattacharyya, U., Yu, W., Singh, S., Luccioni, S., Villegas, P., Kunakov, M., Zhdanov, F., Romero, M., Lee. T., Timor, N., Ding. J., Schlesinger, C.,, Schoelkopf, H., Ebert, J., Dao, T., Mishra, M., Gu, A., Robinson, J., Anderson, J. Dolan- Gavitt, B., Contractor, D., Reddy, S., Fried, D., Bahdanau, D.. Jernite, Y., Ferrandis, C. M., Hughes, S., Wolf, T., Guha, A., von Werra, L., and de Vries, H. Starcoder: may the source be with you! CoRR, abs/2305.0616 2023a. doi: 10.85550AARXIV2205..06666 URL https: / ddoii.or///.ooooooooo /10.//1/.0 000.44550//2222 2305 06161.
Hasan, M., Muttaqueen, T., Ishtiaq, A. A., Mehrab, K. S., Haque, M. M. A., Hasan, T., Ahmad, W. U., Iqbal, A., and Shahriyar, R. Codesc: A large code-description parallel dataset. In Zong, C., Xia, F., Li, W., and Nav- igli, R. (eds.), Findings of the Association for Compu- tational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Find- ings of ACI., pp. 210-218. Association for Compu- tational Linguistics, 2021. doi: 10.18653/V1/2011 FINDINGS-ACL.18 URL https::////// / 18653/v1 1865///1/002...hnddinssss findings-act acl.
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J. Measuring coding challenge competence with apps. NeurlPS, 2021.
Huang, D., Dai, J., Weng, H., Wu, P.. Qing, Y., Zhang, J. M., Cui, H., and Guo, Z. SOAP: enhancing efficiency of gen- erated code via self-optimizaation CoRR, abs/2405. 15189, 2024a. doi: 10:45550AARXII22205555588 URL https: //doi. owg.////////// 10. 485508 2405.15189. 15189.
Li, Y., Choi, D. H., Chung. J., Kushman, N., Schrit- twieser, J., Leblond, R., Becles, T., Keeling, J., Gi- meno, F., Lago, A. D., Hubert, T., Choy, P., de Mas- son d' Autume, C., Babuschkin, I., Chen, X., Huang, P., Welbl, J., Gowal, S., Cherepanov, A., Molloy. J., Mankowitz, D. J., Robson, F. S., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. Competition-level code generation with alphacode. CoRR, abs/2203.0788 2022. doi: 10. 10.4555/ARRXV.2203.0755 / //doi. oro...orr/// /10 10.000000000
Huang, D., Qing. Y., Shang, W., Cui, H., and Zhang, J. M.
Effibench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXiv:2402.02037, 2024b.
Hui, B., Yang, J. Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., Yang, A., Men, R., Huang, F., Ren, X., Ren, X., Zhou, J. and Lin, J. Qwen2.5-coder technical report. 2024. URL attps://apil http://pppsemaaanee semant ieannicschola... ott//oo:pusDD::22777333
URL https: 2203 008814.
Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need II: phi-1.5 technical report. CoRR, abs/2309. .054663,,,,,, 2023b. doi: 10.48550/ARXIV223000085
Jiang N., Liu, K., Lutellier, T., and Tan, L. Impact of code language models on automated program repair. In 45th IEEF/ACM International Conference on Software Engineering. ICSE 2023, Melbourne, Australia, May 14- 20, 2023, pp. 1430- 1442. IEEE, 2023. doi: 10.1 RCSE48669.2023.00122 URL https: htapss//ddoie htt.i....gg////// 10. wtw.1099/ccSR4819.000 202. 00125.
URL. https::/p/// htppp://ddo /doi. org 48544 /05555/aaxxiv.. 2309. 05463.
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances Neural Information P Processing Systems, 36, 2024a.

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Liu, J., Xie, S., Wang, J., Wei, Y., Ding, Y., and Zhang, L. Evaluating language models for efficient code generation. arXiv preprint Ttt.:2200.0045500 2024b.
Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
Luo, Z., Xu, C., Zhao, P., Sun, Q. Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol- instruct. In The Twelfth International Conference on Learning Represenlations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: wtw.:eeiwwee.eetoooooriiiiieeee.
Rozière, B., Lachaux, M., Chanussot, L.,, and Lample, G. Unsupervised translation of programming languages. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Informa- tion Processing Systems 2020. NeurIPS 2020, December 6-12, 2020, virtual, 2020.
MAP. Codeffelbacc-ffirredhsssstttttnn https:
/ htbuggiingaaceecco/ttatttttttts Cooereeebacck-Piiteeee---tssseee 2023.
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu. J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Gaaton-FFerres C., Grattafiori, A., Xiong. W., Défossez, A., Copet, J., Azhar, F. Touvron, H., Martin, L.,, Usunier, N., Scialom, T., and Synnaeve, G. Code Ilama: Open foundation models for code. CoRR, abs/2308. 12950, 2023. doi: 10.48550/ ARXIV.2308. 12950. URL https:: / //t////dooooo/ org/10. 48550/ 2308 12950.
Meta. Introducing meta llama 3: The most capable openly
available Ilm to date, 2024. URL https: / //a///a.. meta.
com/blog wwogl/oooeeoooo. hlogg/mes /meta- 11l1laman 1llmm--3. 3/.
Mir, A. M., Latoskinas, E., Proksch, S., and Gousios, G. Type4py: Practical deep similarity learning-based type in- ference for python. In 44th JEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pitts- burgh, PA, USA, May 25-27, 2022, pp. 2241-2252. ACM, 2022. doi: 10. 10..1445335 10.11455/511000335100144 URL https: 3510124.
Shi, J.. Yang. Z., and Lo. D. Efficient and green large language models for software engineering: Vision and the road ahead. arXiv preprint
//doi. org/ / 10.
Stti::20040455000 2024.
Muennighoff, N., Liu, Q. Zebaze, A. R., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L.,, and Longpre, S. Octopack: Instruction tuning code large lan- guage models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openrevien net / forum? i d- Pww.WWW.
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., and Yazdanbakhsh, A. Learning Performance-Impune Improvinng Code Edits. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Sun, Q., Chen, Z., Xu, F., Cheng, K., Ma, C., Yin, Z., Wang, J., Han, C., Zhu, R., Yuan, S., et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403. 14734, 2024.
Nijkamp, E., Pang, B., Hayashi, H., Tu, L.,, Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: //openreview http::..e/pppppett
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L.., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H.. Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet. X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Sub- ramanian, R. Tan, X. F., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open founda- tion and fine-tuned chat models. CoRR, abs/2307. 09088,9888
Niu, C., Zhang, T., Li, C., Luo, B., and Ng, V. On evaluating the efficiency of source code generated by llms. arXiv preprint 24Xv:22404060441 2024.
OpenAl. GPT-4 Technical Report. CoRR, abs/2303.08774, 2023. doi: 10.488550kaxxi..2333..88 dodoi.oorggooo 100. 4 48550 www.aaawwww........mmm 2303. 08774.
URL https:
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J.. Hilton, J., Kelton, F., Miller, L.,
Simens, M., Askell, A., Welinder, P., Christiano, P. F.,
12

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
2023. doi: 1044550/AARX122230002888 URL https:
Wei, Y., Wang. Z.., Liu, J., Ding, Y., and Zhang, L. Magi- coder: Empowering code generation with oss-instruct. In Forty-first International Conference Machine Learn- ing, ICML 2024, Vienna, Austria, July 21-27, 2024. Open- Review.net, 2024b. URL https: oppnrevie.. net/ vet/forummice (tt.oormmmd--Xuooiiica.... XUeoOBid3x.
//doi //bp:/...o.//....55///////////// 29888
UIUC, I. Magiocader..ool-ssttutt----11 https:
Berpicaaer--vo--nssntt---------- 2023a.
UIUC, I.
Magiiooler-os--nssttutt171... https:
Xia, C. S., Deng, Y., and Zhang, L. Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via 1lm. arXiv preprint
Magicoder -Ss Instruct 75K, 2023b.
Vezora.
Tested-1443k-python-aaatte Tested-143--Python--
https:
arXIv:2403.19114, 2024.
Xu, C.. Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. WizardIm: Empowering large pre-trained language models to follow complex in- structions. The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //opnrevi eww. net fwrum? d=C d=C
2023.
Wang, Y., Wang, W., Joty, S. R., and Hoi, S. C. H. Codet5: identifier-awar unified pre-trained ecoder-decooee mod- els for code understanding and generation. In Moens, M.. Huang, X., Specia, L., and Yih, S. W. (eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vir- tual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 8696-8708. Association for Com- putational Linguisties, 2021. doi: 1118655/112021. EMNIP-MAIN.66., URL https:: /ts:://ddo 18653/v1/2022 emmlp ma-m.ainn.... 685.
Zhao, C., Jia, X.. Viswanathan, V., Wu, T., and Neu- big, G. Self-guide: Better task-specific instruction fol- lowing via self-synthetic finetuning. arXiv preprint 1rXiv:2407.12874, 2024.
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A.. Li, Y., et al. Codegeex: pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 5673-5684, 2023.
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In Rogers, A., Boyd-Graber, L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Pupers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pp. 13484-13508. Association for Computational Lin- guistics, 2023. doi: 10. 10.0865/1/233333333. 1/63AA//222 754. URL https: /tw.///..//////////////../////// wtw.o.oogg/..... 18tp:///...16111111 11233accllomm 754.
Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z.,, Feng, Z., and Ma, Y. Llamafactory: Unified efficient fine- tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computu- tional Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http:/// arxiv. org/ aas///b////// abs/2403 2403. 13372.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models zero-shot learners. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, htt.::t//////eeeeiiiii http///eereeveeeeeeeeeeeeee httpeeewwwwweeeeeeeeeeeeeeeeeeee net forum?
2022. URL https//:::/ id -d=gEZ=GCozdqR.
Wei, J., Durrett, G., and Dillig, I. Typet5: Seq2seq type inference using static analysis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023, URL https: httpp:::///ppeeeeieeee net/ pdE2id=4TyNEhI2GaN
Wei, Y., Cassano, F., Liu, J., Ding, Y., Jain, N., Mueller, Z., de Vries, H., Von Werra, L., Guha, A., and Zhang, L. Selfcodealign: alignmettttnt for code generation. arXiv preprint arXiv:2410.2 24198, 2024a.
13

---

EFFICODER:
Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
A. Appendix
A.I. Prompt Template
Please continue to complete the function. You not allowed to modify the given code and do the completion only. Please return all completed functions in a code block. Here is the given code to complete: 'python [[Prompt)] '..
A.2. Efficiency Metrics
Execution Time (ET) Execution time (ET) measures the average time taken for code execution. Mathematically, ET is defined as:
N T
ET
code
N
where ET is the execution time metric, Toode the execution time of the code (with all the test cases), and N is the number of codes generated by code generation models used for evaluation.
Normalized Execution Time (NET) Normalized Execution Time (NET) measures the execution time required by generated code relative to that of a canonical solution. We define NET as:
N
Teode
NET
N
camonicaa
where Tonde is the execution time of the generated code and Teanonical is the execution time of the canonical solution. A NET value greater than indicates that the generated code is slower than the canonical solution, while value less than I suggests the generated code faster.
Max Memory Usage (MU) Max Memory Usage (MU) measures the average max memory consumption during code execution. Mathematically, MU defined as:
MU Σ Meode
where MU is the memory usage metric, Monte is the max memory consumption of the generated code among all the test cases, and N is the number of code instances generated by code generation models used for evaluation. This metric is critical to assess the resource efficiency of generated code, particularly in environments with limited maximum memory capacity.
Normalized Max Memory Usage (NMU) Normalized Max Memory Usage (NMU) quantifies how the max memory efficiency of the generated code compares the canonical solution. We define NMU as:
Meode
NMU
N Σ
Mennonical
where NU is the normalized max memory usage metric, Modde is the max memory usage of the generated code, and M Mannonical is the memory usage of the canonical solution. An NMU value less than I indicates that the generated code is more memory-efficice than the canonical solution, whereas value greater than I suggests it is less efficient in terms of memory usage. This metric provides relative measure of the memory optimization in the generated code in comparison to a standard baseline.
Total Memory Usage (TMU) Total Memory Usage (TMU) assesses the efficiency of memory usage throughout the execution code, taking into account both the magnitude and duration of memory utilization. To calculate TMU, first, monitor and record the memory usage at discrete time intervals during the execution, resulting in a memory usage profile
14

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
M(), where represents time. Then, compute the area under the curve of M(1 over the total execution time, T using numerical integration methods such the trapezoidal rule:
N Σ TMU = N
Toaa
M(t dt
lower TMU value indicates higher memory efficiency, reflecting an optimized balance between the amount of memory used and the duration of its usage.
Normalized Total Memory Usage (NTMU) The Normalized Total Memory Usage (NTMU) offers a comparison of the dynamic memory efficiency between the generated code and the canonical solution. To determine NTMU, caleulate the TMU for both the generated code and the canonical solution. Normalize the TMU of the generated code by dividing it by the TMU of the canonical solution:
NTMU 1 N TMU code
N Σ
TMU canonical
where T'M Uode is the TMU of the generated code and T'M Uemoninia is the TMU of the canonical solution. An NTMU value less than signifies that the generated code manages dynamic memory more efficiently compared to the canonical solution, while a value greater than indicates less efficient management of dynamic memory. This metric provides insight into the relative use of dynamic memory of generated code compared to an established benchmark.
A.3. Robustness of Overhead Results
The overhead results would be affected by the local environments, which causes that the results of Effi-Code fine-tuned LLMs may not able to represent the results of the efficiency profiling in different environments. To address this issue, we have conducted additional experiments and provided more robust evaluation results.
Firstly, we have evaluated the effectiveness of Effi-Code on seven different hetware-hardwanng setups, as shown in Rebuttal Table 2. The results demonstrate that Effi-Code fine-tuned LLMs achieve higher efficiency than the original LLMs across all setups. For example, in the environment of Python 3. 1.110 Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz, the average execution time decreases from 0.59s to 0.40s when using Effi-Code to fine-tune Qwen2.5-Coden. CCCder-BB reducing the average execution time by 32%.
Secondly, we clarify that for the same setup, where we evaluate the efficiency of LLM--geeaated code several times, the efficiency results are consistent. As shown in Paper Table 8, where execute the LLM- LLM-generated code five times, the standard deviation of execution time (ET) is 0.00548 (s), indicating that the evaluation results are consistent and reliable for a given setup.
Finally, our evaluation setup follows the practices established in recent works on benchmarking the efficiency of automatically generated code, such as Mercury (Du et al., 2024), Effibench (Huang et al., 2024b), and SOAP (Huang et al., 2024a). By adhering to these benchmarks, we ensure that our evaluation is in line with the current standards in the field.
A.4. Test case augmentation
Some of the candidate tasks we collected do not have test cases. To address this, we use GPT-3.5-turo to construct test cases by feeding the task description and source code into the model and requiring it to generate test cases for our experiments. After that, analyze whether each test case generated by GPT-3.5-turbo correct and then filter out incorrect test cases and tasks that do not have the correct test cases. To determine the correctness of the test cases generated by GPT-3.5-turte we execute each test case individually with the initial solution for each task in our collected candidate tasks. We check whether any errors are raised during the execution of each test case with the initial solution. In other words, we verify if the test case passes the initial solution. We treat the test cases that pass the initial solution as correct. On the other hand, test cases that do not pass the initial solution are filtered out. By using the initial solution as a reference, we can effectively assess the correctness of the generated test cases and ensure that only valid test cases are retained for further analysis.
A.5. Comparison with PIE strategies
In Table 7, we compared against the best persoreeforning PIE model, which was the "All" strategy fine-tuned CodeLlama-77B in our evaluation. Now, we included comparisons with other PIE fine-tuning strategies Table EFFINSTRUUT outperforms
15

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Setup
ET
NET
MU
NMU
TMU
NTMU
Python 3.11.10 Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz
Qwen2.5-Coder-7B
0.59
1.95
61.95
0.99
24.29 18.74
1.83
EPPINNTRUCT
0.40
1.01
61.96
0.99
1.02
Python 3.11.10 Intel(R) Xcon(R) Silver 4216 CPU @ 2.10GHz
Qwen2.5-Coder-77 0.28
1.63
36.15
1.00
20.01
1.88
EFFLINTTUUTT
0.25
1.38
36.52
1.01
19.85
1.56
Python 3.11.10 Intel(R) Xeon(R) Silver 41 CPU @ 2.10GHz
Qwen2.5-Coder-7B
0.35
1.45
36.14
1.00
24.28
1.63
ERFINSTRUCT
0.22
1.01
36.51
1.01
15.26
1.09
Python 3.1 1.14 Intel(R) Xeon(R) Silver 4216
CPU 2.10GHz
Qwen2.5-Coder-77
0.67
1.16
61.43
1.00
40.01
1.22
+Effi-Code
0.58
1.02
60.77
0.97
32.50
1.03
Python 3.11.0- Intel(R) Xeon(R) Silver 4216
CPU 2.10GHz
Qwen2.5-Coder--B
0.28
1.64
34.55
1.00
19.39
1.87
EFFINNTTUUCT
0.25
1.39
34.90
1.02
20.03
1.59
Python 3.9.0 Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
Qwen2.5-Coder-7B
0.30
1.60
34.26
1.01
21.02
2.10
+Effi-Code
0.24
1.20
34.52
1.02
19.84
1.32
Python 3.10.0 Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
Qwen2.5-Coder-78
0.29
1.63
33.26
1.01
20.32
2.16
EPHINSTRUCT 0.26
1.43
33.50
1.02
19.53
1.61
Table 8. Evaluation results of EFfiCoder effectiveness on different software-hardware setups.
16

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Table 9, Performance comparison of different model variants efficiency metries.
Model
ET
NET
MU
NMU
TMU
NTMU
CodeL.lama-7b-hf
1.02
0.85
42.33
1.00
23.97
0.82
+ All
0.82 (19.6%)
0.72 (15.3%)
8.87 (79.0%)
0.18 (82.0%)
6.64 (72.3%)
0.24 (70.7%)
+ HQ
1.14 (-11.8%)
0.98 (-15.3%)
10.55 (75.1%)
0.23 (77.0%)
7.06 (70.5%)
0.26 (68.3%)
+ All w/Perf-Cond
0.92 (9.8%)
0.81 (4.7%)
8.91 (79.0%)
0.19 (81.0%)
6.99 (70.8%)
0.25 (69.5%)
+ HQ + Self-Play
0.92 (9.8%)
0.80 (5.9%)
12.46 (70.6%)
0.27 (73.0%)
7.80 (67.5%)
0.28 (65.9%)
+ EFFIINSTRUCT
0.79 (22.5%)
0.70 (17.6%)
11.06 (73.9%)
0.24 (76.0%)
5.77 (75.9%)
0.21 (74.4%)
Table 10. Comparison between EffiLearner and EFFILNSTRUCT the EtfiBench dataset, showing efficiency metrics with percentage
improvements relative to the baseline model.
Model
ET
NET
MU
NMU
TMU
NTMU
deepseek-coder-6. (oaeddr--.77-inntture 7b-instruct
0.56
1.20
40.17
4.09
96.78
13.79
+ EffiLeamer
0.46 (17.9%)
0.98 (18.3%)
40.14 (65.2%)
1.00 (75.6%)
15.50 (84.0%)
1.04 (92.5%)
EFFUINSTUUCT
0.39 (30.4%)
0.83 (30.8%)
40.16 (65.1%)
1.00 (75.6%)
15.03 (84.5%)
0.90 (93.5%)
all PIE variants ET and TMU metrics. It achieves a 22.5% reduction in execution time compared to the base model, which is better than even PIE's best "All" strategy (19.6%) in EffiBench. For total memory usage, EFFIINSTRUCT achieves a 75.9% reduction, outperforming all PIE variants.
A.6. Comparison with EffiLeernee
We provide the comparison for EffiLearner and EFFIINSTRUCT fine-tuned (aapsSeek-Codrr-..77------ where we can observe that both EffiLcarner and EFFIINSTRUCT improve the efficiency of LLM- generated code. For tasks addressed by all models, average ET decreased from 0.56 (s) to 0.46 (s) with EffiLeareer and to 0.39 (s) with EFFINNTRRUCT. For DeepSeck-Code Coder-6.7B 6.B-rr-,7B-Instruccc, EffiLearnrr reduced average execution time by 17.9%, while EFFLINTRRUCT achieved a 30.4% reduction.
in Table 10,
A.7. Evaluation results on ENAMEL
The efficiency results between baseline LLMs and EFFINNSTRUCT fine-tuned models on ENAMEL are shown in Table 1I, where our results show meaningful improvements in both efficiency and correctness metrics, with @ increasing from 0.373 to 0.458 and pass@1 improving from 0.589 to 0.739 for Sven2.5-Coder-Instrutt
A.8. Further discussion with baselines
Compared existing works (Du et al., 2024; Shypula al., 2024), EFFIINSTRUCT introduces a fully automated code optimization framework that transforms initial task descriptions into efficient solutions without human intervention. Unlike PIE, which relies human programmers write efficient solutions, Mercury, which selects the most efficient solution from pre-existing human-written code, EFFIINSTRUCT can optimize code starting from just task description. This automation enables researchers and developers to enhance their existing code generation datasets with minimal manual effort, making efficiency optimization more accessible and scalable. In addition, EFFLINSTRUCT offers broader language
Table 11. Performance comparison between baseline LLMs and EFFILNSTRUCT fine-tuned LLMs the ENAMEL benchmark.
Model
eff@1 1
@ass@1 1
eff@ 10
pass@ 10
eff@ 100
pass@ 100
0.373
0.589
0.628
0.866
0.732
0.951
EFFIINSTRUCT
0,458
0.739
0.653
0.905
0.763
0.972
DeepSeck-Coder I-IInttucct
0.179
0.299
0.549
0.822
0.727
0,922
EFFIINSTRUCT
0.393
0.654
0.633
0.887
0.752
0.937
17

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
coverage and greater generalizability including optimized tasks across multiple programming languages (C++, Python, Java, Rust, and Go). This multilingual approach contrasts with PIE's focus C++ and Mercury's s focus on Python, allowing models fine-tuned on EFFINSTRUCT to perform effectively across diverse language environments. Additionally, EFFIINTTRUCT'S significantly larger scale, comprising 65,710 unique tasks compared to Mercury's 's 1,889 and PIE's 1,474, provides more comprehensive training data, resulting in models demonstrating superior pass@ rates and efficiency metrics as evidenced in Table 7.
A.9. Case Study
To illustrate how the source code generated by EFFIINSTRUCT fine-tuned LLM is more efficient than the source code generated by the LLM without fine-tuning on EFFIINTTRUCT, we provide an example in Figure 5. We can observe that the code generated by Qwen2.5-Coder-1 requires 9.89 (s) to execute all unit tests, while the code generated by EFFIINTTRUCT fine-tuned Aiee2.5-Coder-2 with SFT only requires 0.14 (s) to execute. The key reason is that the code generated by Owen2.5-Coder-- requires significantly more recursive calls, as it lacks optimized pruning strategies such as breaking early in redundant paths. This inefficiency leads to much larger number of computations, ultimately resulting in the observed longer execution time. The code generated by EFFIINSTRUCT fine-ture tued @wee2.5-Coder- on the other hand, incorporates smart optimizations, such as terminating recursion early when certain conditions are met, thereby reducing the overall time complexity.
Task Description
You are given an integer array cookies, where cookies [i] denotes the number of cookies in the ith bag. You are also given an integer k that denotes the number children to distribute all the bags of cookies to. All the cookies in the same bag must go to the same child and cannot be split up. The unfairness of distribution defined as the maximum total cookies obtained by a single child in the distribution. Return the minimum unfairness of all distributions.
Problem:
solution Solution() assert solution.iisrrbuuccooteees/// 15, 10, 20, 8), 2) = 31 assert wwt.etoooo..ssrrbtttoooooee//// 1, 2, 2, 4, 1, 2], 3)= 7
solution solution()
assert solution wwndindss... distri bs (18, 15, 10, 20, 8], 21
assert solution, www.::ooeeeooooeeeeeeeeeeeeeee 1, 3, 2, 2, 4, 1, 21,
Qwen2.5-Coder-7B
Qwen2.5-Coder-7B SFT with EFFILNSTRUCT
Execution Time: 0.14 (8)
Exesution Time: 9.89 (5)
Nemory Peak: 24.39
Max Memory Peak: 24.39 (MB)
Memoxy Usage: 2,47 (MD+s)
Hemory Usage: 210.47
Solution:
class Solution:
def disteiibutetookisstttee
cockies, k):
distri butetookka tself, cookies, k)s
backtrack (11:
mir_unCairnesa float1'inE")
nonlocal
distribution [0] def (istribute
-- min max (chiiddren))
len (cockiess)
nonlocal m.i_Junfsi:ness
return
(cockiiss)
j range (k):
min_unnfairmes min (iin_unaaaaass
children += cookies
max(distrribution)
backtrack +1
return
children1:] cookies[i] if children(5) =-
for in range (k):
discributien(]] scokias(i]
break
dis:ribure + +)
childrer: [0] <
dis:ributicn[j] scokies[i]
float ('inf")
di distribste (0)
backt (0) return ans
return min_untairness
Figure case illustration for the task with code generated by Qwen2.5-Coden and EFFIINSTRUCT fine-tuned Qwen2.5-Coder--7 EtfiBench problem. idx=2305.
18

---

EFFICODER: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Table 12. Code efficiency and pass I of Qwen2.5-Coder-77 with EFFIlNSTRUCT with the five times execution on EffiBench.
Model
4
(Mb)
(Mi*
NTMU
Random Execution
0.17
1.30
32.71
1.03
8,31
2.23
Random Execution
0.17
1.31
32.93
1.04
8.35
2.28
Random Execution
0.17
1.30
32.71
1.03
8.23
2.22
Random Execution
1.30
32.84
1.04
8.30
2.25
Random Execution 5
0.17
1.30
32.88
1.04
8.28
2.27
0.17
1.302
32,814
1,037
8.293
2.249
std
0.0
0.003
0.09
0.003
0.038
0.023
A.10. Randomness
To ensure reliable model performance, we also account for variability in system conditions. Metrics like Execution Time (ET), Max Memory Usage (MU), and Total Memory Usage (TMU) might fluctuate due to factors like server workload and hardware availability, introducing noise that affects performance measurements. To demonstrate whether our results affected by such randomness, provide five results at different times with the mean and std for Qwen2.5-Coder-7B fine-tuned with EFFILNSTRUCT in Table 12. We observe that the results robust the std of the five execution times is very low for all metrics. For example, the std of ET for the five executions is 0.00.
19


Reviews:
[{"review_id": "8VBJ6028K3", "reviewer": "Reviewer_jr1T", "summary": "This paper studies the problem of using an LLM to generate higher performance code. The authors propose a pipeline that first constructs a training dataset by sampling the LLM and choosing generations that have higher performance. Then, they finetune the LLM on slow-fast pairs to get it to generate faster code. They evaluate their approach compared to several baselines.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your insightful comments and suggestions. We provide detailed responses point by point below. We hope that our clarifications, additional experiments, and responses can address your concerns and lead you to consider increasing your rating of our work.\n\n**W1 Limited novelty**\n\nOur paper's primary contribution is application-driven: we introduce the first multilingual code efficiency instruction tuning dataset. As the ICML guidelines note, \"originality need not mean wholly novel methods… a novel dataset... match the needs of the user.\" Our dataset addresses the critical real-world need for efficient code generation, enabling researchers to fine-tune models for improved performance.\n\nNext, our empirical study provides valuable insights by establishing the correlation between training data efficiency and LLM-generated code efficiency - a relationship not previously well understood in the literature. This finding can inspire future construction of SFT datasets that optimize for both correctness and efficiency. Unlike prior methods, PIE [1], which relies on human programmers to write efficient solutions, or Mercury [2], which selects efficient solutions from pre-existing human-written code, SwiftCoder introduces a fully automated framework that transforms task descriptions into efficient solutions without human intervention.\n\nFinally, SwiftCoder offers greater generalizability by including optimized tasks across 5 programming languages, contrasting with PIE's focus on C++ and Mercury's focus on Python. SwiftCoder's significantly larger scale—comprising 65,710 tasks compared to Mercury's 1,889 and PIE's 1,474—provides more training data, resulting in models that demonstrate superior performance. \n\n[1] Learning performance-improving code edits. ICLR 2024\n\n[2] Mercury: A code efficiency benchmark for code large language models. NeurIPS 2024\n\n**W2 Clarify the method in PIE was compared**\n\nWe compared against the best-performed **All** strategy fine-tuned LLM by PIE in the paper. Now, we included comparisons with other PIE strategies in the table below. SwiftCoder outperforms all PIE variants on ET and TMU (e.g., further reduces ET and TMU from **19.6%** and **72.3%** to **22.5%** and **75.9%** for **All** strategy). \n\n|Model|ET|NET|MU|NMU|TMU|NTMU|\n|-|-|-|-|-|-|-|\n|CodeLlama-7B|1.02|0.85|42.33|1.00|23.97|0.82|\n|All|0.82|0.72|8.87|0.18| 6.64|0.24|\n|HQ|1.14|0.98|10.55|0.23|7.06|0.26|\n|All w/Perf-Cond|0.92|0.81|8.91|0.19|6.99|0.25|\n|HQ+Self-Play|0.92|0.80|12.46|0.27|7.80|0.28|\n|SwiftCoder|0.79|0.70|11.06|0.24|5.77|0.21|\n\n**W3 Results with more samples**\n\nOur evaluation follows the setup of prior works (EffiLearner, Mercury), where they use greedy decoding. We conducted additional evaluations for pass@10 (T = 0.8) and will add pass@100 results in our camera-ready manuscript due to rebuttal time limitations. The table below presents our findings across different sampling scenarios. SwiftCoder maintains its efficiency advantage across all configurations.  Even when baselines are sampled 10 times, they still cannot match the efficiency of 1 SwiftCoder sample. When both approaches use the same number of samples, SwiftCoder consistently generates more efficient solutions. Specifically, when both use 10 samples, SwiftCoder improves pass rates by 4-24 percentage points while maintaining better efficiency.\n\n*Reviewer jr1T Ref Table 1*\n\n|Model|ET|NET|MU|NMU|TMU|NTMU|Overlap|Pass@1|\n|-|-|-|-|-|-|-|-|-|\n|**Baseline Sample 1 vs. SwiftCoder sample 1** |\n|deepseek-6.7B| 0.34|2.56|47.26|1.45|30.05|9.97|36.0|44.4|\n|+SwiftCoder| 0.22|1.71|36.31|1.00|9.48|2.11|36.0|51.7|\n|Qwen2.5-7B| 0.31|2.35|31.66|1.00|11.00|2.15|37.2|44.8|\n|+SwiftCoder| 0.16|1.12|31.67|1.00|8.28|1.18|37.2|57.7|\n|**Sample 10 vs. sample 1**|\n|deepseek-6.7B| 0.39|3.00|43.86|1.24|20.14|6.49|48.1|66.8|\n|+SwiftCoder| 0.37|2.68|38.99|1.04|20.13|3.28|48.1|51.9|\n|Qwen2.5-7B| 0.33|2.56|30.99|1.00|9.74|2.33|41.3|49.9|\n|+SwiftCoder| 0.16|1.16|31.01|1.00|7.80|1.24|41.3|57.7|\n|**Sample 10 vs. sample 10**|\n|deepseek-6.7B| 0.44|3.45|41.61|1.19|19.06|6.32|62.5|66.8|\n|+SwiftCoder| 0.40|3.09|42.24|1.19|17.74|5.54|62.5|70.8|\n| Qwen2.5-7B| 0.34|2.59|32.68|1.00|12.09|2.38|48.3|49.9|\n|+SwiftCoder| 0.20|1.49|32.67|1.00|7.82|1.43|48.3|73.8|\n\n**W4 High variance on local machine**\n\nWe use an open-sourced code efficiency platform Monolith for evaluation. Similar to the PIE system simulator, Monolith provides consistent performance evaluation but with broader language support. We also report variability results in the Appendix Table 8, where we execute multiple tasks (32 concurrent tasks) from EffiBench 5 different times on Monolith. Our results demonstrate **consistent performance across different execution runs even under concurrent program load, with a coefficient of variation below 3% across all metrics**. This consistency shows that our measurement approach provides reliable results without requiring specialized system simulators.", "claims_and_evidence": "See below.", "methods_and_evaluation_criteria": "See below.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See below.", "relation_to_broader_scientific_literature": "See below.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n* Important problem: Code performance is a key problem in programming languages and software engineering research, and LLMs show promise in helping with this problem\n\nWeaknesses\n* Unclear novelty: It is not clear what exactly is new about the SwiftCode approach. As far as I understand, they are drawing multiple samples from the LLM, evaluating their performance, and then choosing the best one as the target. There does not appear to be any significant methodological novelty in this pipeline.\n* Unclear PIE baseline: The authors compare to PIE, a recent work on performance optimization for C++ programs. However, PIE actually studies several different strategies for using LLMs for code optimization. The authors should clarify which algorithm in the PIE paper they compared against. Several of these techniques studied in that paper involved finetuning and were highly effective.\n* Lack of test-time search: I’m wondering why the authors only consider taking a single sample from the LLM at test time (i.e., they only study pass@1). For performance optimization, it is pretty typical to take multiple samples, since we can take the fastest program that passes all the test cases. How does the comparison to baselines scale with the number of samples taken?\n* Local execution is noisy: Measuring performance by executing code on a local machine can be very high variance due to a number of factors, including other programs running on the same machine as well as stochasticity in context switches performed by the operating system. The recent PIE paper (Shypula et al., 2024) proposes to use system simulators to mitigate this problem (though it only works for C++, not Python).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nAQ37YkHKe", "reviewer": "Reviewer_pzLu", "summary": "SwiftCode introduces a novel approach to improving code generation in large language models (LLMs) by focusing on both correctness and efficiency. Traditional methods primarily optimize correctness, often neglecting execution speed and memory usage. SwiftCode addresses this gap by fine-tuning LLMs with a curated dataset of high-quality, efficient code. The method involves generating multiple candidate solutions using various LLMs, measuring execution time and memory consumption, and selecting the most efficient option. Experimental results show substantial improvements. This efficiency-aware fine-tuning framework enables LLMs to produce more optimized code, benefiting both software development and computational efficiency.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for his insightful comments and suggestions. We provide detailed responses point by point. We hope our responses can address your concerns and lead you to consider increasing your rating of our work.\n\n**W1 & Q4 Novelty and Comparison with PIE and Mercury**\n\nOur paper's primary contribution is application-driven: we introduce the first multilingual code efficiency instruction tuning dataset. As the ICML guidelines note, \"originality need not mean wholly novel methods… a novel dataset... match the needs of the user.\" Our dataset addresses the critical real-world need for efficient code generation, enabling researchers to fine-tune models for improved performance.\n\nNext, compared to existing works:\n\n1. SwiftCoder introduces a fully automated code optimization framework that transforms initial task descriptions into efficient solutions without human intervention. Unlike PIE, which relies on human programmers to write efficient solutions, or Mercury, which selects the most efficient solution from pre-existing human-written code, SwiftCoder can optimize code starting from just a task description. This automation enables researchers and developers to enhance their existing code generation datasets with minimal manual effort, making efficiency optimization more accessible and scalable.\n\n2. SwiftCoder offers broader language coverage and greater generalizability by including optimized tasks across multiple programming languages (C++, Python, Java, Rust, and Go), contrasting with PIE's focus on C++ and Mercury's focus on Python, allowing models fine-tuned on SwiftCoder to perform effectively across diverse language environments. Additionally, SwiftCoder's significantly larger scale—comprising 65,710 unique tasks compared to Mercury's 1,889 and PIE's 1,474—provides more comprehensive training data, resulting in superior pass@ and efficiency.\n\n\n**W2 Rejection sampling in math and code domain**\n\nWe agree that rejection sampling has been widely used in the mathematical domain. However, this technique is not our key contribution. Our primary contributions are our empirical study establishing the correlation between training data efficiency and generated code efficiency, the development of a multilingual code efficiency dataset, and the end-to-end pipeline for improving code efficiency across multiple languages. \n\nUnlike prior code generation work focuses on binary correctness metrics, efficiency in our work is a continuous metric requiring different optimization strategies. We would appreciate it if the reviewer could suggest specific code-related rejection sampling techniques for efficiency optimization. We would be happy to include them in our paper.\n\n**W3 Sampling params**\n\nWe conduct an ablation study on the sampling size. The evaluation results are shown in **Reviewer jr1T Ref Table 1**, where we evaluate the efficiency results for the most efficient code with 1 and 10 samples. Our results reveal that SwiftCoder consistently achieves higher results.\n\nWe also compared baselines against those fine-tuned on the **original dataset** and SwiftCoder. The table below shows that models fine-tuned on the original dataset decreased efficiency. For example, average execution time increases from 0.33s to 0.37s for DeepSeek.\n\n|Model|ET|NET|MU|NMU|TMU|NTMU|\n|-|-|-|-|-|-|-|\n|deepseek-6.7B|0.33|2.48|34.32|1.00|13.09|2.36|\n|+Original|0.37|2.99|34.18|1.00|10.65|3.04|\n|+SwiftCoder|0.23|1.84|34.17|1.00|8.91|2.30|\n|Qwen2.5-7B|0.30|2.50|26.35|1.00|5.22|2.43|\n|+Original|0.32|2.68|26.23|0.99|5.11|2.54|\n|+SwiftCoder|0.13|1.02|26.32|1.00|2.27|1.03|\n\n\n**Q1&Q2 Inconsistency of column names**\n\nThe purple percentages in paper Table 7 indicate the reduction in overhead metrics compared to CodeLlama-7B-hf without instruction tuning from PIE, Mercury, or SwiftCoder. We provide the detailed results in the Table below and we will add it in our paper in camera ready. \n\n|Method|ET|NET|MU|NMU|TMU|NTMU|\n|-|-|-|-|-|-|-|\n|CodeLlama-7b|0.39|1.94|61.68|1.00|12.78|1.83|\n|+PIE|0.30|1.47|61.39|1.00|11.28|1.68|\n|+SwiftCoder|0.21|1.03|61.33|1.00|7.17|1.04|\n|CodeLlama-7b|0.39|1.94|61.69|1.00|12.78|1.83|\n|+Mercury|0.31|1.51|61.94|1.00|10.24|1.47|\n|+SwiftCoder|0.21|1.01|61.73|1.00|6.95|0.98|\n\n\n**Q3 PIE and Mercury with Deepseek and Qwen**\n\nAll comparative experiments were conducted with fairness as a priority. For Table 7, we used CodeLlama because PIE only provides fine-tuned CodeLlama. For Qwen and DeepSeek, we compared SwiftCoder with Mercury only. The table below shows that both Mercury and SwiftCoder improve the efficiency of LLM-generated code, while SwiftCoder achieves better results compared to Mercury for all LLMs. \n\n|Model|ET|NET|MU|NMU|TMU|NTMU|\n|-|-|-|-|-|-|-|\n|deepseek-6.7B|1.62|2.57|40.92|1.00|49.64|3.03|\n|+Mercury|1.42|2.24|40.82|1.00|46.12|2.78|\n|+SwiftCoder|1.29|2.01|40.79|1.00|42.76|2.55|\n|Qwen2.5-7B|1.23|1.91|46.48|1.00|39.00|1.91|\n|+Mercury|0.86|1.24|47.51|1.01|34.65|1.30|\n|+SwiftCoder|0.70|0.95|46.43|1.00|26.02|0.95|", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes. The authors selected different LLMs to conduct experiments to verify the effectiveness of their method and verified it on multiple datasets.", "relation_to_broader_scientific_literature": "This work is related to code generation. Previously, there have been some baselines and evaluation datasets for evaluating the execution efficiency of code generated by llms.", "essential_references_not_discussed": "As far as I know, no.", "other_strengths_and_weaknesses": "**Strengths**  \n1. The method proposed by the author is simple and effective.\n2. This work has a high reference value for code generation research and is relatively easy to follow. \n\n**Weakness**\n1. The contribution and novelty of this work are relatively limited. \n2. The rejection sampling finetuning method used in this work has been widely adopted by mathematics and code-related research, such as [1]. \n3. There is a lack of more in-depth analytical experiments, such as the exploration of sampling parameters and comparison with the original dataset.\n\n[1] https://arxiv.org/pdf/2308.01825", "other_comments_or_suggestions": "1. Typos: The column names of Table 7 should be kept consistent with those of other tables.\n2. What do the purple percentage codes in Table 7 mean? There seems to be no reference value.\n3. In addition, the comparative experiment in Table 7 selected codellama (which is inconsistent with the baseline model selected in other experiments in the paper). The baseline performance of this model is relatively poor. I would like to ask whether it has been compared with the other two methods on Deepseek and Qwen.\n4. In addition, could you please briefly summarize the differences between your method and PIE and Mercury? As related work, I think these comparisons are very important.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bR0H5C5i5q", "reviewer": "Reviewer_bY75", "summary": "This paper introduces SWIFTCODE, a method to enhance code generation in large language models (LLMs) through efficiency-aware fine-tuning. The paper involves leveraging multiple LLMs to generate diverse code solutions for various tasks across different programming languages, then evaluating these solutions by directly measuring their execution time and memory usage through local execution, selecting the code with the lowest execution time and memory consumption as the final output for each task. Experimental results demonstrate significant improvements when fine-tuning with SWIFTCODE.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for his insightful comments and suggestions. We provide detailed responses point by point. We hope our responses can address your concerns and lead you to consider increasing your rating of our work.\n\n**C1 & W1 Relation & Limitation.**\n\nThank you for acknowledging the practical value of our work on code efficiency optimization. While the concept may seem intuitive retrospectively, SwiftCoder's implementation required significant technical innovation to overcome multiple challenges. \n\nFirst, our multilingual efficiency dataset development required specialized infrastructure to measure code efficiency across five programming languages with consistent metrics, control for system-dependent execution variations, maintain consistent evaluation environments, and scale our pipeline to handle 70,000+ diverse code samples. Second, we established a previously unconfirmed relationship between training data efficiency and LLM-generated code efficiency. Unlike discrete correctness metrics, efficiency exists on a continuous spectrum with complex interactions with functionality, creating unique optimization challenges beyond merely applying the \"quality in, quality out\" principle. Finally, while substantial research has focused on code correctness, efficiency optimization remains critically underrepresented despite its practical importance. SwiftCoder provides both methodology and dataset contributions to this nascent research area.\n\nWe believe these contributions collectively advance the state of efficiency-aware code generation and establish foundations for future research in this important domain.\n\n**C2 & W4 Additional references**\n\nWe couldn't find the latest Effi-Code [1] on ArXiv. If the reviewer provides a link, we'll gladly compare it with SwiftCoder. \n\nWe've added EffiLearner [2] comparison in the Table below, showing SwiftCoder's superior performance. On EffiBench, SwiftCoder reduces average execution time more effectively than EffiLearner. Unlike EffiLearner, which decreases pass@1 rates, SwiftCoder consistently improves pass@1 across all evaluations.\n\n|Model|ET|NET|MU|NMU|TMU|NTMU|\n|-|-|-|-|-|-|-|\n|DeepSeek-6.7b|0.56|1.20|40.17|4.09|96.78|13.79|\n|+EffiLearner|0.46|0.98|40.14|1.00|15.50|1.04|\n|+Ours|0.39|0.83|40.16|1.00|15.03|0.90|\n\n\nWe also evaluated SwiftCoder on ENAMEL [3], which tests efficiency on enhanced HumanEval test cases. As shown in the Table below, SwiftCoder consistently improves both pass@k and effi@k metrics across all models and k values (1, 10, and 100).\n\n| Model|effi@1|pass@1|effi@10|pass@10|effi@100|pass@100|\n|-|-|-|-|-|-|-|\n|Qwen2.5-7B|0.373|0.589|0.628|0.866|0.732|0.951|\n|+SwiftCoder|0.458|0.739|0.653|0.905|0.763|0.972|\n|DeepSeek-6.7B|0.179|0.299|0.549|0.822|0.727|0.922|\n|+SwiftCoder|0.393|0.654|0.633|0.887|0.752|0.937|\n\n\nCoignion et al [4] focus on LeetCode efficiency evaluation, which is partially covered in the evaluated EffiBench (See paper Table 2) dataset.\n\nJiang et al [5] collected 52 bilingual programming questions (Chinese and English) from existing benchmarks. Their work is under review, and the dataset is not publicly available yet. We will evaluate their dataset when it becomes available and include all related discussions and results.\n\n**W2 & Q1 Test cases & ground truth.**\n\nAll tasks in our datasets have verified solutions (ground truth) that correctly fulfill their descriptions. For example, SelfCodeAlign only includes code that passes all test cases in a controlled environment.\n\nSince most datasets lack test cases, we used GPT-4 to generate them based on task descriptions and solutions. We validated these tests by running them against the original solutions from the datasets. Only tests that executed successfully were retained; failed tests were filtered out. Tasks without valid tests were removed entirely.\n\nDuring optimization, we ran LLM-generated code against these validated tests. Solutions failing any test were eliminated. From the remaining valid solutions, we selected the most efficient implementation as the final optimized code.\n\n**W3 & Q2 Data Contamination**\n\nWe addressed data contamination concerns in our SwiftCoder dataset. Analysis shows no exact duplicates between training and evaluation sets, with only 0.20% of evaluation samples having minimal vocabulary overlap (5-10%). Our dataset construction included decontamination processes, as did our source datasets like Evol-Ins, which removed content from common benchmarks. Experiments on EvoEval, designed to avoid benchmark leakage, still demonstrate our approach produces more efficient code than the original LLM-generated solutions.\n\n*Table 1 Data Overlap Analysis*\n\n| Metric | Value | Percentage |\n|-|-|-|\n|Training set size | 65,710| - |\n|EffiBench size|1,000|-|\n|Exact duplicates|0|0.00%|\n\n*Table 2 Vocabulary Overlap Analysis*\n\n| Overlap Range | Test Samples | Percentage |\n|-|-|-|\n|0.00-0.05|0|0.00%|\n|0.05-0.10|2|0.20%|\n|0.10-0.15|0|0.00%|\n|0.15-0.20|0|0.00%|", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The candidate solutions appear not to have undergone any correctness verification, which is perplexing.", "theoretical_claims": "This paper is not concerned with the proof of theoretical claims.", "experimental_designs_or_analyses": "Yes. The experimental design is sound and substantial. But I think some further elaboration and additions about generalizability are missing.", "relation_to_broader_scientific_literature": "I think the idea of this paper is easy to think of in general, constructing high-quality expected training data to fine-tune the LLM to make it output a higher quality response. It meets expectations and works well. In other words, this paper uses a popular idea to solve the still relatively rarely solved problem of poor efficiency of LLM generated code.", "essential_references_not_discussed": "Recently, several studies have been dedicated to improving the efficiency of code generated by large language models (LLMs). The authors should consider these works and select representative examples as baselines. For instance:\n1. Effi-Code: Unleashing Code Efficiency in Language Models\n2. EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization (NeurIPS 24)\n\nAt the same time, the evaluation of the efficiency of LLM-generated code has received considerable attention. In addition to those discussed in this paper, I have identified several recent studies. Given that the literature on this topic is not yet extensive, it warrants thorough discussion. Examples include:\n1. How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark\n2. A Performance Study of LLM-Generated Code on Leetcode\n3. From Effectiveness to Efficiency: Comparative Evaluation of Code Generated by LCGMs for Bilingual Programming Questions.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well structured.\n2. It addresses a very important problem.\n3. The experimental evaluation is extensive, and the empirical results are promising.\n\nWeakness:\n1. Limited Novelty: Although I acknowledge that the paper tackles a critical issue and represents one of the pioneering efforts to enhance the efficiency of code generated by LLMs, it appears to lack distinctive technical contributions.\n2. Unclear Technical Design: It seems that the approach for collecting and filtering candidate solutions relies solely on performance metrics obtained from profiling, without assessing correctness. If my understanding is correct, I am concerned that this step might restrict the fine-tuning of LLMs in generating correct code (The author needs to explain the rationale and motivation behind). Otherwise, the authors should clarify how test cases and ground truth are obtained for tasks across different programming languages.\n3. Generalizability Requires Further Discussion: I have some reservations regarding the generalizability of the trained model. Although diverse datasets were used during training and evaluation, the authors did not discuss the distributional differences between these datasets—for instance, whether the evaluation dataset contains tasks absent in the training data. A more thorough discussion on the distributional discrepancies between the datasets would further substantiate the generalizability of the proposed method in generating more efficient code.\n4. Some Essential References Not Discussed: As mentioned above.\n\nI am very eager to engage in discussions with the authors and to revise my comments accordingly.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. Regarding dataset construction, how were the test cases and ground truth obtained? Alternatively, was there no verification of the correctness of the generated candidates?\n2. What is the task distribution between the training data and the evaluation data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0VX0g3LXEL", "reviewer": "Reviewer_EjXH", "summary": "This work develops a new instruction-tuning dataset called SwiftCode for efficiency-aware fine-tuning of LLMs for code generation. After fine-tuning on SwiftCode, LLMs are able to generate more efficient code on popular code generation benchmarks.\n\n## Update after rebuttal\n\nThe rebuttal has addressed my concerns, so I keep my positive score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for his insightful comments and suggestions. We provide detailed responses point by point. We hope our responses can address your concerns.\n\n**W1: LLM-generated code may be inefficient**\n\nThank you for raising this important point about the efficiency of our candidate code generation. To address this concern directly, we evaluated both DeepSeek-Coder (Lite) and GPT-4o on the ENAMEL benchmark [1]. As shown in Table 1, these models achieve higher effi@k scores compared to most LLMs reported in Qiu et al. [1], confirming their capability to generate sufficiently efficient code candidates.\n\nOur optimization methodology further enhances this efficiency. After applying our techniques, we observed substantial improvements: average ET decreased from 1.14s to 0.27s, while average TMU dropped from 26.24MB*s to 5.13MB*s—representing 75-80% improvements across efficiency metrics (Figure 4 in our paper).\n\nThese results demonstrate that our approach effectively generates and optimizes code that meets high efficiency standards, even when evaluated on benchmarks specifically designed to assess code efficiency.\n\n*Table 1: Evaluation results of DeepSeek and GPT4o generated code in ENAMEL dataset [1]. Due to OpenAI token limitations, we then only provide the results of GPT4o for k=1 and 10.*\n\nModel | eff@1 | pass@1 | eff@10 | pass@10 | eff@100 | pass@100\n|-|-|-|-|-|-|-|\nDeepSeek-Lite | 0.390 | 0.638 | 0.564 | 0.838 | 0.671 |  0.901\nGPT-4o | 0.300 | 0.465 | 0.572 | 0.845 | N/A|N/A |\n\n\n\n**W2: This work uses canonical solutions to evaluate efficiency. The authors evaluate the generated code on the benchmark of [1]**\n\nThank you for highlighting this important methodological consideration regarding canonical solution efficiency. We address this concern in two ways:\n\nFirst, EffiBench, our primary evaluation benchmark, uses optimized canonical solutions provided by the dataset constructors and employs sufficiently large test inputs to meaningfully differentiate between algorithmic complexities (e.g., O(n) vs O(n²)). This allows reliable measurement of efficiency differences across implementations.\n\nSecond, to further validate our approach, we conducted additional experiments using the ENAMEL benchmark [1], which specifically emphasizes efficient canonical solutions and large-scale inputs. Table 2 presents these results, comparing baseline models against SwiftCoder fine-tuned versions.\n\nThe results show substantial improvements across all metrics. For example, eff@1 (measuring both efficiency and correctness on a single generation) increases from 0.373 to 0.458 (+22.8%) for Qwen2.5-Coder and from 0.179 to 0.393 (+119.6%) for DeepSeek-Coder.\n\nIn our final version, we will include these ENAMEL benchmark results to provide a more comprehensive evaluation of SwiftCoder.\n\n*Table 2. Evaluation results of baseline models vs. SwiftCoder on ENAMEL benchmark.*\n\n| Model|effi@1|pass@1|effi@10|pass@10|effi@100|pass@100|\n|-|-|-|-|-|-|-|\n| Qwen2.5-7B| 0.373|0.589|0.628|0.866 |0.732 |0.951|\n| + SwiftCoder|0.458|0.739|0.653|0.905 |0.763 |0.972|\n| DeepSeek-6.7B|0.179|0.299|0.549|0.822 |0.727 |0.922|\n| + SwiftCoder|0.393|0.654|0.633|0.887 |0.752 |0.937|\n\n\n**W3 Discussion on negative results**\n\nWe appreciate your noting the absence of discussion on negative results. You correctly identified that, in some instances, fine-tuning on SwiftCode produced negligible improvements or slight performance decreases in certain metrics, as shown in Table 3.\n\nThese variations stem from our optimization approach using the TMU metric for code selection. When optimizing for this composite metric, improvements in TMU may occasionally come at the expense of a slight degradation in individual metrics (either execution time or memory usage). For example, a solution might achieve substantial memory efficiency while slightly increasing execution time, resulting in an overall improved TMU score but showing a negative trend in the execution time metric when viewed in isolation. We'll include a detailed discussion of these cases to better illustrate the complex relationships between efficiency metrics in the revised version.\n\n\n**T1: line 72 typo**\n\nThank you for identifying this error. We will correct it in the revised version.\n\n\n**Q1: Space-time tradeoff**\n\nWhen selecting the final code for our dataset, we addressed the space-time tradeoff using the TMU (Total Memory Usage) composite metric that combines both execution time and memory consumption into a single evaluation metric.\n\nRather than treating time and memory as separate dimensions requiring individual optimization, TMU provides a holistic efficiency measure that accounts for both resources simultaneously. This approach acknowledges the inherent tradeoffs in algorithm design (such as dynamic programming vs. brute force approaches) and allows us to identify solutions that achieve the most favorable overall resource balance.", "claims_and_evidence": "Weakness 1 (missing evidence): This work uses LLMs to generate candidate code for SwiftCode. However, it has been recently shown [1] that even the strongest LLMs still fall short of generating efficient code on most HumanEval tasks when compared with human expert solutions. Hence, it is unclear whether DeepSeek-Coder and GPT-4o have the ability to generate sufficiently efficient candidate code. This work would be more convincing if, for example, the authors evaluate DeepSeek-Coder and GPT-4o on the benchmark of [1] to provide evidence of how efficient the candidate code is. \n\n- [1] Qiu et al. How efficient is LLM-generated code? A rigorous & high-standard benchmark. ICLR, 2025.", "methods_and_evaluation_criteria": "Strength 1 (dataset scale): This work develops a dataset consisting of 65710 tasks. To curate this large dataset, this paper proposes a versatile and scalable framework to process and select candidate solutions. \n\nStrength 2 (comprehensiveness): The SwiftCode dataset has integrated seven datasets that covers five popular programming languages (Python, C++, Java, Rust, and Go). Hence, this dataset has great potential to benefit a great many programming applications. \n\nWeakness 2 (evaluation): This work uses canonical solutions to evaluate efficiency. However, it has been recently shown [1] that many of the canonical solutions in existing benchmarks are not efficient. Thus, normalized efficiency metrics (like NET) do not fully capture the true efficiency of LLM-generated code. Meanwhile, unnormalized metrics (like ET) alone are not as meaningful besides comparison purposes, because execution time differs for various tasks and typically increases w.r.t. the input scale. For example, an $O(n)$ solution and an $O(n^2)$ solution might have similar execution time when $n$ is small but should have significantly different execution time when $n$ is large. Therefore, the evaluation results would be reflect true efficiency more accurately if, for example, the authors evaluate the generated code on the benchmark of [1], which uses efficient canonical solutions and large-scale inputs in evaluation. \n\n- [1] Qiu et al. How efficient is LLM-generated code? A rigorous & high-standard benchmark. ICLR, 2025.", "theoretical_claims": "This paper does not make theoretical claims.", "experimental_designs_or_analyses": "Weakness 3 (negative results): This paper has no discussion on negative results, so the analysis in Sec 4 (Experiment) seems a little bit misleading. In particular, in a few cases, fine-tuning on SwiftCode has only negaligible improvement or even worsens the performance (see, e.g., Table 3). The authors should analyze and discuss why such negative results happen.", "relation_to_broader_scientific_literature": "Strength 3 (new direction): SwiftCode is the first instruction-tuning dataset designed to improve the efficiency of LLM-generated code. This opens up an interesting new direction for code generation research.", "essential_references_not_discussed": "This paper has discussed most of the essential references.", "other_strengths_and_weaknesses": "There are no other strengths or weaknesses that I want to point out especially.", "other_comments_or_suggestions": "On line 72, the paper mentioned Qwen but cited DeepSeek. This seems like a typo.", "questions_for_authors": "Question 1 (space-time tradeoff): Sec 3.2 mentions that the code with the lowest time and memory is selected as the final code. However, the most time-efficient code might not be the most memory-efficient, and vice versa. For example, a dynamic programming algorithm may be faster than brute force but may need more memory, while a brute force algorithm may need less memory but may be much slower than dynamic programming. This is known as *space-time tradeoff* in algorithmic literature (see, e.g., [2]). How did you handle space-time tradeoff when selecting the final code in your dataset?\n\n- [2] Hellman (1980). A cryptanalytic time-memory tradeoff. IEEE Transactions on Information Theory. 26(4): 401-406.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces SWIFTCODE, a framework for enhancing code generation efficiency in LLMs through efficiency-aware fine-tuning. It constructs a training dataset by sampling the LLM and choosing high-performance generations, then fine-tunes the LLM on slow-fast pairs to generate faster code. Evaluations across multiple benchmarks and programming languages demonstrate improved efficiency over baseline methods.

Reviewers acknowledge several strengths of the paper. It addresses an important problem in programming languages and software engineering, filling a gap in code generation research. The development of the SwiftCode dataset is a strength. The dataset is large-scale and comprehensive, covering multiple programming languages and facilitating efficiency-focused fine-tuning. This directly contributes to the framework's ability to enhance code generation efficiency. The experimental evaluation across diverse LLMs and programming languages is extensive. The framework’s simplicity and scalability are also recognized as strengths, with potential applications in software engineering workflows.

On the other hand, the paper also has some weaknesses. The novelty of the approach is limited, as the pipeline closely resembles prior rejection sampling methods in code/math domains. Methodological ambiguities need to be addressed, including the missing correctness verification for candidate solutions, test-time search, and metrics on noisy local execution. There is a lack of clarity regarding the comparison with baselines such as PIE. More in-depth analytical experiments are also expected, such as the exploration of sampling parameters and the generalizability of the trained model.

The authors have submitted their responses to the reviews. Subsequently, all the reviewers have confirmed that they have read the authors' responses, and some of them have updated their overall assessments based on these responses.
Decision: Accept (poster)