EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization
Authors: Dong HUANG
Jianbo Dai
Han Weng
Puzhen Wu
Yuhao QING
Heming Cui
Zhijiang Guo
Jie Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Code Generation
Efficiency
Primary area: natural_language_processing
TL;DR: In this paper, we propose SOOP to improve the efficiency of LLMs generated code.

Abstract: Large language models (LLMs) have shown remarkable progress in code generation, but their generated code often suffers from inefficiency, resulting in longer execution times and higher memory consumption. To address this issue, we propose EffiLearner, a self-optimization framework that utilizes execution overhead profiles to improve the efficiency of LLM-generated code. EffiLearner first generates code using an LLM, then executes it locally to capture execution time and memory usage profiles. These profiles are fed back to the LLM, which then revises the code to reduce overhead. To evaluate the effectiveness of EffiLearner, we conduct extensive experiments on EffiBench and two commonly used code generation benchmarks with 16 open-source and 6 closed-source models. Our evaluation results demonstrate that through iterative self-optimization, EffiLearner significantly enhances the efficiency of LLM-generated code. For example, the execution time (ET) of StarCoder2-15B for the EffiBench decreases from 0.93 (s) to 0.12 (s) which reduces 87.1\%  execution time requirement compared with the initial code. The total memory usage (TMU) of StarCoder2-15B also decreases from 22.02 (Mb*s) to 2.03 (Mb*s), which decreases 90.8\% total memory consumption during the execution process.

EFFI-LEARNER: Enhancing Efficiency of Generated Code via Self-Optimization
Dong Huang" The University of Hong Kong dhuangges. .ku.hk
Jianbo Dai University of Edinburgh j6dj6d0gmail. com
Han Weng Beijing University of Posts and Telecommunications han. wengdbupt. edu. cn
Puzhen Wu University College Dublin puzhen. wu@ucdconnect.c6 ie
Yuhao Qing
Heming Cui The University of Hong Kong Shanghai Al Laboratory hemingêcs .ku hk
The University of Hong Kong yhqing@cs hku hk
Zhijiang Guo University of Cambridge zg2830cam. ac. uk
Jie M. Zhang King's College London jie zhang@kcl. ac. uk
Abstract
Large language models (LLMs) have shown remarkable progress in code genera- tion, but their generated code often suffers from inefficiency, resulting in longer execution times and higher memory consumption. To address this issue, we propose EFFI-LEARME ENRRR, self-optimization framework that utilizes execution overhead profiles to improve the efficiency of LLM-generated code. EFFI-LEARNER first generates code using an LLM, then exccutes it locally to capture exccution time and memory usage profiles. These profiles are fed back to the LLM, which then revises the code to reduce overhead. To evaluate the effectiveness of EFFI- LEARNER, we conduct extensive experiments on the EffiBench, HumanEval, and MBPP with 16 open-source and 6 closed- foosed-oouree source models. Our evaluation results demonstrate that through iterative seff-optimization, EFFI-LEARNER significantly enhances the efficiency of LLM--genrated code. For example, the execution time (ET) of StarCoder2- 15B for the EffiBench decreases from 0.93 (s) to 0.12 (s) which reduces 87.1 execution time requirement compared with the initial code. The total memory usage (TMU) of StarCoder2-15B also decreases from 22.02 (Mb*s) to 2.03 (Mb*s), which decreases 90.8% total memory consumption dur- ing the execution process. The source code of EITI-LARRER was released in htttps://gitie com/ hww////nngggg9//fffffeeeea.a
Introduction
Large language models (LLMs) have recently achieved significant advancements across various tasks 49 4
LLMs such as GPT-4 (50) and Copilot 44] have demonstrated considerable
efficacy in code related applications, including code completion [1 6 debuggings [2] and translation [54 D. These innovative tools have been seamlessly integrated into popular develop- ment environments. significantly augmenting developer productivity by providing intelligent code
"Equal Contribution.
t "ooresponniigg Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Task Description
Given com-e array of integers nums, every element appears twice except for one. Find that single one. You must implement solution with a linear runtime complexity and use only constant extra space.
Problem:
Example: Input: nums = [4,1,2,1,2]
Output: 4
assert eingleNunber ((4, 1, 1, 2])
Initial Completion with Profile
EFFI-LEARNER Completion with Profile
Timer unit: 1e-06
Timer wnit: 1e-06
Total tine: 0.0312331
Total tine: 0.000112618
Nem unit: MiB
Nem unit: MiB
Time Mem usage
Line Contents
Time Men usage Line Contents
def singleNunber (nuna):
def singleRunber (nums):
371.2 50.9
for 1 in range (len (nuns));
112.6 50.7
return reduce (operator 2O2, nume)
30860.9 50.9
if count (nuns (i])
0.9 50.9
retura nuns [i]
Figure I: A case for the task with code and EFFI-LEARNER refined version. The lower left panel shows the initial completion generated by an LLM, its profile shows its inefficiency. The lower right panel shows the final efficient answer output by applying EFFI-LEARNER.
recommendations based natural language instructions. However, the primary focus of existing efforts has predominantly been on the correctness of the generated code 18]. ensuring it meets the functional requirements and adheres to syntactical norms.
Despite advancements in ensuring code correctness, there remains significant gap in the literature regarding the efficiency of code produced by LLMs. Efficiency is crucial as it translates to faster execution and lower memory and processing power consumption, which especially important in resource-constrained environments such as mobile devices or embedded systems [18] 53 13 48
17 [6, 9 41, 52]. Recent studics [56] 48 reveal that LLM-generated code often exhibits lower efficiency in terms of execution time and memory usage when compared to canonical solutions. For instance, on a benchmark that evaluates efficiency, EffiBench [27], even the most powerful LLM (c.g.. GPT-4-Turbo) generates code with suboptimal efficiency. The average and worst-case execution times are 1.69 and 45.49 times longer than those of the canonical solutions, respectively. This inefficiency underscores the need for developing new methods focused evaluating and improving the efficiency of code generated by LLMs, ensuring that they produce correct and highly efficient code.
To bridge this gap, we draw inspiration from the methodology used by coders coding platforms. When addressing programming problem, coders typically write an initial program that is executable on the test cases. Next, they execute the code and obtain a profile of the execution time and memory usage overhead, as shown in Figure I Based on this overhead profile, the coder optimizes the code to enhance its efficiency. During this process, the coder extracts key information (e.g., execution times and memory usage of each line) from the overhead profile, which helps identify lines or operators that require significant overhead (e.g., loops that execute multiple times unnecessary variables being saved). This information assists the coder in optimizing their code.
With this motivation, we propose EFFI-LEARNER to improve the efficiency of LLM-gennrated code. As shown Figure 2 EFFI-LEARNER first requires the LLM to generate code based on the task description. Then, LEAR-E executes the generated code locally and captures the execution time and memory usage profile. These overhead profiles are fed back into the LLM, which then revises the code to reduce the overhead. Through multi-iteration self-optimization, the efficiency of the LLM-generated code is improved. While it's true that the iterative process requires extra time, it's crucial to recognize the long-term advantages that come with this investment. By optimizing the code, we can enhance the overall efficiency once it's deployed.
To evaluate the effectiveness of EFFI- LEARNER, we conduct extensive experiments on the EffiBench and two commonly used code generation benchmarks (i.e. HumanEval [IT] and MBPP 6 with 16 open-source and 6 closed-source models. We compare the efficiency of the code generated by the L.M before and after applying EFFI-LFARNER. The experimental results demonstrate that EEEI-LEARNER significantly improves the efficiency of the LLM-generated code. For example, the

---

execution time (ET) of 15B decreases from 0.93 (s) to 0.12 (s) which reduces 87.1% execution time requirement compared with the initial code. The max memory usage (MU) of DeepSeek-6.1 -.7-ek-s also decreased from 259.73 (Mb) to 36.97 (Mb), which reduces 85.8% max memory consumption for the code execution requirement. The total memory usage (TMU) of StarCoder2-15t 155 also decreases from 22.02 (Mb*s) to 2.03 (Mb*s), which decreases 90.8% total memory consumption during the execution process.
2
Related Work
2.1 LLMs for Code Generation
The increasing popularity of LLMs for code generation has coincided with the growing availability of open-source code repositories and the need to boost developer productivity. Initial efforts focused on training models specifically for coding tasks, such as CodeT5 [61], AlphaCode [33], CodeGen [47), InCoder [19]. StarCoder [32]. SantaCoder
3 and DeepSeek Coder [T]]. Contrastingly, models 55 represent subsequent stride, being finetuned from
such as Codex [ and CodeLLLMM
foundation models
g
59 These code LLMs have been applied to various tasks, including code
generation [T 14
4
program repair [5 28
automated testing [6), code translation [54] 11. type
prediction 45 64 and code summarization [26] 2). Among these, code generation, where models
generate code snippets based on natural language descriptions docstrings, has become a critical domain for evaluating LLMs. While LLMS have achieved impressive results in code generation tasks like HumanEval
I1 and MBPP 5 their efficiency has received less attention. Recent studies [56] 27 48) have shown that LLM -eMeerendteed code exhibits lower efficiency in execution time and memory usage compared to canonical solutions. These findings highlight the need for further research and development to improve the efficiency of LLM-generater code. In this work, we propose the first method that significantly improves the efficiency of code generated by a wide range of LLMss
2.2 Learning From Feedback
A prevalent strategy for improving the behavior of LLMs is learning from feedback, mirroring human learning where individuals refine their actions through trial, error, and correction 77] 43).
Early efforts involve using human feedback to evaluate and refine models [30 51 [1]. To minimize human intervention, another strategy focuses on automated feedback. These methods iteratively learn from automatically generated feedback signals, understanding the consequences of their actions and adapting their behaviors. The source of this automated feedback can be diverse, ranging from the LLM itself 44 (7). external tools 22 37 or verifiers (36]. external knowledge sources [20 688 and
even generation logits (67]. In code generation, the program executor is frequently used as source of feedback for refining the model's initial code. For example, Self-Edit [69] and Self-Evolve (29) execute the initial program on example test cases and provide the execution results as feedback, prompting the LLM to refine the code. Self- Delf-Debug [12] explores using program explanation, unit tests, and program interpreters as feedback types. ALGO [70] employs a more fine-grained approach by generating a reference oracle program that solves the problem with exhaustive search. Feedback is then collected by comparing the generated outputs with the oracle. While existing work primarily focuses using feedback to edit the initial code to ensure correctness, our method explores using overhead profiles to improve the efficiency of the code.
3 EFFI-LEARNER
Inspired by the optimization strategies employed by human coders on coding platforms, we propose a framework EFFI-LEARNER to enhance the efficiency of LLM--genrrttr code. Human coders typically analyze execution time and memory usage profiles to identify bottlenecks and optimize their code. EFFI-LEARNER leverages this principle by integrating self-optimization loop into the code generation process. As illustrated in Figure EFFI-LEARNER consists of three main components: Code Generation, Overhead Profiling, and Code Refinement, each playing a crucial role in the self-optimization process.
3

---

CODE
LLM
Self-Oplimization
Efficioncy Analysis
m
Overhead
Profile
Figure 2: Pipeline of EFFI-LEARNER, LLMs first generate code for the given problem. This code then executed locally to gather overhead profiles. These profiles are subsequently utilized by the LLMs to optimize the code in successive iterations, thereby enhancing the overall efficiency of the generated code. A comprehensive illustration is provided in the Appendix Figure Figure
3.1 Code Generation
Given task description or code generation requirement, the LLM generates an initial version the code. The LLM takes the task description input and produces code that aims to solve the task.
3.2 Overhead Profiling
The generated code is executed locally to capture its execution time and memory usage overhead profiles. During this step, the code is run on a set of open test cases, and the execution time and memory usage for each line of code are recorded. This information forms the overhead profiles that provide insights into the efficiency of the generated code.
Execution Time Profiling In this step, we measure the execution time of each line of code to identify potential bottlenecks and inefficiencies. To perform execution time profiling, we utilize the sine_profiles library in Python. During the profiling process, we run the generated code on set of open test cases provided by the dataset. The 1 inn_proofller library tracks the execution time of each line of code for all the test cases combined. This helps assess the code's performance under different conditions and identify any performance bottlenecks. The execution time profiling results reported based on the total consumption for all open test cases. The profiling output includes information such as the line number, the number times each line executed, and the total time spent each line. These profiles serve as input for the subsequent code refinement step.
Memory Usage Profiling Memory usage profiling another essential aspect of the EFFI-LEARNER framework. It helps us understand how the generated code utilizes memory resources and iden- tifies any memory-related inefficiencies or leaks. To profile memory usage, we employ the hemory-prov iier library in Python. During the memory usage profllng process, we run the generated code on the set open test cases. The cammoryprrofllle library monitors the memory usage of each line of code throughout the execution of all the test cases combined. It captures the memory usage at different points, such as before and after function calls, loop iterations, and memory allocation statements. The memory usage profiling results are reported based on the total consumption for all open test cases. The profiling output includes information such as the line number and the corresponding memory usage. These profiles provide valuable insights into the memory efficiency of the generated code and help identify areas for optimization.
3.3 Code Refinement
This component leverages the execution time and memory usage profiles to optimize the generated code. In this step, the LLM analyzes the overhead profiles to refine the code for better efficiency. To

---

enable self-oppimizaion, we feed the overhead profiles back into the LLM along with the generated code. The LLM analyzes patterns in the overhead profiles, such as high execution time or excessive memory usage, and correlates them with specific code segments. It then applies optimization techniques, such as loop unrolling, memorization, data structure optimization, algorithm substitution, and code simplification, to improve the efficiency of the identified code segments.
During the self-optimization process, the LLM considers factors such as the impact of each opti- mization on the overall efficiency, the trade-off's between execution time and memory usage, and the preservation of code correctness. It aims to strike balance between performance improvement and maintaining the functional integrity of the code. The LLM iteratively refines the code based on the overhead profiles, applying optimizations until a satisfactory level of efficiency achieved or a predefined number of iterations is reached. The optimized code is then validated against the open test cases to ensure its functional correctness. By leveraging the execution time and memory usage profiles, the self-optimization step enables the LLM to improve the efficiency of the generated code.
3.4 Prompt Construction
We carefully design prompts to guide LLMs in optimizing code efficiency while ensuring the optimized code passes predefined test cases. The prompt template (Figure used in EITI-LEARNER'S self-optimization stage includes a task description, test case, initial code, overhead analysis, and optimization rules. The task description provides context and requirements, the test case ensures correctness, and the initial code is the starting point for optimization. The overhead analysis highlights performance metries and areas for improvement, while the optimization rules focus the LLM on enhancing efficiency, encapsulating the optimized code, and excluding the test case from the code block. This comprehensive prompt equips the LLM with the necessary information effectively optimize code, maintain consistency across models and tasks, and facilitate comparison of their code optimization capabilities, advancing the field of LLM-driven code optimization. Details of the template can be found in Appendex A.3
4 Evaluation
4.1 Dataset and Metrics
We evaluate EFFI-LEARNER on EffiBench [27] Following their settings, we use Execution Time (ET), Normalized Execution Time (NET), Max Memory Usage (MU), Normalized Max Memory Usage (NMU), Total Memory Usage (TMU), and Normalized Total Memory Usage (NTMU) as metrics. We provide a detailed definition of these metrics in A.5 Following the setup of EffiBench, evaluation metrics were only calculated on the tasks that generated code for both the initial version and EFFI-LEARNER optimized code that can pass all private test cases provided by the datase
Following Huang et al. we utilize the open test cases to calculate the efficiency metrics during the self-optimization process, while private test cases provided by EffßBnch were used for the final result evaluation. For HumanEval and MBPP datasets, we set the test cases provided by HumanEval and MBPP as open test cases, while test cases provided by EvalPlus (35] (i.e., HumanEval-Plun MBPP-Plus) as private test cases that were used to calculate the final results.
4.2 Implementation Details
All of the experiments are conducted in edge server with an Intel Xeon Platinum 8336C CPU with 128 cores, and * NVIDIA A100-SXM GPUs Total memory capacity of 2.0TiB.
Models We evaluate EFAR-LEARNER'S effectiveness on both open-source and closed-source LLM 4 For open-source models, we evaluate
with OpenCodeeIneepretter (1.3B, 6.7B, and (1.3B, 6.7B, and 33B) 224, CodeLlama (7B, 13B, 34B, and 70B) [55], XwinCoder (7B and 34B) 588 StarCoder (3B, 7B, and 15B) B2, and WizardCoder- 13B (38) where the detailed versions of LLMs are demonstrated in supplementary file. For closed-soure models, we
33B) 722), DeepSeek-Insteee
'Some I.LMs may not generate correct initial code, we do not report the efficiency results for WWe provide results for more LLMs in Table 5 in Appendix.
5

---

Table I: Code efficiency of LLMs with EFFI-LEARNER on EffiBench. The percentage in the brackets indicates the extent of the reduction for each respective item. Top performing LLMs are highlighted.
Model
ET
NET
MU (Mb)
NMU
TMU (Mb*s)
NTMU
Open-source LLMs
1.60
1,52
38.91
1,00
89.16
1.11
OpenCCodkineerreeter-...88
1.29 (19.44)
1.23 (19.198)
(0.0%)
(0.0%)
(20.8%5
0.88 (20.718)
OpeenCodeenneepppeeer-...
0.34
2.41
36.82
1,00
13.36
1,56
0.28 (17.66))
1.91 (20.7%)
(-4.8%)
(0.0%)
14.16 (-6.0%)
1.44 (7.7%)
0.29
2.10
35.48
1,00
13.06
1,93
OpenCooddmccrpeetr-338
0.28 (3.4%)
2.00 (4,89%
36.30 (-2.3%)
1,00 (0.0%)
11.54 (11.6%)
1.64 (15.098)
1.42
1.32
36.04
1.00
40.61
1.12
DoepSeck-L.3B-Img
1.15 (19.06)
1.07 (18,9%)
36.04 (0.0%)
1.00 (0.0%)
35.48 (12.6%)
0.98 (12.5%)
0.37
2.60
259.73
7.25
555.18
67,70
DoopSeck-6.7B--ng
0.34 (8.1%)
2.37 (8.8%)
36.97 (85,8%)
1,00 (86.2%)
13.66 (97.59))
1.46 (97,8%)
DeepSeek Ins
0.29
2.21
34.53
1,06
14.44
2.91
0.25 (13.85)
1.84 (16.79)
32.67 (5,4%)
0.99 (6.6%)
8.15 (43.6%))
1.55 (46.79))
CodeLLlama
4.70
3.68
46.76
0,99
212,41
1,93
4.52 (3.8%)
3.54 (3.8%)
38.67 (17.3%)
0.82 (17.2%)
157.76 (25.7%)
1.43 (25.9%)
CodeLLamm-1138
2.45
2.19
42.46
0,93
137.40
1,51
2.28 (6.9%)
2.04 (6.8%%)
42.12 (0.8%)
0.93 (0.0%)
119.36 (13.11%)
1.31 (13.218)
1.05
7.75
57.57
1,70
94.79
15:65
CodeLllmma-34)
1.02 (2.9%)
7.34 (5,3%)
40.62 (29.4%)
1.11 (34.7%)
52,12 (45.0%
7.02 (55.
3.93
109.61
3,57
203.92
54.15
CodeLlamm-770
0.47 (9.6%)
3.84 (2,3%)
26.42 (75.9%)
1.00 (72.0%)
14,53 (92.9%)
6.52 (88.0%)
XwinCnder-7R
2,80
2.81
55.54
1.52
208.23
3.47
2.43 (13.25%
2.44 (13.29)
49. 10 (11.61)
1.34 (11.8%)
158.20 (24.0%)
2.64 (23.99)
XwinCoder:34B
0.77
5.68
49,77
1.49
61.36
12
0.69 (10.4%)
(10.0%)
52.12 (-4.7%)
(5.7%%
9.92 (18.11%)
StarCoder2-3B
1.10
1,25
24.31
1,00
17.47
1,19
1.02 (7.3%)
1.15 (8.0%)
24.28 (0.1%)
1,00 (0.0%)
16.38 (6.2%)
1.12 (5.9%8
3.69
5,34
26.42
1.08
82.38
7.62
StarCoda2--BB
2.99 (19.0%)
4.32 (19.1%)
26.40 (0.1%)
1.08 (0.0%)
68.61 (16.7%)
6.35 (16.7%)
7.58
1.00
10.88
StarCoder2-115 15B
0.12 (87.1%)
1,03 (86.4%)
27.67 (.5.0%)
1.01 (-1.0%)
2.03 (91).8 (
1.06 (90.3%)
WizardCoder-133B
3,43
86,72
1.35
324.85
1.92
2.93 (14.6%)
1.80 (14.7%)
71.02 (18.1%)
1.11 (17.8%)
219.69 (32,49)
1.30 (32.3%)
Closed-source LLMs
0.36
2.50
91.25
2.45
157.50
19.75
GPT-3.5-Turbo-000
0.28 (22.2%)
2.01 (19.6%)
36.08 (60.5%)
0.99 (59.6%)
12.43 (92.1%)
1.64 (91.7%)
GPT-3.5-Turbo-1100 106
0.28
1.96
36.12
1.01
12.79
1.73
0.26
1,90
(5.8%)
(1.0%)
(10.8%)
1.62
GPT-4-Turbo-Pooviwww
0.27
1,96
33,94
1,00
11.82
1.89
0.25 (7.4%)
1.88 (4.1%%
33.17 (2.3%)
1.00 (0.0%)
10.18 (13,9%)
1.76 (6.9%)
GPT-4
0.31
2.19
80,88
2.26
129.91
17.90
0.28 (9.7%)
2.06 (5.9%)
63.82 (21.1%)
1.83 (19.0%)
80.74 (37 8%)
11.86 (33.7%)
48.33
1.30
6.73
Clande-3-Haiku
0.33 (8.3%)
2.30 (8.4%)
37.37 (22.7%)
1.03 (20.8%)
17.18 (67.4%)
2.37 (64.8%)
0.42
2.90
60.46
1.62
82.52
10.32
Mlaude-3-Sonnet
0.35 (16.7%)
2.47 (14.8%)
42.31 (30.0%)
1.17 (27.818)
28.95 (64.9%)
3.76 (62.8%)
use GPT-3.5-Turbo, GPT-4-Turbo, GPT-4 (50), Claude-3-haiku, and Claude-3-sonnee These L.MSs have achieved competitive pass@ scores in various code generation tasks [TT 35T.
Setup We first collect the generated code from each LLM and evaluate its correctness using open test cases. Only the code that passes all test cases is considered for efficiency evaluation. This approach ensures consistency in the evaluated tasks across different self-optimization iterations, as EFFI-LEARNER focuses on improving the efficiency of initially correct code without altering its pass@1 I score. By evaluating diverse set of open-source and closed-soure LLMs, we aim to provide a comprehensive assessment of the efficiency of EFFI-LEARRER in improving code efficiency across different models and architectures.
-enerated code and the effectiveness
4.3 Main Results
Open-source LLMs As shown in Table | we observe that the efficiency metrics for all models have been increased in most experiments once we apply EFFI-LEARNER to optimize the efficiency of LLM-generated code. For example, in OpenCoddlnterpreterr 1.3B, the execution time for its generated code decreases from 1.60 (s) to 1.29 (s), reduction of 19.4% in execution time. Additionally, the
'ee do not include Claude-3-opil in our experiments due limited resources.

---

Table 2: Effect of the number of self-optimization steps in EFFI-LEARNER.
Steps
RT (6)
NET
MU (Mb)
NMU
TMU (Mb*s)
NTMU
CodeLlama-700
0
0.52 0:48 (7.7%)
3.93
109.61
3.57
203.92
54.15
1
3.94 30))
26.47 (75.9%)
1.00 (72.0%)
14,91 (92.7%)
6.69 (87 6%%)
0.48 (7.7%)
3.89 (1.0%)
26.47 (75.9%)
6.60 (87.89)
3
0.47 (9.6%)
3.85 (2.0%)
26.42 (75 9%)
1.00 (72.0%)
14,60 (92.8%)
6.56 (87.9%)
(9.6%)
3.84 (2.3%)
26.42 (5.9%
(72.0%)
14.54 (92 (2.9%)
6.53 (87 9%)
5
0.47 (9.6%)
3.84 (2.3%%
26.42 (75.9%)
1.00 (72.0%)
14.53 (92.9%)
6.52 (88.098
GPT--.5-Turbo-0301
0
0.36
2.50
91.25
2.45
157.50
19.75
1
0.33 (8.3%)
2.35 (6.0%)
36.09 (60.4%)
(59.6%)
13.70 (91.3%)
1.81 (90.8%)
2
0.31 (13.9%)
2.18 )
36.09 (60 4%%)
0.99 (59.69)
13.04 (91.7%)
1.72 (91.3%)
3
0.29 (19.4%)
2.06 6%)
36.08 (60.5%)
(5%
12:57 (220099
1.66 (91.6%)
4
0.29 (19.4%)
2,03 (18.8%)
36.08 (05)
0.99 (59.6%)
12.50 (92.19)
1.65 (91.6%)
0.28 (22.25)
2.01 (19.6%)
36.08 (60.5%)
(59.6%)
12.43 (92.
1.64 (91.7%)
TMU of OpenCodeeInteppreeers 1.3B decreases from 89.16 (Mb*s) to 70.63 (Mb*s). Furthermore, in certain edge cases, EFFI-LEAARER significantly enhances efficiency. For example, the ET of 0.12 (s) and the NET also decreases from 7.48 to 1.03, reducing execution time requirements by 87. 97.1% compared to the initial code. The MU and NMU of DeepSeek-66. 7B-In also decrease from 259.73 (Mb) and 7.25 to 36.97 (Mb) and 1.06, reducing the maximum memory consumption by 85.8% for the code execution requirement. Moreover, we can also observe that the TMU and NTMU of StarCoder2-15B also decrease from 22.02 (Mb*s) and 10.88 to 2.03 (Mb*s) and 1.06, which decreases 90.8% memory consumption during the execution process. These results demonstrate the effectiveness EFFI-LEARNER in optimizing the code generated by open-source LLMs.
2arCooer2-15B decreases from 0.93
Closed-source LLMs Similar to open-source LLMs, we observe that the efficiency metrics for most closed-source I.LMs have been improved after applying EFFI-L_EARN FEARNER to optimize the efficiency of the generated code. For instance, the execution time for code generated by GPT-3.5-Turbo
decreases from 0.36 (s) to 0.28 (s), reducing the execution time by 22.2%. The MU and NMU of GPT-3.5-Turbo--23
also decrease from 91.25 (Mb) and 2.45 36.08 (Mb) and 0.99, respectively, which reduces the max memory consumption for code execution by 60.5%. Furthermore, the TMU and NTMU of GPT-3.5-Turbo-O25 decrease from 157.50 (Mb*s) and 19.75 to 12.43 (Mb*s) and 1.64, respectively, decreasing memory consumption during the execution process by 92.1%.
The improvements in efficiency metrics across both open-source and closed-source LLMs highlight the generalizability and adaptability of EFFI-LEARNER. By iteratively refining the generated code based on efficiency profiler feedhack, EFFI-LEARNEE without compromising the correctness of the generated solutions. The consistent improvements across various models and architectures demonstrate the potential of EIFI-LEARNEE as a model-agnostic approach for optimizing the efficiency of LLM-generated code in real-world applications.
enables L.MMs to produce more efficient code
4.4 Impact of Self-Optimization Steps
To investigate the impact of the number of self-optimization steps the efficiency of the EFFI- LEARNER-OPtimizze code, we conduct an ablation study by varying the number of steps from 0 to 5. Table 2 for Came--la and GPT-3.5-Turbo-0300 at different self-optimization steps.
CodeLlama-708F We can observe that after the first self-optimization step, the MU decreases from 109.61 (Mb) to 26.47 (Mb), reducing 75.9% maximum memory requirement compared with the initial code generated by CodeLlama- 70B. Similarly, the TMU decreases from 54.15 (Mb*s) to 6.69 (Mb*s), reducing 87.6% of memory consumption during code execution. As the number of steps increases, the efficiency metrics gradually improve. By the fifth step, the ET reaches 0.47 (s), reducing the 1.9% execution time requirement compared with the first-step generated code, and the TMU settles at 14.53, reducing 0.2% total memory usage from the first step.
GPT-3.5-Turbo--00 Similar to CodeLlama-70B: the MU decreases from 91.25 (Mb) to 36.09 (Mb) after the first self-optimization step, reducing 60.4% maximum memory requirement compared with the initial code. The TMU also shows substantial reduction from 157.50 (Mb*s) to 13.70 (Mb*s),

---

Table 3: Contribution of different components in EFFI-LEARNER. We evaluate how different feedback profilers affect the efficiency of LLM-generated code. Unsupervised self-refine only requires LLMs to optimize the efficiency of the code. Result-Aware Self-Refine feedback the ET, MU, and TMU to the LLMs and require it to improve the efficiency. Memory Profiler and Execution Time Profiler feedback the memory profiler and execution time profiler to the LLMs and then LLMs can based on the profile optimize the efficiency of the code.
Optinizzzion Protlee
NET
MU
NMU
TMU (Mb*s)
NTMU
Codel 200
Initial Version
0.52
3.93
109.61
3.57
303.92
54.15
Unsupevised Refine
(-5.9%)
6.87 (-74.8%)
279.41 (-154.9%)
10.58
126)
Ausul Self-Refine
0.79 (-51.99)
6.87 (74)8%
282.57 (-157
10.70
(270,01
605.20 (-1017.888)
Memory Proffler
0.53
4.34 (-104)))))
26.35 (75.995)
0.99 (72.54)
15.77 (92. 39)
7.06 (87.099)
Executio Time Profiler
(1.95)
4.17 19)
26.44 (75.9%)
1,00 (2.0%)
15.53 (92.49)
6.97 (87.15)
0.47
(75.3(775.9))
(72.0%)
14.53 (92.99)
GPP--3--TTbb--000
Initial Version
91.25
157,50
19.75
Unsupervised Refine
0.32(11.118)
(1.6%6)
(13.51)))
(-98.16))
(-14.88))
Result- Aware Self Retine
(16.7)))
2:25 (10.0%)
58.65 75))
1.61 (34.3%%
195.49 (:24.19%)
2-
Memory Proftler
00.54 (5,6983
2.40 (4.056)
36.85 (39 098)
1,00 (59.256)
16.34 (89. 6/93
2.10 (89.848)
Esecution Time Profiler
0.33 (8.318)
2.34 (6.4%)
36.43 (60.19)
0.99 (39.888)
14.07 (91.115)
(90.815)
EIVI-LLARNNS
(22.222)))8)
(19.699)
36.08 (60.598)
0.99 (59.69)
12.43.092.15))
1.64 (91.79))
reducing 91.3% of memory consumption during code execution. As the number of steps increases, the efficiency metries continue to improve steadily. By the fifth step, the ET reaches 0.28 (s), reducing the 15.2% execution time requirement compared with the first-step generated code, and the TMU settles at 12.43 (Mb*s), reducing 9.3% total memory usage from the first step.
Table demonstrates the significant impact of the number of self-optimization steps on the efficiency of the ELmn-LEARNGR-Optmizze code. For both CodeLlama-70B and GPT-3.5-Turbo-000) the first self-optimization step yields the most substantial improvements in code efficiency. As the number of steps increases, the efficiency metrics continue to improve, albeit with diminishing returns. By the fifth step, the efficiency metrics reach their lowest values, demonstrating the effectiveness of SEFI-LEANNER 's iterative self-optimization approach in enhancing the efficiency of LLM-ggnerated code. The evaluation results highlight that the majority of efficiency improvements occur the first few steps, with subsequent steps contributing to further refinements of the optimized code.
4.5 Feedback of Overhead Profile
To show the effectiveness of the overhead profile in guiding LLMs to refine their generated code, we compare the performance of EFFI-LEARNER with two alternative approaches: Unsupervised Self-Refine and Result-Aware Self-Refine 4O (7). Unsupervised Self-Refine uses prompt that directly requires the L.MM refine the code without providing additional information. Result-Aware Self-Refine feeds the ET, MU, and TMU, then requires the LLM to refine the code based on these metrics. Table 3 presents the code efficiency metrics for CodeLlama-700 and GPT-3.5-Turbo-0300 using different code refinement approaches.
CodeLlama-700 Unsupervised Self-Refine and Result-Aware Self- result in significant in- creases in ET. memory usage (MU), and TMU compared to the initial version. Unsupervised Self-Refine increases ET by 51.9%, MU by 154.9%, and TMU by 518.8%, while Result-Aware Self- Refine increases ET by 51 .%,, MU by 157.8%, and TMU by 523.2%. In contrast, EFFI-LEARMY incorporates the overhead profile feedback and achieves a 9.6% reduction in ET, 75.9% reduction in MU, and a 92.9% reduction in TMU compared to the initial version.
Unsupervised Self-Refine and Result-Aware Self-Refine show some improve- ments in ET and MU compared to the initial version. Unsupervised Self-Refine reduces ET by 11.1% and MU by 14.1%, while Result-Aware Relf-Refine reduces ET by 16.7% and MU by 35.7%. However, both approaches lead to substantial increases in TMU, with Unsupervised Self-Refin increasing TMU by 98. 7% and Result-Aware Self-Refine increasing TMU by 24.1% 1%. On the other hand, EFFI- LEARNER achieves a 22.2% reduction in ET, a 60.5% reduction in MU, and a 92.1% % reduction in TMU compared the initial version.
GPT-3.5-Turbo-UU-
These results highlight the importance of the overhead profile feedback in guiding L.MS to generate more efficient code. Without the overhead profile, the code refinement process using alternative
8

---

Table 4: Results of EFFI-LEARNER HumanEval dataset, where evaluate CodeLLama family generated code's efficiency. Full results are listed in Appendix Table 8
Steps
ET (s)
NET
MU (Vb)
NMU
TMU (Mb*s)
NTMU
CodeLllma-7b
0.71
$7.39
0.91
0.18 (10.0%)
0.63 (11.39)
57.07 (0.6%)
0.91 (0.0%)
6.18 (12.7%)
0.61 (12.9%)
CodeLlama-Iae 13b
0.23
0.95
58.13
0.96
7.97
0.94
0.20 (13.0%)
0.80 (15.8%)
58.03 (0.2%)
0.96 (0.0%)
6.64 (16.7%))
0.79 (16.0%)
Codel
0.24
0.95
61.79
1,01
8:45
0.96
(14.7%)
(1.05)
(17.338)
Codee-lama-700
0,21
0.93
60.19
1,01
6.76
1,01
(14.3%))
0.79 (15.1%)
59.49 (1.2%)
(1.0%)
5.75 (14.9%)
(14,9%)
prompts fails to improve code efficiency and even leads to significant performance degradation. The overhead profile provides valuable insights into the resource consumption of the generated code, enabling LLMs to make targeted optimizations and achieve substantial efficiency improvements.
4.6
Discussion
Table 5: Code efficiency of widely-studied LLMs reported by EFHI-EERRNER
Model
(s)
NET
MU (Mb)
NMU
TMU (MBrs)
NTMU
Phind-Cadie Jaama-44mPP 34B.22
0.52
3.28
157. 16.16
3.36
337,30
24.44
(23.
(23.5%)
(56.6%)
(56.8%)
(80.5%))
(80. (80.1%)
0.39
2,75
65,73
1.70
95.65
10.87
Ariigez-Codaa-DS-66788
0.32 (17.9%)
2.30 (16.4%%
59.00 (10.2%)
1.62 (4.7%)
79.67 (16.7%)
10.73 (1.3%)
Magicoder-S.
0.22
1.59
40.19
1.09
17.58
2.28
0.21 (4.5%)
1.50 (5.7%)
38.29 (4.7%)
1.07 (1.8%)
15.27 (13.1%)
2.22 (2.6%)
Mistral 7B codealpaca lora
2.36
1.45 (38.6%)
12.17 (33.9%)
27.45 (5.0%)
1.03 (-1.0%)
35.46 (38.8%)
17.28 (29. 1%6
CodeFuss--DeSSoek-3333
0,40
3.10
70.39
2,06
191,15
0.39 (2.5%)
3.01 (2.9%)
63.22 (10.2%)
1.85 (10.2%)
156.81 (18.0%)
26,42 (18.0%)
CodeLLama 34b-hf
2.08
15.68
46.41
1.26
128.46
17.87
1.95 (6.3%)
14:67 (6.4%)
46.40 (0.0%)
1.26 (0.0%)
125.22 (2.5%)
17.42 (2.5%)
spechless-ttrroote2--55
0.19
1.74
27.39
0.99
3,20
1.75
(31.6%)
(31.6%)
(0.0%)
(32.255)
(32.00%)
gpt-3.5ttrbb-0613
0,56
4.32
35,48
1.00
20.
3.00
0.49 (12.5%)
3.75 (13.2%)
35:47 (0.0%)
1.00 (0.0%)
17.84 (11.39))
2.66 (11.3%)
Generalizability across benchmarks In Table T we evaluated EFFI-LEARNER'' effectiveness on the EffiBench dataset. To illustrate EFFI-LEARNER generalizability in other datasets, conduct experiments on the HumanEval and MBPP datasets in Appendix Table and Table We also provide EFFI-LEARNER'S effectiveness the HumanEval dataset in CodeLlama models in Table 4 We can
Table 6: Pass@1 1 of LLMS generated initial code and EFFI-LEARNER optimized code.
Model
Initial Pass @
EFFI-LEARNER Pass @
OpanCodeennepreeee-----1..
5.8
5.4
OpeniCodeinteerreee---66--7-
13.6
13.2
OpenCoodennerppeeee-----33
24.7
24.4
deepseek- 1.33-IIss
4.8
4.5
deepseek-6.766 -.7b-IIns
7.2
7.0
beepseck-33b-11
10.0
9.9
CodeLlama-7b
7.0
7.0
9.7
9.6
Codel.lama-344
13.5
13.0
Codel
7.8
7.4
XwinCoder-133
10.5
10.2
XwinCoder-344
21.2
21.2
starcoder2-3b
1.6
1.2
starcoder2-7b
1.9
1.8
starcoder2-1 15b
0.7
0.4
WizardCoder- 13B
4.0
3.9

---

observe that the coding efficiency of CodeLlama and other LLMs (See Table B) also increases when we utilize EEH-LEARNER to optimize HLM--eeeratee code. For example, the ET of CodeLlama-700 decreases from 0.21 (s) to 0.18 (s), which reduces 14.3% execution time. As shown in Table B
and Table 2 results demonstrate that EFFI-LEARNER can consistently improve the efficiency of L..Mggenerated code for other datasets.
Generalizabilityy across LLMs In Table 1 we evaluate EFFI-LEARNER'S effectiveness on six types of open-source LLMs. To illustrate EFFI- LERNER'S generalizabilityy in other LLMs, we also conduct experiments on other LLMs in Table Our evaluation results demonstrate that EFFI-LEARNER can improve the efficiency of LLM-generater code for different LLMs. For example, the execution time decreases from 2.36 (s) to 1.45 (s), which reduces 38.6% execution time compared with the initial code. The total memory usage of Codd--CaaLLaaa34P vv2 also decreases from 337.30 (Mb*s) to 65.64 (Mb*s), which reduces 80.5% total memory requirement.
of Mistral- Bistral--B-coot cBBcooealpaaca-oono
Impact on correctness We provide the pass @ of LLM-ggenrrted initial code and LEARRRE optimized code for EffiBench in Table [ We observe that the pass @ I of EFFI-LEARNER opti- mized code may be lower than LLM-generated initial code. The key reason is that during the self-optimization process, EITI-LERRNER only uses public test cases guide code efficiency opti- mization for correct initial code. However, since public test cases may not cover all edge cases in the private test cases (test cases used to evaluate pass @ of LLMs), this can cause the pass@ I of EFFI LEARNEE generated code to be lower than the initial code. Nevertheless, we observe that the pass@ I of EPHI-LEARNER only decreases by about 0% to 0.5%, which means that only a few of the codes will be incorrect. As shown in Table 1 we can observe that the code efficiency is largely increased. We believe that this minor decrease pass@ 1 is worthwhile considering the significant efficiency gains.
Case study To illustrate how EFFI-LEARNER improves the efficiency of LLM-generated code, we provide a case illustration in Appendix Figure 4 Figure [1] As shown in Figure 6 we can observe that the execution time of the initial code is 23.59 (s) while in the self-optimized code, the execution time decreases from 23.59 (s) to 3.36 (s). The key reason is that in the initial code, the algorithm uses a standard unidirectional Breadth-Fins Search (BFS), which explores all possible states level by level starting from the initial state. This method results in a large number of states to explore, leading to significant computational overhead. In contrast, the self-optimized code employs a bidirectional BFS, which simultaneously searches from both the initial state and the target state. This reduces the search space by meeting in the middle, significantly decreasing the number of states that need to be explored and thereby improving the execution time.
Error Analysis We also provide case illustration to explain why some code efficiency does not improve significantly when EFFI-LEARNER is applied to LLM-generated code. As shown in Appendix Figure 12 [8] we observe that the initial code only requires 0.0012 (s) to execute, while in the optimized code, the execution time is still 0.001 I (s). The key reason for this minimal improvement is that both implementations already operate with the same theoretical time complexity of Ollo(minn(m) n))). Given the problem's 's constraints and small input sizes, the actual runtime differences are overshadowed by the inherent efficiency of the binary search algorithm. Additionally, the overhead of function calls and Python runtime operations can further minimize the observed performance gains. Therefore, while the optimized code may offer clearer partition management and slight improvements, the overall efficiency remains largely unchanged due to the already optimized nature of the initial approach.
5
Conclusion
This paper focuses on the critical issue of efficiency in code generated by LLMs. While LLMs have shown impressive capabilities in code generation, their output often suffers from suboptimal efficiency, leading to slower execution and higher resource consumption. To tackle this challenge, we propose EFFI-LEARNER, novel oel-oottmizztiion framework that leverages execution overhead profiles to guide LLMs in improving code efficiency. Extensive experiments and analysis demonstrate that EFFI-LEARNER significantly enhances the efficiency LLM-ggnerraed code, achieving substantial reductions in execution time and memory usage. For future work, we would like to investigate the application of EFFI-LFAARER to other programming tasks and languages, as well as explore the potential benefits of incorporating domain-specific knowledge into the optimization process.
10

---

6 ACKNOWLEDGMENT
The work is supported in part by National Key R&D Program of China (20222D0160201), HK RGC RIF (R7030-22), HK ITF (GHP/169/00S2), Huawei Flagship Research Grant in 2023, HK RGC GRF (Ref: 17208223 & 17204424), and the HKU-CAS Joint Laboratory for Intelligent System Software.
References
[1] Ahmad, W. U.. Tushar, M. G. R., Chakraborty, S., and Chang, K. AVATAR: A parallel corpus for java-pyhoon program translation. In Rogers, A., Boyd-Graber, J. L.., and Okazaki, (eds.), Findings of the Association for Computational Linguisties: ACL 2023, Toronto, Canada, July 9- 14, 2023, pp. 2268 2281.. Association for Computational Linguisties, 2023. doi: 10. 10.6//1/V1/ 2023. FINDININDINGSSACO htt..///.......aaaa///////o////.
URL JtLttss::://doi 10.5:.000001111111//111
findings-
143
[2] Ahmed, and Devanbu, T. Few-shot training Ilms for project-specific code-summarization.
In 37th IEEF/ACM International Conference on Automated Software Engineeriny, ASE 2022, Rochester, MI, USA, October 10- 14, 2022, pp. 177:1-177:5. ACM, 2022. doi: 10. 1.411/55/53399
3559555, URL https://doi org/ 1144/355113339 5555//33555144333555555
[3] Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., Umapathi, L. K., Anderson, C. J., Zi, Y., Lamy- J., Schoelkopf, H., Troshin, S., Abulkhanov, D., Romero, M., Lappert, M., Toni, F. D., del Río, B. G., Liu, Q. Bose, S., Bhattacharyya, U., Zhuo, T. Y., Yu, L., Villegas, P., Zocca, M., Mangrulkar, S., Lansky. D., Nguyen, H., Contractor, D., Villa, L., Li, J., Bahdanau, D., Jernite, Y., Hughes, S., Fried, D., Guha, A., Vries, H., and von Werra, L. Santacoder: don't t reach for the stars! CoRR, abs/2301 03988, 2023. doi: 10.48550/ARXIV.200000 103988. URL https://doi org/10. 48550/ 48550/arXiv arXiv 2301 03988
[4] Anil, R., Borgeaud, S., Wu, Y., Alayraly J., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K. Silver, D., Petrov, S., Johnson, M., Antonoglou, I,, Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lillicrap. T. P., Lazaridou, A., Firat, O.,, Molloy, J., Isard, M., Barham, R.R Hennigan, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Doherty, R., Collins, E., Meyer, C., Rutherford, E., Moreira, E., Ayoub, K., Goel, M., Tucker, G., Piqueras, E., Krikun, M., Barr, I., Savinov, N., Danihelka, I., Roelofs, B., White, A., Andreassen, A., von Glehn, T., Yagati, L., Kazemi, M., Gonzalez, L., Khalman, M., Sygnowski, J., and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/ 21 2023. doi: 10.48550ARXIV.2312.11
URL https://d// org 10. 48550/ arxiv 2312 11805
[5] Anthropic. Introducing the next generation of claude, 2024. URL
anthropic
Son//ewws/claude-3-ee
[6] Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.07733 2021. URL hattps://arxijl www.//gg/////////e//..///ee/eeee abs/2108. 07732
[7] Boyd, E. M. and Fales, A. W. Reflective learning: Key to learning from experience. Journal of humanistic psychology, 23(2):99-117, 1983.
[8] Brown, T. B., Mann, B., Ryder, N., Subbiah, M.. Kaplan, J., Dhariwal, P., Neelakantan, A.. Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Vosss A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Newral Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httts::tts://. p/roceeedg////rooeodigi.gg. http:////...///pp///////eeeaaaa
1457 htt.:////eeoooe0001/111111eaaae.
Awteoeaaaa-atttttttcm html

---

[9] Capra, E., Francalanci, C., and Slaughter, S. A. Is software "green"? application development environments and energy efficiency in open source applications. Information and Software Technology, 54(1) (4-1)660-71, 2012.
[10] Chaudhary, S. Code alpa: An cnttruction-follovviii llama model for code generation. https //github. htttshub.cooaaee sahi www.111111111////////11111a111e1 catpe//aooaappaaaaaaaaaaaaa 2023.
[11] Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y. Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, I.., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J.. Babuschkin, I., Balaji. S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, L., and Zaremba, W. Evaluating large language models trained on code. CoRR, abx/2107.03374 2021. URL httph:httassiiiiiii org/ abs 2107 03374
[12] Chen, X., Lin, M. Schärli, N., and Zhou, D. Teaching large language models to self-debug. CoRR, abs/2304.05128, 2023. doi: 1046550/ARXIV23300055122 URL https:///ooi. org/
10. 1.8550/arXiv..o wwtpexxiiv......00 2304.00124 05128
[13] Coignion, T., Quinton, C., and Rouvoy, R. åå. In Proceedings of the 28th International Conference Evaluation and Assessment in Software Engineering, pp. 79-89, 2024.
[14] Dai, J., Lu, J., Feng, Y., Ruan, R., Cheng, M., Tan, H., and Guo, Z. MHPP: exploring the capabilitiess and limitations of language models beyond basic code generation. CoRR. abs/2405.1 1430, 2024. doi: 10.48550/ARXIV.24000. 1430. URL https://aoi.org: org/10
48550/ 20550//rXiv.. www...ii............ 11430
[15] DeepSeckAL. Deepseek coder: Let code write itself, 2023. Ut//atpss//aappaeccoll
github.io/ /
[16] Deng, Y., Xia, C. Yang, C., Zhang, S. D., Yang. S., and Zhang. L. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. CoRR, htt/23344020144 2023. doi: 10.48550/ARXIV..300.4 02014. URL. httpt::////ot org/10. 48000/885///// /ar550//rxiv../.. 2304.02014
[17] Du, M., Luu, A. T., Ji, B., and Ng. S.-K. Mercury: An efficiency benchmark for Ilm code synthesis. arXiv preprint M4X2::4402007885 2024.
[18] Fan, A., Gokkaya. B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., and Zhang, J. M. Large language models for software engineering: Survey and open problems. In 2023 IEEF/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FOSE), pp. 31-53. IEEE, 2023.
[19] Fried, D., Aghajanyan. A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L.,, and Lewis, M. Incoder: generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OeenReview.net, 2023. URL htttthtst:///ereetteeee
net. net/pdf http:/dfddddddpd weechhub-lbM66.
[20] Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V. Y., Lao, N., Lee, H., Juan, D., and Guu, K. RARR: researching and revising what language models say. using language models. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 16477-16508. Association for Computational Linguistics, 2023. doi:
1/533//1/22233CLLLLOO URL httpt::///o....... org/ 10. 18653/v1/2000 acl long..n..... 910
[21] Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M. J., Thacker, P., Camppell-GGllinghann L., Uesato, J., Huang. P., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig. R., Chen, C.. Fritz, D., Elias, S., Green, R., Mokrá, S., Fernando, N., Wu, B., Foley, R., Young, S., Gabriel, I., Isaac, W., Mellor, J., Hassabis, D., Kavukcuoglu, K., Hendricks, L. A., and Irving, G. Improving alignment of dialogue agents via targeted human judgements. CoRR, abs/2209, 14375, 2022. doi: 10.48550/ARXIV222009 14375 14375. URL https://://aad org/ org/10...///./ 48550/ 148855//rXiii arivv 2209 14375
12

---

[22] Gou, Z., Shao, Z., Gong, Y. Shen, Y. Yang. Y., Duan, N., and Chen. W. CRITIC: large language models can self-correct with tool-interactive critiquing. CoRR, abs/2305.11738, 2023. doi: 10.48550/AXXIV220005 11738. httpttttt///o...ooo
201500/aXiiv.2000 11738
[23] Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. Giorno, A. D., Gopi, S., Javaheripi. M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need. CoRR, abs/2306. 1644. URL https://:///o org/ 48550/ art.
2023, doi: 10.48550/ARXIV.2335 11644
[24] Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., al. Deepseek-coder: When the large language model meets programming the rise of code intelligence. arXiv preprini arXiv:2401. 14196, 2024.
[25] Haque, M. M. A., Ahmad, W. U., Lourentzou, I., and Brown, C. Fixeval: Execution-based evaluation of program fixes for competitive programming problems. CoRR. abs/2206.07796 URL BRL/https://dol www.oo..////////../e////// 48550/arXiy 2206
2022. doi: (044855//RXXV.2206.07777 07796
[26] Hasan, M., Muttaqueen, T., Ishtiaq, A. A., Mehrab, K. S., Haque, M. M. A., Hasan, T., Ahmad, W. U., lqbal, A., and Shahriyar, R. Codesc: A large code-description parallel dataset. In Zong. C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguisties: ACI/IICNIP 2021, Online Event, August 1-6, 2021, volume ACL/UJCNLP 2021 of Findings of
ACL, pp. 210-218. Association for Computational Linguistics, 2021. doi: 10. 10.18653/V1/2221 FINDINGS ACL. 18. URL httthttp:://doi org/10..1110 wwwpg//...1665///1 w///////2221121// indings www.nnggs-cl......
[27] Huang, D., Zhang, J. M.. Qing, Y., and Cui, H. Effibench: Benchmarking the efficiency of automatically generated code. arXiv preprint arXiv:2402.02037 2024.
[28] Jiang, N., Liu, K., Lutellier, T., and Tan, L. Impact of code language models on automated program repair. In 45th IEEE/ACM International Conference Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 1430-1442. IEEE, 2023, doi: 10. 109/ 00123300125
ICSE48619 033E48619220230000 2023.0015.. URL.ttps:://e/ http:///o..or//////////////
[29] Jiang, S., Wang. Y. and Wang. Y. Selfevolve: A code evolution framework via large language models. CoRR, abs/2306. 000/2206.02907, 2023. doi: 10.48550/ARXIV22006.000
URL https
aadoi.or//il /8550/ arxiv. 2306 2306.02907
[30] Kreutzer, J., Khadivi, S., Matusov, E., and Riezler, S. Can neural machine translation be improved with user feedback? Bangalore, S., Chu-Carroll, J., and Li, Y. (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 3 (Industry Papers), pp. 92-105. Association for Computational Linguistics, 2018 doi: 10.18653/V 3012. URL https: fttt::::/ddo.oreee wwt......./.......... 18653/v1/ n18 3012
[31] Lemieux, C., Inala, J. P., Lahiri, S. K., and Sen, S. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 919-931 IEEE, 2023. doi: 10. 109/1CC9/4 900.9222230000 URL httphttts/://o org/ 10 1109/ICSE4881
2023300083
[32]
Li, R., Allal, I.. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J.. Chim, J., Liu, Q.. Zheltonozhskii, E.,, Zhuo, T. Y.. Wang, T., Dehaene, O., Davaadorj. M.. Lami-Poirier, J., Monteiro. J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee, M., Umapathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M., Wang, Z., V, R. M.. Stillerman, J., Patel, S. S., Abulkhanov, D., Zocca, M., Dey. M., Zhang, Z., Moustafa-Fahmy, N., Bhattacharyya, U., Yu, W., Singh, S., Luccioni, S. Villegas, P., Kunakov, M., Zhdanov, F., Romero, M., Lee, T.. Timor, N., Ding, J., Schlesinger, C., Schoelkopf, H., Ebert, J., Dao, T., Mishra, M., Gu, A., Robinson, J., Anderson, C. J., Dolan-Gavit, B., Contractor, D., Reddy, S., Fried, D., Bahdanau, D., Jernite, Y., Ferrandis, C. M., Hughes, S., Wolf, T., Guha, A., von Werra, L., and Vries, H. Stareoder: may the source be with you! CORR, abs/2305.06111 06161,, 2023. doi: 0..48550/ARII.22000
URL htRpst :/op:::://ooi.o org/ 100/0. 10. 48550/ arviv. 2305 06161
13

---

[33] Li, Y., Choi, D. H., Chung, J., Kushman, N., Schrittwieser, J., Leblond. R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., Hubert, T., Choy, P., de Masson d' Autume, C., Babuschkin, I., Chen, X., Huang. P., Welbi, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Robson, E. S., Kohli, P., de Freitas,, N., Kavukcuoglu, K., and Vinyals, O. Competition-level code generation with alphacode. CoRR, abs/2222.02888844444 2022. doi: 10. 10448550/ARRXI.2200..077 hotLhhtps://// http:////...l.lgg100004666////// /rrp50///xii... 2203. 07814
[34] Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar. S., and Lee, Y. T. Textbooks are all you need II: phi- 1.5 technical report. CoRR, abs/2309,05563, 2023. doi: (0.48550/ARXIV23300 05463. URL httpp:///ooi. http:///oooorr////// 10. 48550/8055333 2309 05463
[35] Liu, J., Xia, C. Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rig- orous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
[36] Lu, J., Dou, Z., Wang. H., Cao, Z., Dai, J., Wan, Y., Huang, Y., and Guo, Z. Autocv: Empowering reasoning with automated process labeling via confidence variation. CoRR, abs/2405. 2024. doi: 10.48550/ARXIV.4005. 16002. https//s://ddi org/ 10. 48550/ar5iiy 2405 16802
2405
[37] Lu, J. Liu, Z., Wan, Y., Huang, Y., Wang. H., Yang, Z.,, Tang, J., and Guo, Process-driven autoformalization in lean 4. CoRR, abs/2406.01900, 2024. doi: 10.4850/ARXIV.240600122 URL htthht::: http://ooo..oo//00048550//////1
2406. 01940
[38] Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang. D. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306..0 08568. 2023.
[39] Luo, Z., Xu, C., Zhao, Sun, Q., Geng, X., Hu, W., Tao, C.,, Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: htttp:://openreviww net/forum iid=ind=dnnnn-- gK5W
[40] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L.., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-refine: Iterative refinement with self-feedback. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http::/ppppps.....t.m//// nips cc/ wappppppr.... talles/paperr (etpper/223///e 22//2/2222hhh/// Abstract- Conference html
[41] Mancebo, J., Garcia, F., and Calero, C. A process for analysing the energy efficiency of software. Information and Software Technology. 134:106560, 2021.
[42] Meta. Introducing meta llama 3: The most capable openly available I1m to date, 2024. URI https://ai. meta. cow wwt.tooooo////...oo..o..oo.ooooo meta- mesta.con/blog/mmta-nnaa 11ama-3/
[43]
Metcalfe, J. Learning from errors. Anmual review of psychology, 68:465 4894654-49..0 2017.
[44]
Microsoft.
The world's most widely adopted ai developer tool., 2024. URL
com htt///eaturss/coplloot
[45] Mir, A. M., Latoskinas, E.,, Proksch, S., and Gousios, G. Type4py: Practical deep similarity learning- based type inference for python. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh. PA. USA, May 25-27, 2022, pp. 2241-2252, ACM, 2022. doi: 0..1145//51000333510111 URL https: //dotps://do... org/ 10. 1111 1145/3510003 3510124
[46] Muennighoff, N., Liu, Q., Zebaze, A. R., Zheng, Q., Hui, B., Zhuo T. Y., Singh, S., Tang, X., von Werra, L., and Longpre, S. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representutions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. (t /ttp:////peeeiiieeee wtwpiiiwwwweteeeeeeee iid=m (1=SWIPWN
14

---

[47] Nijkamp, E.. Pang. B., Hayashi, H., Tu, L., Wang. H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. UKI.httpss//ppenrreiiek wttprrvviiiieeeeeed........ 1aYcJKpY2B_
[48] Niu, C., Zhang, T., Li, C., Luo, B., and Ng. V. On evaluating the efficiency of source code generated by Ilms. arXiv preprint arXiv:2404.06041, 2024.
[49] OpenAl. GPT-3.5 Turbo, 2023.
URL
gpt-3-5
[50] OpenAL. GPT-4 Technical Report. CoRR, abs/2303.08774, 2023. doi: 10.48550/a2Xiv23333. 0000000002...20300..77777
08774. URL
[51] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instruc- tions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, Novem- ber 28 December 9, 2022, 2022. URL /ttt:::/ppppraa.e www.pperrs..pp....mmm wwwppppppppppppppppppp willsspppppl html
[52] Pereira, R., Couto, M., Ribeiro, F., Rua, R., Cunha, Fernandes, P., and Saraiva, Ranking programming languages by energy efficiency. Science of Computer Programming, 205:102609, 2021.
[53] Qiu, R., Zeng, W. W., Tong, H., Ezick, J., and Lott, C. How efficient is Ilm-generated code? a rigorous & high-standard benchmark. arXiv preprint arXiv:240ks 240,0:6700606647, 2024.
[54] Rozière, B., Lachaux, M., Chanussot, L.. and Lample. G. Unsupervised translation of programming languages. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-
12. 2020, virtual, 2020. URL https: httttp::/pproceedinggs httpooceeddinnnnnnurnissee. www.nnew..ccpppppeeeee http:/ppppr///// h020/hash// htw.ellffoo5c0od--btttrccte Abstract hhtm
[55] Rozière, B., Gehring. J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Canton-Ferree C.,, Grattafiori, A.,
Xiong. W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N., Scialom, T., and Synnaeve, G. Code Ilama: Open foundation models for code. CoRR, abs/2308. 12950. 2023. doi: 10.48550/ARXIV.22000 12950. https:ts::///a org/ 10. 48550/ arrvv./rrvvvv.. 2308 12950
[56] Shi, J., Yang, 7., and Lo, D. Efficient and green large language models for software engineering: Vision and the road ahead, arXiv preprint arXiv:2404.045666 2024.
[57] Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: language agents with verbal reinforcement learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:pp:///ppppprrrrr htt.o//ooooooooaaa.a/o//o/ooaaaa
nips wwwpppprccppppprprppp httpp/ppeerrfleaeppppeeeeeeeeepe 2023 /tt. htaa:///saaaaaaaeeeeeeeeeeeeeee0
[58] Team, X.-L. Xwin-lm, 2023. URL httLLtttpp///iithtb.ooom// ctt.:/bboooooooiiioiooeeeee
[59] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P.. Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi,
15

---

K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E.. Tang, B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, I.. Zhang, Y., Fan, A., Kambadur, M., Narang. S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288. 2023, doi: 10.48550/ARXV2233700022 URL https://doi. org/10. 48550 arxiv 2307 09288
[60] Viswanathan, V., Zhao, C., Bertsch, A., Wu, T., and Neubig, G. Prompt2model: Generating deployable models from natural language instructions. arXiv preprint arXiv:2308. /221/, 2023.
[61] Wang. Y., Wang, W., Joty, S. R., and Hoi, S. C. H. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. Moens, M., Huang, X., Specia, L., and Yih. S. W. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 8696-8708, Association for Computational Linguistics. 2021. doi: D0.18653/V1/0011 WWW.20...ENLM-MAIIN. 6855. URL URLLhttpss//// //doi. org/10 10. 18653/v1/ 2021 em mainn.ppmii.... 685
[62] Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with reff-generated instructions. In Rogers, A., Boyd- Graber, L.,, and Okazaki, N. (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pp. 13584-13508. Association for Computational Linguistics, 2023, doi: 10. 18653/111 2023.ACLLLONG.75 URL httph:ttt//:oiii. org/ wwtp:/.0..1665651 18653/v1/2000 acl- 754
[63] Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[64] Wei, J., Durrett, G., and Dillig, I. Typet5: Seq2seq type inference using static analysis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open 2023. httLhhtts////eerreeeee http:/rwewwiiienee.peeeee wwt.://n.een//dn/ii.neo 4TyNEhI2GaN]
[65] Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L. Magicoder: Empowering code generation with ass-instruct.. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL htttps://openreviiee forum?id=XUeddBitts:
[66] Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng. J., Tao, C., Lin, Q.. and Jiang, D. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. http:htttt:///peeee/
[67] Yao, Y., Wu, H., Guo, Z., Zhou, B., Gao, J., Luo, S., Hou, H., Fu, X., and Song, L. Learning from 2024. doi: correctness without prompting makes LLM efficient reasoner. CoRR, 2403. 19094 10.48550/ARXIV.4403.1 19094. URL http::: 4twt://....o........../..////.
[68] Yu, w., Zhang, Z., Liang, Z., Jiang, M., and Sabharwal, A. Improving language models via plug- pla- retrieval feedback. CoRR, wwtpa///...10000..... 2023, doi: 10.48550/ARXI.23055.000
URL https://d// org 10. 48550/80500110// 2305 14002
[69] Zhang. K., Li, Z., J., Li. G., and Jin, Z. Self-edit: Fault-aware code editor for code generation. In Rogers, A., Graberr, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 769-787. Association for Computational Linguistics, 2023. doi: 10.
URL https:///oi..rr//00 httpa//000r1111113/333333 18653/v1/
2023. ao13 45
[70] Zhang. K., Wang, D., Xia, J., Wang, W. Y., and Li, L. ALGO: synthesizing algorithmic programs with generated oracle verifiers. Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual
16

---

Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA. USA, December 10 16, 2023, 2023. URL URL h wwtp:perr.........o............. httppppperrileeeppppppeeee 2023/
/ httpa//aaa....eeeooooooaaaaaaaaa Conffrrrraaa--oeeeeenceee html
[71] Zhao, C., Jia, X., Viswanathan, V., Wu, T., and Neubig, G. Self-guide: Better task-specific instruction following via synthhtt finetuning. arXiv preprint arXiv:2407.12874, 2024.
[72] Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W. and Yue, X. Open- codeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14633 14658, 2024.
17

---

A Appendix
A.1 Limitations
EFFI-LEARNER presents compelling solution for enhancing the efficiency of code generated by LLMs. However, it's crucial acknowledge several potential limitations. Firstly, the multi-iteration self-optimization process at the heart of EFFI-LEARNER can be time-consuming, particularly when applied to intricate programming tasks. Yet, it's important to note that this investment of time can yield significant long-term benefits, as the optimized codes, once deployed, can considerably improve efficiency. Moreover, the process of feeding overhead profiles to LLMs to prompt code optimization may consume necessitates additional tokens. Lastly, the effectiveness EffiLe has been primarily evaluated on Python. Therefore, its performance in different programming languages or environments may vary, underscoring the need for further testing and validation this approach in a diverse range of contexts.
tokens. This is due to the fact that the length of the overhead profiles
A.2 Broader Impacts
Positive Societal Impacts The proposed method EFFI-LEARNER has the potential bring about several positive societal impacts. Primarily, it can significantly enhance productivity by enabling developers to complete tasks more quickly and efficiently, due to the faster execution times and lower memory and processing power consumption of the optimized code. This is particularly beneficial in resoure--onsstrained environments, such as mobile devices or embedded systems, where conserving resources is crucial. Moreover, EFFI-LEARNER' focus on improving code efficiency can lead to more cost-effective solutions, as lower resource consumption translates to reduced operational costs.
Negative Societal Impacts The increasing effectiveness of LLM-basee tools like EFFI-LEARNER could also have negative societal impacts. For instance, it could lead to an over-reliance on these systems, potentially resulting in a decline in human coding skills and lack of understanding of the underlying code. This could be problematic in situations where human intervention is necessary. Ad- ditionally, there is risk of job displacement in the coding and programming industry, the demand for human coders may decrease, particularly for routine coding tasks. Lastly, while EFFI- LEARNER can improve code efficiency, does not necessarily address potential security vulnerabilities or privacy issues in the code, which could have significant socictal impacts if not properly managed.
A.3 Prompt Template
Prompt Template
prompt gran
optinize efficiency folloving Python based task description, case, 4 and overhead analysis provided. Ensure the optimized code can pass the given test case.
Task Description:
taask_descripptinn
Test: Case: (test_case)
Original Code: ** python (complation))
Overhead Analysis: [overhead, joverhaa_prropp:
Figure 3: Prompt template used by EFFI-LFARNER in the self-optimization stage.
18

---

Table Evaluation results EFFI-LEARNER and baselines. Since the finetuned link for GPT-3.5- turbo from PIE is not available, we use the fine-tuned CodeLlama 7B for experiments. Due to the fine-tuned PIE Cada 7B does not have the same correct tasks as the original CodeI.lama, we then do not provide the initial version for the experiments.
OptimizationProfle
BTis
NET
(U(Mb)
NMU
TMU(MB's)
NTMU
GPT-3.5 T-rrbo--rro-000
IontialNension
0.36
2.50
91.25
2.45
157.50
19.35
UneagerissettaffRReee
2.46
312.99
Resah--AwareSee Retine
58.63
195.49
Self-Fdit
0.42
3.67
$9.86
1.65
24.87
3.28
Critic
0.60
4.25
102.75
2.82
351.91
46.39
2.88
Self-Refio:Fiicienny
2,83
2.65
Reeif-Refiie
0.40)
2.88
61.33
1.81
36.29
5.69
Self- Self-Reasoning
0.89
6.31
60.64
1.62
45.91
5.61
Sell-Rellection
(22.2%)
2.01 (19.6%)
36.00 (60.5%)
(3966%)
(92.1%%
(91.7%)
PIE+
PIE+ LEARNER +Zero Shot
0.79
5.41
65.78
1.68
39,90
7.84
PIE+ Shot
0.82
5.58
73.57
1.74
98.02
8.92
PIE+BeR LEARNER +Fow Shot
2.97
73.10
1.74
59,64
5.09
PIE+CUT
PlE+ LEARNER +COT
0.45
2.84
71.15
1.71
58,06
4.27
0.74
5.36
68.64
1.51
85.24
7.78
3.36
Supersonic
Supersonic
1.40
10.33
113.06
3.18
$29.59
56.24
Sipenonic-ERn--EeA
267.47
A.4 Comparison with Baselines
To demonstrate EFFI-LEARNER'S 's effectiveness with existing baselines and some prompt engineering methods that can refine the prompt from the correctness the efficiency of LLM--generted code.
Table 8: Evaluation results of EFFI- LEARNEERRNR'SSS effectiveness in the HumanEval dataset.
Steps
ET (s)
NET
MU (Mb)
NMU
TMU (Mb*s)
NTMU
0.20
0.86
$7.24
1,00
0.19 (5.0%)
081 (5.8%)
57.17 (0.19)
(.000.05)
6.20 (6.55)
0.79 (5.0%)
0.21
0.98
58.83
1,06
6.79
0.99
(0.0%)
(.060.00))
0.21
0.95
$9.90
1.05
0.21 (8.0%)
0.95 (2.1%)
59.93 (-0.19))
1,05 (0.09)
6.87 (2.69)
0.92 (2.1%)
deeporkk-nddrr1 3.biinstmet
0.23
0.90
62,80
1,00
7,85
0.87
(.000.))
deepseck cuder i.biinst
0.22
0.76
59.57
1,00
7,34
0.77
0.19 (13.68)
0.68 (10.58)
59.72 (-0.3))))))))
1.0000.05)
6.50 (10.25))
0.60 (1)
deepseek-cnder.. Tesender---bbiisst.. Sibinstrect
0.21
0.95
63.52
0.99
7.18
0.95
0.9900.0%)
(2.68)
Codel lf
0.20
0.71
7.08
0.70
0.18 (10.09)
0.63 (11.39)
$7.07 (0.69))
0.91 (0.05)
(118
0.61 (12.955)
Codel Ibs-nnsruct-hff
0.23
0.95
58.13
0.96
7.97
0.94
(13.05)
(0.255)
0.96(0.05)
6.64
Codel Jan.ma-a----tttmettdd
0.24
0.95
1.01
0.21 (12,58)
0.81 (14.78)
61.55 (0.4%)
1.00
6.99 (17.39))
0.80 7%)
Codee Luma-7Uh-tnumettar
0.21
0.93
60.19
1.01
6.76
1.01
(15.19)
(1.29))
(.00(1.0)
5.75 (14.9)))
XwinCuder-13B
0.27
1.08
61.14
1.04
9.25
1.09
0.25 (7.4%)
1.01 (6.5%)
1.04
8.63 (6.85)
1.02 (6.45)
XwinCuser-34dD
0.25
1.07
60.75
1.05
8.46
1.08
1.0500.05)
WizantCists-7R
0.21
0.91
58.50
1.01
0.18 (14131%)
0.79 (13.29)
57.97 (1.1%)
1.00(1.0))
5.79 (12.79))
(178x12.45)
WiomiCisder-338
0.21
0.81
60.50
1,00
7.22
0.79
(9.5%)
(10.1%)
0.22
0.79
58.13
1,00
7,10
0.78
0.17 (32.78)
0.62 (21.55)
58.42 (-0.5)
1.000..05
(23.11)))))
0.60 1%)
starcoder2333
0.24
1.02
62.45
1,00
7.75
1.0000.01
Stareoda227bb
0.21
0.89
62.53
1,00
0.19 (9,5%5
0.78 (12.0%)
62,85 (-0.5%
(.00
(13.6)3)
0.74 (12.9%)
19

---

Table 9: Evaluation results of EFFI- effectiveness in the MBPP dataset
Steps
(Mb)
TMU (MB*S)
OpenCodellnepprteeDS.....
59.01
11:73
0.25 (10.7%)
0.84 (10.698)
58,99 (0.055)
1.01 (0.05)
10.59 (9.758)
0.89 (9.25)
0.26
1.06
58.39
1.00
0.25
1.08
0.21
0.87
1.00
7.3.
0.83 (23.15))
UpenCodelnneprreeeeD......
58.22
20:19
0:31 (20.59)
1.14 (3A 353)
78 (0 (9))
1.00 (0.05)
13.22 (14,55))
1,22 (14.45)
despseck-onder-1 Sibiinsrtttttt
0.63
1.68
354.1
6.05
1463.46
89.12
0.62
1.64
339
(4.08)
(3.41))
86.11
despwock-owder-6. Thiinstrctt
0.76
58.44
39.11
5.69
0.21 (72.48))
0.99 (72) 9881
59:34
(0:2591
1.00 (0.098)
6.67 (822999))
0.97 (83301%)
derpsocl-cuder- weweoo---oo-------nneeee Nhbinstract
0.58
2.33
53.48
0.00
28.74
3.16
0.19
0.75
160
(007))
5.88
0.65 (79,45)
Codel Tb-Instruct-fff
2.04
56-86
13:26
0.42 (6.78)
1.89 (7 (7-48)
66.8800088
(0.39))
0,97
(0008)
11.98 (9.75))
1.62 59)
Codel Jama- Sh-Insimeet-af
0.53
2311
55.37
6000
21.75
2.34
(1.99)
2.04
(2-918)
2.28
Codel Juma- 34h-Instmee--ff
1418
69/80
1:11
84.01
5,47
0.41 (2.45)
1.13 (4.288)
69,32 (0.798)
1.19
(0108)
74.78
4.87 (1) (115))
Codel 700-Instruct-ll
0.23
1.06
13
0.98
7.65
1.05
0.20
0.93 (12.189)
8600 (0/07))
(12.88))
(17.35))
XWinCoder-7B
1.00
1.10
0.18 (31.7))
0.90 (19,)
58.44 (0.05)
1.00 (0.02)
(18.1)))
0.90 (18.28)
XwisColler-13R
0.50
1.96
58.38
1100
23.88
2,50
1.61
1.00 (0.0%)
(20.612
1.98 (20.85))
XwinCCoder34B
0.38
1.44
58.27
1.00
14,77
1.48
(7.99)
1.32 (8. (8))
58.22 (0. 198)
1.00 (0008)
13.54 559)
1,36 (8. 18)
WiiardCuderr Python-7B-V VLO-GT
0.22
1.06
58.44
0.99
7.19
1.03
0.20 (9.19)
0.93
(0.25)
0.99 (0.0%)
6.41
0.91 (11.7%)
57.74
30.66
0.59 (1.8%)
1.28 (5.25)
57.70
0.99 (0.073)
29.56 (3.61%)
1,38 (3.5%%
WirsrdCuder- SeerrlCCokr-Pthho-44-3 hww.eeooooooo---1-----000-
0.68
2.43
56.75
0.97
34,06
3.14
0.65
2.33
0.97 (0.0%)
3.01
suncuder2-3b
0.17
0.83
45.82
0.79
5.10
0.77
0.16 (5.99)
0.80 (1 65)
43,46 (5.2())
0.74 (6. 35)
4.89 (8.05)
(970 15)
surcoderr2 7b
1.72
8.63
25.61
0,44
40,42
1.72
(0.25)
0:44
6.19
1tancoder211hh
0.19
1.05
58.62
1.01
6:23
1.05
0,18 (5.35)
0.99 (5.75)
38.14(0.855)
1.00 (L.03)
5.92 (5.0%)
(1.8%)
A.5
Detailed Evaluation Metric for Efficiency
In our work, adopt the efficiency metrics proposed by EffiBench [27 to evaluate the effectiveness of EFFI-LEARNER improving the efficiency of LLM-generated code.
Execution Time (ET) Execution time (ET) measures the average time taken for code execution. Mathematically. ET is defined as:
ET =
ET Σ Toode
1 N
where ET is the execution time metric, Teode is the execution time of the code (with all the test cases), and N is the number of codes generated by code generation models used for evaluation.
Normalized Execution Time (NET) Normalized Execution Time (NET) measures the execution time required by generated code relative to that of a canonical solution. We define NET as:
NET
1
Σ
N
Teode
N
T Teanonical
where Taxl is the execution time of the generated code, and Teanonical is the execution time of the canonical solution. A NET value greater than I indicates that the generated code is slower than the canonical solution, while value less than suggests the generated code is faster.
Max Memory Usage (MU) Max Memory Usage (MU) measures the average max memory con- sumption during code execution. Mathematically, MU is defined as:
1 N N
MU =
Σ Meode
where MU is the memory usage metric, Mesde is the max memory consumption of the generated code among all the test cases, and N is the number of code instances generated by code generation models used for evaluation. This metric critical for assessing the resource efficiency of generated code, particularly in environments with limited maximum memory capacity.
20

---

Normalized Max Memory Usage (NMU) Normalized Max Memory Usage (NMU) quantifies how the max memory efficiency of the generated code compares to the canonical solution. We define NMU as:
1 N
Moode
NMU
Meanonical
where NMU is the normalized max memory usage metric, Monde is the max memory usage of the generated code, and Mennonical is the max memory usage of the canonical solution. An NMU value less than indicates that the generated code is more memory-efficiet than the canonical solution, whereas value greater than suggests it is less efficient in terms of memory usage. This metric provides relative measure of the memory optimization in the generated code in comparison to standard baseline.
Total Memory Usage (TMU) Total Memory Usage (TMU) assesses the efficiency of memory usage throughout the execution code, taking into account both the magnitude and duration of memory utilization. To calculate TMU, first, monitor and record the memory usage at discrete time intervals during the execution, resulting Then, compute the area under the curve of (t) over the total execution time, Twint. using numerical integration methods such as the trapezoidal rule:
memory usage profile (t). where represents time.
N TMU Σ
Tuai
M(I)d dt
lower TMU value indicates higher memory efficiency, reflecting an optimized balance between the amount of memory used and the duration of its usage.
Normalized Total Memory Usage (NTMU) The Normalized Total Memory Usage (NTMU) offers a comparison of the dynamic memory efficiency between the generated code and the canonical solution. To determine NTMU, calculate the TMU for both the generated code and the canonical solution. Normalize the TMU of the generated code by dividing it by the TMU of the canonical solution:
N TMU code NTMU = N Σ UMMnannnicall
TMU d'canonical
where code is the TMU of the generated code and TMU canonical is the TMU of the canonical solution. An NTMU value less than 1 signifies that the generated code manages dynamic memory more efficiently compared the canonical solution, while value greater than indicates less efficient management dynamic memory. This metric provides insight into the relative use of dynamic memory of generated code compared established benchmark.
A.6 Additional Related Work
Instruction Tuning for Code Instruction tuning has proven effective in enhancing the usability
and overall performance of LLMs across various language tasks [5] 63 60 (71). This approach has been extended to the domain of code generation. The core challenge the acquisition of high-quality instructional data, which is often labor-intensive. To address this, recent research has focused
developing methods to generate synthetic instruction data. Studies have shown that textbook-quail synthetic data alone can improve model's coding and reasoning capabilities 23 34 One early
effort was Self-Instruct [62], which utilized LLMS to generate synthetic instruction- response pairs using carefully crafted prompts. The same LLM was then instruction-tuned on this synthetic data. Code Alpaca (10) applied the Self-Instruct approach with GPT models, tailoring it specifically for code generation, editing, and optimization tasks. Building upon this, WizardCoder (39 adapted the Evol-Instruct technique [66] to the coding domain by designing heuristic prompts to create more complex and diverse synthetic data. OSS- (65) took a different approach by leveraging LLMs to automatically generate new coding problems inspired by random code snippets from open-source repositories. contrast, Octopack (46) focused on collecting and filtering high-quality Git commit messages that resemble natural language instructions.
21

---

Task Description
Problem: You have lock in front of you with circular wheels. Each wheel has 10 slots: 'O', 'I', '2', '3', '4', '5', '6', '7', '8", '9'. The wheels can rotate freely and wrap around: for example we can turn "9" be '0', or "0" be "9". Each move consists of turning one wheel one slot. The lock initially starts at '0000", string representing the state of the 4 wheels.
You are given list of deadends dead ends, meaning if the lock displays any of these codes, the wheels of the lock will stop turning and you will be unable to open it.
Given a target representing the value of the wheels that will unlock the lock, return the minimum total number of turns required to open the lock, or -1 if is impossible.
Example 1:
Input: deadends = 0"0201"
"""00""""""0""""""""""""""""111" "0101", "0100","0002""""""" 1212", "2002"], target = "0202"
Output:
Explanation: A sequence of valid moves would be "0000" - > "1000" - "1100"
"1200" > "1201" "1202" "0202". Note that sequence like "0000" "0001" " -> "0002" - "0102" > "0202" would be invalid, because the wheels of the lock become stuck after the display becomes the dead end "0102".
Example 2:
Input: deadends = ["8888",, target = "0009"
Output:
Explanation: We can turn the last wheel in reverse to move from "0000" > "0009".
Example 3:
Input: deadends ["8887", "8889", "8878", "1878"""8898"" "8898", "8898"""7788", "8988", "7888", "9888"], target "8888"
Output: -1
Explanation: We cannot reach the target without getting stuck.
Constraints:
deadends.length S 500
lendtth
larget.length = 4
target will not be in the list deadends.
target and deadends[i] consist of digits only.
aolution Solution()
assert solutioa. speaLock(["022)) [["0201", *0101", "0102", "0102"," "1212", "2002"), "0202") --
assert solution. opeaLock ["aaee"], "0009")
assert solution. (ow.iancnoeeecc(((((
(penLock(["aae]) ["aa87", *8889* "8878", *889** "8788", *896a* "7888", "7888","9888"1 "8888*) -1
Figure 4: A case illustration of GPT-4-turbo-preview (OpenLock). Part A: Task Description.
22

---

Completion
Trom typing inport List
from collections inport deque
claes Solution:
def openLock(selff, deadends: List[str), target: str)
def (nods):
in ra)ga(4):
int (node [i])
for in (-1, 1):
y yield bode[:1] str(y) mode[iv1:)
d) X 10
dead set (deadends)
queue deque([ 0000", 0)])
( 0000
while queue:
node, depth queue popleft)))
target:
return depth
node in deed:
continue for nei in (nogh:ors(nods):
seen aneei)
quaue. appand depth 1))
Ze burn
Figure 5: A case illustration of GPT-4-turbo-preview (OpenLock). Part B: GPT-4turbo-preview generated code.
23

---

Execution Time Profile
Total time:
File: 752_tmp tap.py py
Function: openLock at line
Line
Fite
Time
Per Ret
X Time
Line Contents
24
@profile
opanl.oc(sael1,
Liat] targat:
- atx)
>> int:
26
200
14151.9
70.8
D.1
Oprofila def neighbora
107.6
0.5
28
for i in range (4):
(node
30
for d in (-1, 1):
31
y yield mode [:1] str(y)
32
node [i+1:]
34
100
189.1 1
1.9
0.0
dead (etadeeedds))
queue doque((('0000)
36
100
68.8
0.6
0.0
seen 0'000
37
38
478172
109953,5
0.2
0.5
while queue:
39
478172
183034.8 88
0.4
0.8
node, depth puele-top)eft()
node target:
41
100
18.8
0.2
0.0
return depth
105361.0
43
253
44.3
0.2
0.0
continue
44
4300371
21651001
5.0
91.8
for nel in geighbors((ode)
45
3822552
969789.3
0.3
4.1
if nei not in seen add (nei)
46
576544
200262.3 9
0.3
0.8
queue queue append depth
48
reture
Total time: 0.0129757 8
File: 752_tmp-pp py
Function: neighbora at line 26
Line
Péta
Pime
Per Hét
X Time
Léne Contents
26
Oprofile
27
def (eidhbors(aode)
28
5345
1206.2
0.2
for range (4))
29
4276
1695.9
0.4
12.3
int [inde[i])
(-1,
31 8562
2248.4 4
0.3
17.3
(x d) X 10
32
8552
5619.3
0.7
43.3
yield bode [:1] str
node [1+1:]
Figure 6: A case illustration of GPT-4-tubo-previeew (OpenLock). Part C: Execution time profile.
24

---

Memory Usage Profile
Line
Now usage
Increment
Becarnences
Lise Contents
Ereciaion-PROFILE PROCITII,PPPECCCIOOO) PRECISION)
15
def openLock(eelI, deadende: [ist(str), target:
stx)
+>
16
48.8 M1B
0.0 HiB
4225
def naighbors (nade):
M1B
21123
range (4)
18
48.8
M1B
0.0 M1B
16898
x int (node [i])
(-1, 1):
20
48.8
M1B
0.0 M1B
33797
(x d) X 10
21
48.8 MiB
0.1 HiB
67594
yield node[:i] + str(y)
mode [i+1:]
22
(deadenda)
24
48.5
M1B
0.0 M1B
1
queue deque [( 0000 0)])
25
M1B
0.0
seen 0000
26
27
48.8
MiB
0.0 MiB
4229
while queue: nade, depth gueue.popleftt.
28
48.8
MiB
0.1 MiB
4229
29
48.8
M18
0.0 M1B
4229
1f node targat:
depth
31
48.8
MiB
0.0 MiB
4228
if node in dead:
33
48.8
M1B
0.0 MiB
38022
for nei neighbors (node):
34
48.8 MiB
0.0 MiB
33797
if nei not in seen:
36
48.8
M1B
0.2 MiB
4977
(nei)
36
48.8
M18
0.0 M1B
4977
queue (pein depth + 1))
38
return
Figure 7: A case illustration of GPT-4-turbo Grr-4-turbo-previeew (OpenLock). Part D: Memory usage profile.
25

---

EFFI-LEARNER Completion
from typing ingort List from collactions import deque
clasa Bolution:
def openLock(eelf, deadende: Liet(stz), target:
"""Calculate the minimun number of moves to open the lock from '0000" to the target avoiding now
if '0000' in deadends:
return -1
if target '0000"; return
def getllextStatensstate:: str) -> List[atr]: """Generate the nest states for glven lock state.
[
for in ranga(4):
current_ digit intt(ttteei))
for diff in (-1, 1]: decrease or increase the digst
nev_digit (current (eurrent_digit digit diff)
res. irsaata + str(nev_digit) state
return rea
def bta (quaue: Deque[str] distances: dict, other_diatancea, dict) -> int: "o*Perform step BFS, expanding for in rango(lenqqueue)) queue. poplaft()
gueue. van
step_count distances
for nextState in SetNextttateeeuurree]
if nextState in deadends_set nextState distances: continue
nextState diber_distancee:
reture stop_count + 1 other dtw.ddisstaccennetttttttee
distances (nextStattt stee_cconn.
queue. appond (mextState)
return
deadends_eee _eet set (deadends) Early if instial if 10000 in deadends_set: return
deadend
Instialise 8F5 structures for bidirectionss search queue1, quese2 (['00([)00001]] doqua ([target]) distancel, dietance2 ("0000": 0), (target: 0)
Penforn BPS from both ends
while queuel and queue2:
result (gueuue dietancel, distance2)
result !- -1; result
result (queue, dietance2, distancel) -1:
return result.
return
Figure 8: A case illustration of GPT-4-turbo-preet -preview (OpenLock). Part E: Self-Optimization Code.
26

---

Execution Time Profile
Total time: 3.36559
File: 752_ typ.py
Function: openLock at line 24
Li1n
Peta
Tine
Per Set
X Time Line Contenta
24
Oprofile
25
def opent.ckksee11, deadenda: Liat[str], target:
-
stx) >> int:
26
*""Calculate the minimun nunber of moves to open the lack from 10000! the target avoiding deadends. *a.
27
100
65.1
0.7
0.0
if 10000" in deadends:
29
100
35.0
0.3
0.0
if target -- 10000": retura
30
31
67
58
100
99.3
1.0
0.0
deadenda_sed _et set (deadends)
Sarly if state
deadend
60
100
36.7
0.4
0.0
'0000' deadende_set:
61
62
Instsal<se BFS structures for bidérectional
+ #earch
1.2
queuel, queue2 deque deque (('00001))
daque
(oque([targert (target])
distancel,
<"0000% 03, (target:
0}
66
Perfore both while queue1 and queue2:
68
518
154.2
0.3
0.0
1936061.9
3737.6
67.7
result (quequl distancel,
<
distance2)
70
153.0
0.3
reault result result bis (queue2, diatance2,
71
62
10.3
0.2
72
466
1388682.7 7
2980.0
41.4
distancel)
73
466
1
0.3
0.0
if result 1- -1: result
75
76
recuro -1
Figure 9: case illustration of GPT-4-uurbo-reeview (OpenLock). Part Fl; Execution time profile for Self-Optimization Code.
27

---

Execution Time Profile
Total tine: 0.000697948
File: 752_tmp.pt
Function: getNextStates at line 32
Line
Bets
Time
Per Höt
X Time
Line Contents
32
Sprofile
33
def gelMextSlatee(statee str) [str] **"Generaee the states for given
34
+
lock atate."
xee for in range (4):
36
260
62.4
0.2
8.9
37
10.7
currant_digit (nt(tttta[)))
38
624
127.4
0.2
18.2
for diff in [-1, 1]: # decrease
increase the dégit
39 416
104.5
0.3
15.0
nev_digit (eurrent_digit
diff) 10
308.4
44.2
ces.appon(ettate(:il
str (nav_digits)
[t=te[1+1:]]
41
8.8
0.2
1.3
return rea
Total time: 0.00182689
File: 752_tmp-pF py Function: bfs at line 43
Line
Béta
Pime
Per Sit
X Time Line Contents
43
Oprofile
44
def bfo(queue: Deque(str), distances: dict,
distanddstta:e::: dict)
int:
45
**"Perform a step of BFS, expanding the
<->
quaue. naz
46
57
12.9
0.2
0.7
for range(Ien (qunqu)))::
47
52
14.0
0.3
0.8
current queue popleft step_count distancer (current) for nextState in
48
52
11.4
0.2
0.6
49
462
1499
3.2
82.1
getliextStaten (curreent) (current):
50
411
114.9
0.3
6.3
if nextState in deadends_set
<
distances:
51
235
34.5
0.1
1.9
continue
52
176
38.7
0.2
2,1
if nextState in other distances: return step_count
53
0.7
0.7
0.0
<4
other distances [nextState]
distances [nextState] atep_count
4
1
0.3
queue (pptnnde
56
0.9
0.2
0.0
retura
Figure 10: A case illustration of GGPT-4turbo-prevvee (OpenLock). Part F2: Execution time profile for Self-Optimization Code.
28

---

Memory Usage Profile
usage
Increment
Precieion-PROPILE PREISIIEPPEPECCCIOO PRECISION)
15
def openLock(self, deadende: Liet(str) target:
stx)
>> int:
16
C'Callulate the minimun number of noves to 10000! target avoiding deadends. ta.
open lock
17
48.4 MiB
0.0 MiB
if '0000' in deadends: return if target -- 10000'; return
19
48.4 M1B
0.0 MiB
20
21
22
48.4 M18
0.0 M1B
819
def getflextStatea (state atr) -> List[atr]: un Generate the
given
lock state.
24
48.4 M1B
0.0
MiB
818
rea for in range
25
48.4 MiB
0.0 MiB
4090
26
48.4 MiB
0.0 HiB
3272
current, current_digit digit int (statee))))
27
48.4 M1B
0.0
MiB
diff [-1, 1]: decrease
+
increase the degst
nex_digit (current_digit diff)
X
10
29
48.4 MiB
0.0 MiB
6544
ree aps.aappnu(etate[:.
(nev_digit)
state(i
30
48.4 MiB
0.0 MiB
818
return res
31
32 48.4 MIB
0.0 MIB
10
def bla (queue: Deque [stz], distances: dict,
+
other ather_distancer distances: dict)
33
"""Perforn step BFS, expanding the
+ current queue,
34
48, 4 MiB
0.0 MiB
826
for in range (qquqqqeq) current ppaue.poplafte step_count distances for nextState in
36
48.4 M1B
0.0 HiB
818
M1B
37
48.4 MiB
0.0 M1B
7356
+
getnexttStttea (current):
38
48, M1B
0.0 MiB
6539
nextState in deadends_eee
nextState in distances:
39
48, 4 MiB
0.0 MiB
5067
continue
40
48.4 M18
0.0 MiB
1482
1f nextState in other_distances atep_count
M1B
other_ distances [nextState]
MiB
Eastaancesneextttttn step_count
43
48.4 MiB
0.0 MiB
1481
queuo apon
44
48.4 M1B
0.0 MiB
45
MiB
deadends_ (deedende) # Farly eait if the instial state is deadend 10000' deadends_cet:
47
49
return
50
61
Iestsaliza atructures bédirectsonal
-
#earch
MiB
queue1, queue2 deque(['0000) (['0000 distancel,
<ua daque((target)) [target])
MiB
<"0000": 0), (target: 0}
54
66
Perforw BPS free both ends
56
M1B
while queuel queue2:
57
48.4 MiB
0.0 M1B
result bis distancel, distance2)
MiB
result |- return result
69
M1B
0.0
reault bta(queue2, distance2, distance1)
61
48.4 MiB
0.0 MiB
if result I- -1: raturn result.
62
64
zet burr
Figure 11: A case illustration of GPT-4-turbo-preview (OpenLock). Part G: Memory usage profile.
29

---

Task Description
Problem:
Given two sorted arrays numsi and nums2 of size and respectively, return the median the two sorted arrays.
The overall run time complexity should be (Ilog(m n)))
Example
Input: numsi [1,3], nums2 [2]
Output: 2. 2.00000
Explanation: merged array [1,2,3] and median is
Example 2:
Input: numsi [1,2], nums2 [3,4] Output: 2.50000
Explanation: merged array [1,2,3,4] and median is (2 3) 2 2.5.
Constraints:
nums1 length
nums2 length ==
0 S m - 1000
S n - 1000
1 1 m + n $ 2000
- < nums1 [i] nums2 [i] < 10
solution Solutioa()
findkediaaSSorteddrraaa(( 3] [2])
assert solution. solution. findhedianssrrtedarras((( 2], [3, 4]) - 2.5
Figure 12: A case illustration of GPT-4-turbo-previeer (FindMeddannSorredArrays)) Part A: Task Description.
Task Description
class Solution:
fnni nunsi, nuns2)
A, B numal, nuns2 total len(A) len(B)
half total
if len(B) len(A): A, B B. A
0, len(A)
while True:
//
j -- half 1 8
Aleft A[i] if >- 0 else float("
Aright
len(A) float (*infinnn))))))
Blaft B[j] if >- elae float((-inrinitty)
Bright B(j if (j 1) len(B) else floss("anfinity))
if Aleft <- Bright and Bleft <- Aright:
total
return min (Aright, Bright)
return (nax(Aleft, Bleft) nin(Aright, Bright))
elif Aleft Bright:
I -i-1
-i+11
Figure 13: A case illustration of SPT-4-turbo-previeer (FindMclianSorcdAArray))) Part B: GPT-4- turbo-preview generated code.
30

---

Task Description
Timer unit: 1e-06
Total time: 0.00119913
File: s_tap.pi
Function: finteeiiaSSortedrrrey
X
45
Oprofile
46
def IhadheelaaSrrtedkrrapeeell,, nunal, nuna2)
- flost:
nunel, total len (A) + len(B)
48
100
57.2
0.6
4.8
49
0.4
3.4
half total //
50
51
100
54.9
0.5
4.6
len(B) < len(1):
46
53
len(A)
65
169
68.6
0.3
4.9
while Trus:
63.3
0.4
5.3
r) //22 f
57
169
62.1
0.4
5.2
half -
58
Aleft A[i] if
float ('iinfinityy))
Aright
len(A)
-
e1se float inityi
61
169
84.3
0.5
7.0
Bleft B[j] if j >- 0 else
float (* (oaatt--nninityy inniiiinity)
62
169
95.3
0.6
8.0
Bright B(j 1] if (j 1) len(B)
elae float ('infinity"))
63
Bright if total X 2: return nirggig Bright) (max(Aleft, Bleft)
Aright:
65
100
51.6
0.5
4.3
66
51
86.1
1.7
7.2
67
49
103.0
2.1
8.6
<
min (Arrght, Bright)) 2
69
Aleft Bright:
69
45
16.8
0.4
1.3
r -1
70
else:
71
24
8.4
0.3
0.7
1+++1
Figure 14: A case illustration of GPT-4-turbo-previeww (FindMedinnSortccdArray)) Part C: Execution time profile.
31

---

Task Description
Line
Ken usage
Increment
Pocurrences
Lise Contents
54
24.7 MiB
24.7 MiB
precision PRRcISSOnPPROFTLE..C PRECISSON)
55
def IindteeissSotttdrrrayeeeell nunsl, nuna2)
float:
M1B
57
24.7 M1B
0.0 M1B
1
total len(A) lea(B)
69
24.7 M1B
0.0
(B) len(A):
61
A. 8 B,
62
len(A)
64
24.7
MiB
0.0 MiB
while True:
66
24.7 M1B
0.0 MiB
half 1 2 8
67
68
24.7 MiB
0.0 MiB
Aleft A[I] else
float(" float("-ints ('iiniiitty)))
Aright
1) len(A)
float ("infinity") MiB
B[J]
float(" ('ont("innfnite
71
24.7 MIB
0.0 MiB
Bright B[J 1] (J 1) len(B) else
(lont("infiniee
72
24.7 M1B
0.0 MiB
if Alaft Bright Bleft Aright: if total X 2: return min(Aright, Bright) return (nax(Aleft, Bleft) min(Arggh,
74
24.7 MiB
0.0 MiB
76
4
Bright)) / 2
77
elif Aleft Bright: x -1-1
78
80
+++1
Figure 15: A case illustration of GPT-4-uurbo-preview (FindMedianSSorccAAAaay)) Part D: Memory usage profile.
32

---

Task Description
class Solution:
Sarraddddaaarrrddrrasasaea,,, nunal, muns2) Enesire Nans éa the sealler array to miséénize binary search domain if len(nums1) len (nune2):
nunal, numa2 nuns2,
x, (emmme))) lea (nuns2)
lou, high 0, while high:
partitionX (1ov high) //
partitionY
1) partitionX
If partitionI is 0 i8 aeans nothing is there on left side. Use -inf for manteftx float("-inff) partitionX partationl <a Zength of input thes there is nothing on right séda. dhe thf for windightt
numel (partitiit 1]
minRightX float('inf') if partitionK else numel (partiitaii
flost("-inf') partitionY
aume[parrtitonr" 1]
minRightY - float('inf') if partitionY -- else aume
if maxLafeX <= minRightY and maxLartY <= minRightx:
The partition is correct
(x y) % 2
return (nax (maxmaxxaf,,,, maxlaftY) min(ninRightt, minRight!)) / 2.0
else:
return max mmxmmxttf,,,, maxleftY)
minRightY:
are too far right aide for partetiont. Left
side.
high partitionl
else: Ve los partitionX
left side for partitionl. Go right side.
Figure 16: A case illustration of GPT-4-turbo-preview (FindMedianSSotteeAAaaa)))) Part E: Self- Optimization Code.
33

---

Task Description
Timer unit: 1a-06
Total time: 0,00114754
File: 4_tap-pP
Line
Fite
Time
Per BEt
X Time
Line Contents
45
0profile
def findheliasScreeddrryyyeee,, nunsi, nume
1- float:
armay
-
minseise binery search domais
48
100
63.3
0.6
5.5
len (nune1) >en(
46
numal, buna2, len (nunss), (enme)))
50
100
44.7
0.4
3.9
52
100
32.3
0.3
2.8
1ox, high 0, high:
79.6
0.4
54
184
72.1
0.4
6.3
partitionX (1ow high) // 2 partiticnY (x 1) //
55
69.7
0.4
6.1
partitionx
56
If partétion means nothing
<
there on left séda. Use -inf for marlaftZ
92.6
0.5
8.1
naxLeftX float('-iof') if partitionx
-
elee munsi(partitiin 1]
59
If partition is length of input then ninRightX (loat("inf') if partitionl
there Nothing right side. tinf for minhightX
60
184
61.5
0.4
7.1
munsi [partition]]
61
62
184
94.9
0.5
8.3
naxLeftY float('-ir'") if partition
else cums(paartiioonY
1]
63
184
86.9
0.5
7.6
ninRightY float('inf") if partitionY
mumaa[partiiionn]]
64
ninRightY
-
minRightX:
66
The partition correct if (x X -= return (max maxlefty)
67
100
62.8
0.5
4.6
68
49
108.3
2.2
9.4
(ninoigggg,,,, minRightY))
2.0
69
alse:
70
1.7
return (ax naxLefey)
71
84
34.7
0.4
3.0
elff naxLefiX minRightY:
< far on right side for partstionI, Go on left side.
72
30
11.0
0.4
1.0
high partitionX
73
else: de are too for on left side for
partstionz, right
74
64
18.2
0.3
1.6
lov partitionX +
Figure 17: A case illustration of GPT-4-uubb--preview (FindMedianSorteddAraaa)) Part Execution time profile.
34

---

Task Description
Line
Hom usage
Increment
Oecarrences
Line Contents
M1B
precision PROPILE PROPILL PROOTLLE_PEECISIIOO PRECISION)
def fiadteeiaaBorteedrrraaaelii,, munal, nuna2)
flost:
56
Ensure nansi is the smaller array minimize
bénery search domain
57
24.8 M18
0.0 M1B
1
ir len(mua1) len (numa2):
nunsl, nume2 nuns2, numel
59
24.8 MiB
0.0 MiB
1
X, lenmmums)), (nune2)
60
61
24.8 MiB
0.0 MiB
lov, high shile lox high: partitionx (1aw high) //
62
24.8 MiB
0.0 HiB
63
24.8 M1B
0.0 MiB
64
24.8
M1B
0.0 M1B
1
partitionY (x y 1) // 2 partitionX
66
If partitionl is 0 i8 meons nothing is
+
there on left side, Use -inf for mozteftI
67
24.8 MiB
0.0 MiB
maxteftI float("-inf')) if partitionX --
<
alse nunsl (partitionX
partstiont tength there 88 nothing on right side. Use tinf for windightZ
-
MiB
minRightX fleat('inf') partitionX
else nunsl [partitienX]
70
71
24.8 MiB 0.0 MiB
maxLefty (loat((-inf')) if partitionY minRightY float('inf') if partitionY
<
alse mums((partition'
M18
0.0
-
else nune2 (partition))
74 24.8 8 M1B
0.0 MiB
if naxLeftX <- minRightY and maxlafty
-
minRightX:
75
The partstion is connect
76
24.8 M18
0.0 MiB
12 + y) X =n 0:
(nax(taxtei,,, maxLeftY)
-
min(ninRightX, minRight!)) / 2.0
79
return max maxtafty)
80
clif maxLeftX ninRightY: A Ve are too
right side partstiont. left side.
81
high partitionx too for left for
~ partitionI. So on right side.
partitionX
Figure 18: A case illustration of HeT---turbo-previev (FindMedianSoreedAAraas))) Part G: Memory usage profile.
35

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, provide EFFI-LEARNER improve the efficiency of LLM- generated code.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of EFFI-LEARNER in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
36

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all source code of EFFI-LEARNER in Supplementary file.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
37

---

Answer: [Yes]
Justification: The source code has been uploaded into the supplementary file. The dataset used in the paper also has instructions access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https www.sSuwwisssispoooooooooooooooi
/nipss.cc/pulli//guu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the experimental setting in the main paper and appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our proposed method is inference-only approach for LLM and adopt the greedy-decoding strategy for all of our experiments, making the experiment results of each session consistent.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
38

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All information was provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss societal impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
39

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cite all evaluated models and datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
40

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "t43NGx8hX8", "reviewer": "Reviewer_H3VZ", "summary": "This paper presents SOAP, a new code generation approach that improves the efficiency of code generated by a LLM. SOAP adopts a self-refinement method that iteratively prompts the LLM to re-generate the code based on the profiling results. Specifically, it uses the line-profiler package in Pyhton to get the execution time profiling results and the memory_profiler package to get the memory usage profiling result. SOAP is evaluated on three benchmarks and 22 LLMs, including both 16 open-sourced and 6 closed-sourced models. The results show that SOAP can effectively improve the execution time and memory usage of code generated by most LLMs. While it slightly decreases the LLM's performance on functional correctness, the impact is small.", "questions": "1. What is the key technical novelty of this work?\n2. How would you compare SOAP with existing transformer or LLM-based code optimization work?\n3. What are the prompts used in the experiments with Self-Refine and Reflexion?", "limitations": "The discussion on limitations looks reasonable.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. SOAP is evaluated on multiple benchmarks and many LLMs. \n2. The results look promising. \n3. The paper is well-written and easy to follow.", "weaknesses": "1. The technical novelty of this work is limited. The self-refinement method proposed in this work is very similar to existing self-refinement methods such as Self-Edit and Critic. The only difference is that SOAP uses profiling results while existing methods use testing results, etc.  \n2. The authors ignored existing work on using transformers or LLMs for code optimization. They should discuss these papers in the related work and also use them as comparison baselines. \n- Liu, Shengcai, et al. \"Large language models as evolutionary optimizers.\" arXiv preprint arXiv:2310.19046 (2023).\n- Chen, Zimin, Sen Fang, and Martin Monperrus. \"Supersonic: Learning to generate source code optimisations in c/c++.\" arXiv preprint arXiv:2309.14846 (2023).\n- Shypula, Alexander, et al. \"Learning performance-improving code edits.\" arXiv preprint arXiv:2302.07867 (2023).\n3. Section 4.3 compares SOAP with Self-Refine and Reflexion. It is unclear how Self-Refine and Reflexion were configured in this experiment. Did the authors use the original prompts from Self-Refine and Reflexion? Or did the authors modify their prompts to include instructions on code efficiency? If the former, there is a comparison fairness issue since the original prompts in Self-Refine and Reflexion are not designed for code optimization. If the latter, the authors should provide the prompts used in Self-Refine and Reflexion and justify the prompt design. \n4. The experiments on the impact of step-optimization steps (Section 4.2) only include two big models. The findings may not be generalizable to smaller models. Why not also include some smaller models?\n5. The experiments on the impact of functional correctness were only done on EffiBench. Since EffiBench is not specifically designed for \n functional code generation, the results may not be representative. The authors should also experiment with HumanEval and MBPP and report the pass@1 scores on these two benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n\n**W1 & Q1: Novelty and Similarities with Self-Edit and Critic**\n\nTo address the reviewer’s concern about the novelty of SOAP in comparison to Self-Edit and Critic, we provide detailed evaluation results in Table below. These results demonstrate that SOAP outperforms both Self-Edit and Critic significantly.\n\n| Optimization Profile | ET (s) | NET | MU (Mb) | NMU | TMU (Mb*s) | NTMU |\n|---|---|---|---|---|---|---|\n|**GPT-3.5-Turbo-0301**|||||||\n|InitialVersion|0.36|2.50|91.25|2.45|157.50|19.75|\n|Self-Edit|0.42|3.67|59.86|1.65|24.87|3.28|\n|Critic|0.60|4.25|102.75|2.82|351.91|46.39|\n|SOAP|0.28|2.01|36.08|0.99|12.43|1.64|\n\nSOAP introduces a unique overhead profiler that significantly enhances the efficiency of LLM-generated code by optimizing both execution time and memory usage. In contrast, existing methods focus primarily on execution time and cannot comprehensively address efficiency concerns. Specifically, Critic requires progressive amendment of outputs, which does not effectively tackle efficiency-critical issues.\n\nThe novelty of SOAP lies in two aspects:\n\n- **Dual Optimization Focus**: Unlike existing works that primarily target execution time, SOAP optimizes both execution time and memory usage, providing a more holistic approach to code efficiency.\n\n- **Overhead Profile-Guided Optimization**: SOAP utilizes overhead profiles, including execution time and memory usage profiles, to guide the optimization process. This allows SOAP to achieve SOTA results in efficiency, as demonstrated by our ablation studies.\n\nWe would like to highlight that self-refinement is a broad direction with many published papers exploring various novelties. The specific challenge addressed by SOAP is improving the efficiency of LLM-generated code, a critical and valuable contribution to the field. Our experiments clearly show that the inclusion of overhead profiles significantly enhances optimization effectiveness compared to existing methods.\n\n**W2 & Q2: Discuss and compare transformers or LLMs for code optimization.**\n\nWe would like to highlight that SOAP and these methods [1,2,3] are orthogonal, which means we can combine SOAP and these efforts. \n\nLMEA [1] uses evolutionary search to generate more efficient code compared with the initial code, which requires 250+ solutions for each task to achieve optimal results, whereas SOAP only generates 5 solutions. LMEA requires massive token usage and execution time consumption. Due to limited time and resources during rebuttal,  we are not able to conduct experiments with LMEA. However, we commit to include the results for LMEA in the revised manuscript.\n\nFor Supersonic [2] and PIE [3], we conduct experiments in **Author Rebuttal Table 1**. For PIE, we conduct experiments for five different prompts. As shown in Table 1, the performance of these methods is further boosted when SOAP is applied to them. On average, PIE+Few-Shot requires 0.82s for each generated code to execute. While PIE+SOAP+Few-Shot generated code only requires 0.41s to execute. \n\nRef:\n\n[1] Large language models as evolutionary optimizers. 2023\n\n[2] Supersonic: Learning to generate source code optimizations in c/c++. 2023\n\n[3] Learning performance-improving code edits. 2023\n\n\n**W3 & Q3: Prompts for Self-Refine and Reflexion**\n\nFor a fair comparison, we conduct experiments for baselines by revising the prompt used by SOAP in our experiments, which specifically requires LLMs to refine the coding efficiency. We provide the detailed prompts used in the paper in the Link: https://anonymous.4open.science/r/SOAP-FF6C/prompts.py\n\n\n**W4: Step-optimization steps only include two big models.**\n\nWe provide three smaller models, i.e., OpenCodeInterpreter-1.3B, DeepSeek-1.3B-Ins, and CodeLlama-7B below. We can observe that SOAP consistently optimizes the efficiency of LLM-generated code for each optimization step. \n\n| Steps | ET (s) | NET | MU (Mb) | NMU | TMU (Mb*s) | NTMU |\n| --- | --- | --- | --- | --- | --- | --- |\n|**OpenCodeInterpreter-1.3B**|||||||\n|0|1.60|1.52|38.91|1.00|89.16|1.11|\n|1|1.60|1.51|38.91|1.00|88.77|1.11|\n|2|1.59|1.51|38.91|1.00|88.16|1.10|\n|3|1.59|1.50|38.91|1.00|87.85|1.09|\n|4|1.59|1.50|38.91|1.00|87.77|1.09|\n|5|1.29|1.23|38.91|1.00|70.63|0.88|\n|**DeepSeek-1.3B-Ins**|||||||\n|0|1.42|1.32|36.04|1.00|40.61|1.12|\n|1|1.42|1.23|36.04|1.00|39.30|1.09|\n|2|1.42|1.20|36.04|1.00|37.99|1.05|\n|3|1.15|1.16|36.04|1.00|37.33|1.03|\n|4|1.15|1.12|36.04|1.00|36.43|1.00|\n|5|1.15|1.07|36.04|1.00|35.48|0.98|\n|**CodeLlama-7B**|||||||\n|0|4.70|3.68|46.76|0.99|212.41|1.93|\n|1|4.59|3.58|38.94|0.82|165.16|1.50|\n|2|4.56|3.55|39.11|0.82|162.16|1.47|\n|3|4.56|3.56|39.03|0.82|161.98|1.47|\n|4|4.56|3.55|39.03|0.82|161.57|1.47|\n|5|4.52|3.54|38.67|0.82|157.76|1.43|\n*SOAP for different optimization steps.*\n\n**W5: Pass@1 on HumanEval and MBPP.**\n\nAs shown in the tables below, we provide the pass@1 for HumanEval+ and MBPP+, where the evalplus version is used in our experiments to measure the efficiency of LLM-generated code as it contains hundreds of tests. We can observe that the pass@1 of LLM-generated code **only has merely changed for these benchmarks**. The performance of our evaluated LLMs on HumanEval+ does not show a decrease. On MBPP+, the pass@1 decrease is only 0.0% and 0.84% for HumanEval+ and MBPP+, on average. Notably, for more powerful LLMs like GPT and Claude, the performance remains essentially unchanged.\n\n| Model | Initial pass@1 | SOAP pass@1 |\n|---|----|----|\n|OpenCodeInterpreter-DS-1.3B|30.5|30.5|\n|CodeLlama-7b-Instruct-hf|39.6|39.6|\n|CodeLlama-70b-Instruct-hf|45.7|45.7|\n|gpt-3.5-turbo-030|62.8|62.8|\n|claude-3-sonnet|59.1|59.1|\n*Results on HumanEval+.*\n\n| Model | Initial pass@1 | SOAP pass@1 |\n|-------|-----|------|\n|OpenCodeInterpreter-DS-1.3B|57.9|55.8|\n|CodeLlama-7b-Instruct-hf|40.7|40.7|\n|CodeLlama-70b-Instruct-hf|53.9|52.1|\n|gpt-3.5-turbo-0301|59.8|59.5|\n|claude-3-sonnet|58.4|58.4|\n*Results on MBPP+.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BVriChBtKk", "reviewer": "Reviewer_5c1j", "summary": "The paper introduces a novel method for code super-optimization by iteratively refining code using LLMs with profiling information. The focus is on enhancing the efficiency of LLM-generated code, addressing execution time and memory requirements. The proposed methodology involves generating code with an LLM and then iteratively feeding profiling results back to the LLM to optimize the code's efficiency. The profiling information includes per-line execution time and memory usage data, helping to identify specific lines that require significant overhead. Experimental results show that this approach outperforms existing unsupervised self-refine and result-aware self-refine methods on efficiency benchmarks like EffiBench, with supplementary experiments on HumanEval and MBPP.", "questions": "## Questions & Suggestions:\n\n1. What is the average number of iterations required for optimization? What are the stopping criteria for iterative optimization? Did you set it to 5 for all the experiments? How is it determined when to stop the refinement process?\n2. How significant is the profiling overhead? Is there any analysis on the average number of extra tokens required for including the profiling information as prompts? How does this affect the overall efficiency and effectiveness of the optimization process?\n3. The downgraded pass@1 results in Section 4.4 suggest that the proposed method cannot guarantee semantic equivalence after code refinement. This could impact the practicality and reliability of the code superoptimizer, especially in safety-critical applications. How can this issue be mitigated?\n4. Analysis of the optimization techniques applied by the LLM is necessary. Are there certain types of optimization where the method may fall short?\n5. The current approach relies on the LLM to analyze and identify hot spots from the profiling results. Given the standardized format of profiling results, could rule-based post-processing techniques that filter out hot spot information improve or accelerate the self-refinement process?\n6. In line 189, only code that passed all test cases is considered in the evaluation. What if the refined code fails the test cases during optimization? How is this situation handled?\n7. In Table 2, the optimization effect on memory usage appears limited compared to execution time. Also, in the ablation studies (Table 3), self-optimization with only execution time profiling feedback achieves significant memory usage reduction. Does this mean that memory optimization is relatively easier in the evaluated benchmark, or does execution time information play a more critical role in optimization?\n\n---\n[1] Madaan, Aman, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon et al. \"Self-refine: Iterative refinement with self-feedback.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Olausson, Theo X., Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. \"Is Self-Repair a Silver Bullet for Code Generation?.\" In The Twelfth International Conference on Learning Representations. 2023.\n\n[3] Chang, Pohua P., Scott A. Mahlke, and Wen‐Mei W. Hwu. \"Using profile information to assist classic code optimizations.\" Software: Practice and Experience 21, no. 12 (1991): 1301-1321.\n\n[4] Agakov, Felix, Edwin Bonilla, John Cavazos, Björn Franke, Grigori Fursin, Michael FP O'Boyle, John Thomson, Marc Toussaint, and Christopher KI Williams. \"Using machine learning to focus iterative optimization.\" In International Symposium on Code Generation and Optimization (CGO'06), pp. 11-pp. IEEE, 2006.", "limitations": "The limitations related to the overhead of profiling and the semantic correctness of the optimized code are mentioned in the paper but not analyzed in detail. A more comprehensive cost-benefit analysis, including the extra tokens required for profiling information in prompts, would provide a clearer picture of the method's overall efficiency. The authors should also consider the broader implications of this optimization approach, including its potential impact on code reliability and safety in practical applications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses a gap in current LLM code generation research by focusing on code efficiency.\n- The proposed iterative refinement method with profiling information is still innovative and provides a clear path to optimizing execution time and memory usage.\n- Experimental results demonstrate that the method significantly outperforms existing approaches in terms of code efficiency, showing the potential practical value of this technique.", "weaknesses": "- LLM self-refinement or self-repair techniques have been extensively employed in generating responses/codes [1] and debugging tasks [2], whereas utilizing code profiling information for optimization is also a long-established concept [3, 4], which raises concerns in the proposed method's technical novelty. \n- The paper seems to treat the number of optimization steps as a hyperparameter. While Section 4.2 covers the impact of self-optimization steps, the average number of iterations required to achieve the optimization results in Table 1 is not clearly disclosed. This information is critical for comprehending the efficiency and practicality of the proposed method. Automating the stopping criteria for iterative optimization is also important in practice, but it is not supported by the proposed method, thus limiting its applicability.\n- The potential semantic degradation in code correctness after refinement, particularly in scenarios where test cases do not cover all edge cases (which are common in real-world applications), raises concerns about the reliability of the optimized code, particularly in safety-critical scenarios. \n- The methodology section, particularly section 3.4, is overly verbose. The explanation of each part of the prompt could be condensed to improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. \n\n**W1 & Q1: Impact of self-optimization steps**\n\nIn our paper, the self-optimization steps are **constraint as 5** Section 4.2, consistent in our experiments. 5 is the default setting for many existing self-optimization methods [1, 2]. \n\n[1] Islam M A, et al. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving[J]. arXiv preprint arXiv:2405.11403, 2024.\n\n[2] Zhou, Andy, et al. \"Language agent tree search unifies reasoning acting and planning in language models.\" arXiv preprint arXiv:2310.04406 (2023).\n\nBased on your suggestion, we require GPT-3.5-turbo-0301 to consider stopping the SOAP self-optimization process and only constrain the max step as 5. As shown in the Table below, the adaptive self-optimization process improves the efficiency of LLM-generated code. However, it does not achieve SOTA performance compared with SOAP. For example, when optimizing its initially generated code, it still needs 0.35s to execute, on average. SOAP only requires 0.28s. One key reason is that in each step, the efficiency of most code can be optimized. If the self-optimization stops early, the code’s efficiency remains in the sub-optimal results.\n\n| Steps | ET (s) | NET | MU (Mb) | NMU | TMU (Mb*s) | NTMU | Step |\n|-------|-----|-----|---|-----|---|--|---|\n|InitialVersion|0.36|2.50|91.25|2.45|157.50|19.75|0|\n|Adaptive|0.35|2.47|55.00|1.52|178.98|23.83|1.2|\n|SOAP|0.28|2.01|36.08|0.99|12.43|1.64|5|\n\n**W3 & Q3: Semantic degradation in code correctness after refinement where test cases do not cover all edge cases.**\n\nWe clarify that most edge cases have been covered in SOAP. For EffiBench, the private test cases covered most edge cases for each canonical solution. For the other two datasets, HumanEval and MBPP, we also utilize the test cases generated by EvalPlus to cover more edge cases. \n\nOn the other hand, the degradation is small (0.5% pass@1 for EffiBench, 0% pass@1 for HumanEval+, and 0.84% pass@1 for MBPP+ on average as follows). We believe that the potential degradation in correctness after refinement is manageable.\n\n||Initial pass@1|SOAP pass@1|\n|---|----|---|\n|OpenCodeInterpreter-1.3B|30.5|30.5|\n|CodeLlama-7b|39.6|39.6|\n|CodeLlama-70b|45.7|45.7|\n|gpt-3.5-turbo|62.8|62.8|\n|claude-3-sonnet|59.1|59.1|\n*Results on HumanEval+.*\n\n| | Initial pass@1 | SOAP pass@1 |\n|---|---|----|\n|OpenCodeInterpreter-1.3B|57.9|55.8|\n|CodeLlama-7b|40.7|40.7|\n|CodeLlama-70b|53.9|52.1|\n|gpt-3.5-turbo|59.8|59.5|\n|claude-3-sonnet|58.4|58.4|\n*Results on MBPP+.*\n\n**W4: Section 3.4, is verbose**\n\nThanks for your suggestion. We will revise it accordingly.\n\n**Q2: Analysis of the profiling overhead**\n\nWe provide the breakdown of the overhead for SOAP and other prompt engineering in **Author Rebuttal Table 4**. Compared with the initial version, SOAP would increase the total token usage from 1.1M to 5.4M. \n\n**Q4: Certain types of optimization where the method may fall short**\n\nThere are certain types of optimization where the SOAP method may fall short. Specifically, tasks that **do not require algorithmic improvements** or **have already been implemented with optimal time/space complexity** may not benefit significantly from SOAP. One example is that as shown in Paper Fig. 12 to Fig. 18, when LLM first generates code for the task to sort two arrays, if the code already has an optimal time/space complexity, SOAP may not optimize the code into a more efficient version. We will add this discussion in the revised version.\n\n**Q5: Could rule-based post-processing techniques improve or accelerate the self-refinement?**\n\nBased on your suggestion, we conduct experiments for the SOAP by extracting the Top 5 lines with the highest execution time and memory usage and then use these profiler parts to guide LLM to optimize its previously generated inefficient code. \n\nAs shown in the Table below, rule-based optimization improves the efficiency of LLM-generated code compared with the initial version. For example, the average execution time (ET) of GPT-3.5-turbo-0301 decreases from 0.68s to 0.50s. \n\n| Steps| ET (s) | NET| MU (Mb) | NMU  | TMU (Mb*s) | NTMU  |\n|---|---|---|---|---|---|---|\n|InitialVersion|0.68|4.83|102.39|2.82|302.04|39.98|\n|Rule-Optimized|0.50|3.64|57.55|1.50|23.36|3.02|\n|SOAP|0.39|2.77|42.67|1.18|18.57|2.46|\n\n*Results for GPT-3.5-turbo-0301*\n\nHowever, the efficiency is lower than SOAP. For example, SOAP further decreases the average execution time for GPT-3.5-turbo-0301 from 0.50s to 0.39s. Only providing part of the overhead profile information may not make LLMs understand how to optimize the code to improve efficiency as the efficiency problems are usually based on all code.\n\n**Q6: Refined code fails the test cases during optimization?**\n\nIn our experiments, we follow a two-step process: \n\nFirst, if the refined code does not pass the public test cases during the optimization process, we directly use the initial version of the LLM-generated code for that particular task. \n\nSecond, if the code passes the public tests but fails the private tests, we filter out the entire task from our experiments for the respective model and do not calculate the optimized code's efficiency for any step of that task. \n\nFor example, if step-3 optimized code is correct, but step-4 optimized code is incorrect in private test cases, we will not calculate the efficiency for any step of that task for the corresponding model, ensuring that we only consider code that passes all test cases in the evaluation and provide a fair comparison across different optimization steps and models.\n\n**Q7: Memory optimization is easier in the benchmark**\n\nOne reason for memory usage being hard to optimize is that most of the tasks already achieve SOTA memory usage compared with execution time. Specifically, as seen in Table 1 in Section 4.2, most of the experiments achieve 1.00 NMU for LLM-generated initial code, which causes the code to be hard to further optimize compared with execution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pbhgUwbJ7f", "reviewer": "Reviewer_VC11", "summary": "This paper studies an important and timely issue: the inefficiency often found in code generated by current Large Language Models (LLMs), which can result in longer execution times and higher memory consumption. To mitigate this issue, the paper proposes a self-optimization framework SOAP, designed to improve the efficiency of LLM-generated code. SOAP starts from the code generated by an LLM and then executes it locally to profile its execution time and memory usage. These profiles are fed back to the LLM, allowing it to revise the code and reduce overhead iteratively. The authors verify the effectiveness of SOAP through empirical experiments on the leading open-source and closed-source models, showing SOAP can substantially optimize both execution time and memory consumption.", "questions": "1) In this paper, SOAP optimizes both execution time and memory usage simultaneously. However, in real-world scenarios, people might prioritize a single objective, either time or space. Do you think it is possible for SOAP to include optimization options (similar to -O3 or -Oz in GCC) to cater to different preferences?\n\n2) The iterative self-optimization process might increase the risk that the final generated code is either non-executable or deviates from the original problem specifications. Do you have any mechanisms in place to prevent such deterioration?\n\n3) How does the token usage in the SOAP profile optimization process compare to that in direct code generation (one-shot)?", "limitations": "The authors addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1) This paper proposes a novel general framework to optimize the efficiency of code generated by LLMs. It utilizes execution overhead profiles for self-diagnosis and iteratively enhances code efficiency.\n\n2) Empirical experiments conducted on various leading LLMs demonstrate that this framework effectively optimizes both the execution time and memory consumption of LLM-generated code.\n\n3) The framework does not require additional model training or tuning, making it flexible enough to be attached to arbitrary LLMs as a standalone pipeline.\n\n4) The paper is well-written; the methodology introduced is concise, and the experimental results are robust and clear.", "weaknesses": "1) The multi-iteration optimization method requires large token overhead, which can be both resource-intensive and time-consuming.\n\n2) Overhead profiles serve as a good indicator for local code optimization. Nevertheless, the framework may struggle with the scalability of extended code global optimization due to the limited context window of LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n\n\n**W1 & W2 & Q3: The multi-iteration optimization method requires a large token overhead.**\n\n\nTo address the reviewer's concern about overhead and context window limitations, we provide detailed metrics on execution time, token usage, per iteration input/output token usage， and efficiency in the Table below:\n\n\n| Model | Execution Time (s) | Token Usage (million) | Per Iteration Input/Output Task Token Usage (K) |ET(s)|NET|MU(Mb)|NMU|TMU(Mb*s)|NTMU|\n|-------|-------------------|----------------------|--------------------------------------------------|---|---|---|---|---|---|\n|InitialVersion|76.81|1.1|1.1|0.36|2.50|91.25|2.45|157.50|19.75|\n|UnsupervisedSelf-Refine|416.81|3.9|1.16|0.32|2.46|78.39|2.12|312.99|42.42|\n|Result-AwareSelf-Refine|419.24|3.9|1.16|0.30|2.25|58.65|1.61|195.49|27.16|\n|LineProfiler|555.68|4.7|1.49|0.33 |2.34 |36.43 |0.99|14.07 |1.81|\n|MemoryProfiler|536.26|4.7|1.49|0.34 |2.40 |36.85 |1.00 |16.34 |2.10|\n|SOAP|566.67|5.4|1.78|0.28|2.01|36.08|0.99|12.43|1.64|\n\n\n*Table: Overhead of different code efficiency optimization methods for GPT-3.5-turbo.*\n\n\nWe can observe the following:\n\n\n- **Execution Time and Token Overhead**: SOAP requires approximately 8x more execution time compared to the Initial Version. However, this overhead is justified as the optimized code resulting from SOAP significantly reduces the execution time and memory usage for real-world software that could be executed millions of times. Besides, while the optimization process itself is resource-intensive, it yields substantial efficiency gains in deployment scenarios. For example, the average memory peak (MU) of SOAP-generated code **only requires 40%** compared with the initially generated code, which can help source code applied in memory-constrained environments, such as embedded systems or mobile devices. Furthermore, the reduced memory footprint and improved execution speed of the optimized code can lead to better overall system performance, especially in scenarios where the software is frequently used or runs on resource-limited hardware. As a result, the upfront computational cost of the optimization process is offset by the long-term benefits of more efficient and lightweight code in real-world applications.\n \n- **Context Window and Scalability**: SOAP requires **an average of 1.78K tokens** for each input+output task iteration. Given that existing LLMs such as GPT-3.5 and GPT-4 have context windows of 4K tokens or more, they can effectively handle the global profiler information provided by SOAP. This capability allows the LLMs to understand comprehensive profiling data and perform global code optimizations, addressing concerns about scalability within the context window limitations.\n\n\nBy leveraging the available context window efficiently and optimizing the code based on detailed profiling data, SOAP manages to enhance the performance and scalability of the generated code, making it a valuable tool despite its initial overhead.\n\n\n**Q1: Include optimization options (similar to -O3 or -Oz in GCC) to cater to different preferences.**\n\n\nThanks for your suggestion. As shown in Table 3 in the paper, the memory profiler and line profiler focus on the memory usage and execution time individually. To specify the optimization options, developers can directly revise Lines 508-509 in the [Link](https://anonymous.4open.science/r/SOAP-FF6C/src/code_efficiency_calculator.py) to specify the optimization object (e.g., execution time and memory usage). In our final version, we will refine our source code argparse to provide optimization options for SOAP in the version.\n\n\n**Q2: Mechanisms to prevent code deterioration?**\n\n\nTo mitigate this risk and ensure the reliability of our experiments, we have implemented a robust two-step verification process:\n\n\n- **Initial Verification**: During the optimization process, if the refined code fails to pass the public test cases, we revert to using the initial version of the LLM-generated code for that specific task. This ensures that any code considered in our experiments at least meets the baseline correctness as defined by the public test cases.\n\n\n- **Comprehensive Filtering**: In cases where the code passes the public test cases but fails the private test cases, we exclude the entire task from our experiments for that respective model. Consequently, we do not calculate the optimized code's efficiency for any step of that task. For example, if the code optimized in step 3 is correct but the code in step 4 is incorrect for a specific task, we exclude all steps of that task from our efficiency calculations for the corresponding model.\n\n\nBy adhering to this two-step process, we ensure that only fully verified and correct code is included in our evaluations. This approach allows us to provide a fair and accurate comparison of code efficiency across different optimization steps and models, thereby preventing any deterioration in code quality through the iterative optimization process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PLIZKmChjn", "reviewer": "Reviewer_pc3i", "summary": "The paper aims to address the inefficiency issue of code generated by LLMs in terms of execution time and memory consumption. The authors propose a framework called Self-Optimization based on OverheAd Profile (SOAP), which improves the efficiency of LLM-generated code by iteratively refining it based on execution overhead profiles. These profiles capture the code's execution time and memory usage, which are then fed back into the LLM to optimize the code. The paper demonstrates the effectiveness of SOAP through extensive experiments on various benchmarks and selected LLMs. Results show reductions in execution time and memory usage, highlighting SOAP's ability to enhance code efficiency without compromising correctness.", "questions": "* How is SOAP compared to more reasonable baselines below:\n  1. Prompting an LLM to generate efficient code in zero-shot\n  2. Ask the LLM to refine its first generation for code performance\n  3. Fewshot prompting mentioned in PIE\n* Are profilers necessary? What if we just replace profilers with self-reasoning or self-reflection?\n* How's the cost (in the form of token consumption and pipeline running time) of SOAP and baselines?\n* How is the overall variation of mentioned metrics including execution time? For example, when running and measuring the code execution for N times, what is the overall coefficient of variation?", "limitations": "* Using a profiler and code executor in the pipeline is expensive with respect to the degree of improvement. In production, integrating code execution in a sandbox is already challenging, and integrating a profiler (for multiple languages) would take more engineering cost. Meanwhile, the performance gain (e.g., execution time) overall looks minimal compared to the baseline whose prompt does not even encourage code optimization. \n* Overall the technical contribution of the paper is incremental: compared to prior techniques such as ALGO and PIE, the major difference is the use of code profiler.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 4, "contribution": 2, "strengths": "* The paper is well presented and the targeted problem of code inefficiency is well motivated.\n* The evaluation covers a good number of Python coding tasks and LLMs.\n* The authors mentioned that the code of the work will be open-sourced.", "weaknesses": "* The baseline selection can be largely improved. The current baseline in the paper is to ask LLMs to directly generate code -- it is not the models' fault to generate non-optimal code when they are not asked to. At least two reasonable and simple baselines should have been included: (1) ask the model to generate an efficient version of code, and (2) ask the model to generate code and then directly optimize it (similar to the ALGO approach, despite not using test generation and execution to soften the environmental assumptions). Furthermore, the PIE [1] paper mentioned various few-shot prompting methods to perform code optimization, which should also be used as baselines.\n* To perform an \"apple-to-apple\" comparison over the models, Tables 1-4 should be better aligned. To understand which model is better at generating efficient code, they must be compared under the same set of tasks. While due to correctness inconsistency, the attributed coding tasks are not aligned, authors may compare the models under a set of mutually passing coding tasks.\n* Efficiency evaluation can be noisy without carefully configuring the metrics and testbeds. The considered metrics such as execution time and memory usage are flaky and non-reproducible as they are measured in physical platforms. To mitigate these limitations, authors may follow PIE [1] to use architecture simulators or other better-defined metrics.\n* While the author highlighted efficiency is critical in resource-constrained scenarios, the proposed technique is specialized for Python. It is unclear if Self-Optimization can work for performance-demanding languages such as C/C++ and related experiments are ignored.\n* [1] is a closely relevant and published work that should be well discussed and compared.\n\n[1] Shypula A, Madaan A, Zeng Y, Alon U, Gardner J, Hashemi M, Neubig G, Ranganathan P, Bastani O, Yazdanbakhsh A. “Learning Performance-Improving Code Edits.” ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:\n\n**W1 & Q1 & W5: Baseline selection can be largely improved.**\n\nThanks for your suggestion. We provide the evaluation results of SOAP and suggested baselines in **Author Rebuttal Table 1**. \n\n- For the first suggested method, we named it **Directly Efficiency**, where LLMs are required to generate efficient code for the task description. \n\n- For the second suggested method, we name it **Self-Refine Efficiency**, where we ask LLMs to generate code and then directly optimize it. \nThen, since PIE fine-tunes LLMs to improve the efficiency of LLM-generated code, which is orthogonal with SOAP, we conduct experiments for PIE and PIE + SOAP for five different prompts to demonstrate that with SOAP, PIE fine-tuned LLMs generated code could be further improved. \n\nAs shown in **Author Rebuttal Table 1**, we can observe that SOAP achieves SOTA results compared with Directly Efficiency and Self-Refine Efficiency. For example, Directly Efficiency and Self-Refine Efficiency decreases NTMU from 19.65 to 2.88 and 2.65, while SOAP decreases from 19.65 to 1.64. Besides, we can observe that with SOAP, the efficiency of PIE-generated code would further improve, e.g., decreasing the average execution time from 0.74s to 0.41s for PIE+Dynamic Retrieval,k=4.\n\n\n**W2: Compare the models under a set of mutually passing coding tasks.**\n\nThanks for your suggestion. The evaluation results are shown in **Author Rebuttal Table 2**, where we conduct experiments on seven closed-source LLMs with 210 tasks, which have higher pass@1 compared with open-source LLMs, and then can collect massive tasks addressed by all closed-source LLMs. As shown in Table 8, we can observe that SOAP with GPT-3.5-turbo-0301 achieves SOTA efficiency compared with other LLMs. For example, the average execution time decreases from 0.37s to 0.28s.\n\n\n**W3: Follow PIE to use architecture simulators or other better-defined metrics.**\n\nWe provide the evaluation results of SOAP and the initial code in **Author Rebuttal Table 3**. We can observe that in the PIE-provided default simulator, SOAP can also improve the efficiency of LLM-generated initial code. For example, the average execution time of OpenCodeInterpreter-1.3B decreases from 0.80s to 0.52s.\n\n\n**W4: Unclear if Self-Optimization can work for performance-demanding languages such as C/C++.**\nTo address the reviewer's concern, we conduct experiments for SOAP in the HumanEval-ET (C++) dataset. Our evaluation results are shown below, where we can observe that **SOAP can also improve the efficiency of LLM-generated C++ programs**. For example, the average execution time of CodeLlama-70B-Instruct-hf decreases from 667.9ms to 459.5ms.\n\n| Evaluation in HumanEval (C++)     | ET (ms) | NET  | MU (KB) | NMU  | TMU (KB*ms) | NTMU |\n|------------------|--------|------|---------|------|------------|------|\n| CodeLlama-70b-Instruct-hf       |        |      |         |      |            |      |\n| Initial Version                 | 667.9  | 1.5  | 93.2    | 1.3  | 58.9       | 2.1  |\n| SOAP                            | 459.5  | 1.0  | 72.2    | 1.0  | 34.1       | 1.3  |\n| gpt-3.5-turbo-0301              |        |      |         |      |            |      |\n| Initial Version                 | 668.1  | 1.5  | 78.9    | 1.1  | 79.0       | 2.6  |\n| SOAP                            | 577.3  | 1.2  | 71.5    | 1.0  | 63.8       | 2.1  |\n\n**Q2: Are profilers necessary? What if we just replace profilers with self-reasoning or self-reflection?**\n\n\nProfilers are necessary for the code efficiency optimization process. Specifically, to demonstrate the importance of profilers, we conduct experiments for straightforward code optimization (Unsupervised Self-Refine and Result-Aware Self-Refine), reasoning-based code optimization (self-reasoning and self-reflection), single profiler-based code optimization, and provide both memory profiler and execution time profiler (SOAP). \n\nThe evaluation results are shown below. Our experiments demonstrate that by providing both memory profiler and execution time profiler (SOAP), the code efficiency achieves SOTA results compared with other methods. For example,  SOAP decreases the average execution time from 0.36s to 0.28s.\n\n\n|OptimizationProfile|ET(s)|NET|MU(Mb)|NMU|TMU(Mb*s)|NTMU|\n|---|---|---|---|---|---|---|\n|**GPT-3.5-Turbo-0301**|||||||\n|InitialVersion|0.36|2.50|91.25|2.45|157.50|19.75|\n|Unsupervised Self-Refine|0.32|2.46|78.39|2.12|312.99|42.42|\n|Result-Aware Self-Refine|0.30|2.25|58.65|1.61|195.49|27.16|\n|Self-Reasoning|0.89|6.21|60.64|1.62|45.91|5.61|\n|Self-Relfection|0.81|5.67|60.64|1.62|39.35|4.80|\n|SOAP|0.28|2.01|36.08|0.99|12.43|1.64|\n\n\n**Q3: How's the cost (token consumption and running time) of SOAP and baselines?**\n\nTo address the reviewer’s concern about the cost of SOAP and baselines, we provide the evaluation results in **Author Rebuttal Table 4**, where we can observe that the initially generated code requires 1.1M token usage and 76.81s for all tasks. While SOAP requires 5.4M token usage and 566.67s for all tasks. Although SOAP requires more tokens compared with the initial version, the execution time and memory usage (See paper table 1) are largely improved. We believe that this exact token usage is affordable.\n\n\n**Q4: When running and measuring the code execution for N times, what is the overall coefficient of variation?**\n\nThe overall coefficient of variation (CoV) for each model and metric in the SOAP benchmark is presented in **Author Rebuttal Table 5**, which is calculated by running and measuring the code execution 5 times for each model, then computing the mean and standard deviation of each metric. From the table, we can observe that all of the metrics' coefficients of variation are lower than 3% for all models, suggesting that the **performance of these models is relatively consistent across multiple runs**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Tables included in the rebuttal to address the comments made by Reviewer H3VZ, Reviewer VC11, and Reviewer pc3i 

|OptimizationProfile|ET(s)|NET|MU(Mb)|NMU|TMU(Mb*s)|NTMU|
|---|---|---|---|---|---|---|
|**GPT-3.5-Turbo-0301**|||||||
|InitialVersion|0.36|2.50|91.25|2.45|157.50|19.75|
|UnsupervisedSelf-Refine|0.32|2.46|78.39|2.12|312.99|42.42|
|Result-AwareSelf-Refine|0.30|2.25|58.65|1.61|195.49|27.16|
|Self-Edit|0.42|3.67|59.86|1.65|24.87|3.28|
|Critic|0.60|4.25|102.75|2.82|351.91|46.39|
|DirectlyEfficiency|0.43|3.03|59.11|1.67|20.37|2.88|
|Self-RefineEfficiency|0.40|2.83|59.11|1.67|18.80|2.65|
|IsSelf-Refine|0.40|2.88|61.83|1.81|36.29|5.69|
|Self-Reasoning|0.89|6.21|60.64|1.62|45.91|5.61|
|Self-Relfection|0.81|5.67|60.64|1.62|39.35|4.80|
|SOAP|0.28|2.01|36.08|0.99|12.43|1.64|
|**CodeLlama7B(PIE:HQ+SelfPlay)**|||||||
|PIE+Zero-Shot|0.87|5.73|74.83|1.81|109.29|9.69|
|PIE+SOAP+Zero-Shot|0.79|5.41|65.78|1.68|89.90|7.84|
|PIE+Few-Shot|0.82|5.58|73.57|1.74|98.02|8.92|
|PIE+SOAP+Few-Shot|0.41|2.97|73.10|1.74|59.69|5.09|
|PIE+CoT|0.79|5.14|73.14|1.74|63.93|5.35|
|PIE+SOAP+COT|0.45|2.84|71.15|1.71|58.06|4.77|
|PIE+DynamicRetrieval,k=4|0.74|5.36|68.64|1.51|85.24|7.78|
|PIE+SOAP+DynamicRetrieval,k=4|0.41|3.36|68.63|1.51|52.34|4.52|
|**Supersonic**|||||||
|Supersonic|1.40|10.33|113.06|3.18|329.59|56.24|
|Supersonic+SOAP|1.34|9.91|102.26|2.87|267.47|45.64|

*Author Rebuttal Table 1: Evaluation results of SOAP and baselines. Since the finetuned link for GPT-3.5-turbo from PIE is not available, we use the finetuned CodeLlama 7B for experiments.*

|Steps|ET(s)|NET|MU(Mb)|NMU|TMU(Mb*s)|NTMU|
|-------|--------|-----|---------|-----|------------|------|
|**gpt-3.5-turbo-0301**|||||||
|InitialVersion|0.37|3.10|66.91|1.90|20.89|6.32|
|SOAP|0.28|2.31|67.31|1.92|15.66|4.71|
|**gpt-3.5-turbo-0613**|||||||
|InitialVersion|0.37|3.05|66.99|1.90|20.92|6.18|
|SOAP|0.32|2.69|67.09|1.91|18.40|5.45|
|**gpt-3.5-turbo-1106**|||||||
|InitialVersion|0.37|3.07|66.94|1.90|20.78|6.22|
|SOAP|0.32|2.72|66.98|1.91|18.29|5.54|
|**gpt-4**|||||||
|InitialVersion|0.37|3.06|66.91|1.90|21.17|6.17|
|SOAP|0.32|2.69|66.97|1.91|18.42|5.36|
|**gpt-4-turbo-preview**|||||||
|InitialVersion|0.37|3.10|66.92|1.90|20.78|6.28|
|SOAP|0.32|2.67|67.01|1.91|18.65|5.42|
|**claude-3-haiku**|||||||
|InitialVersion|0.39|3.27|66.90|1.90|22.52|6.68|
|SOAP|0.32|2.65|67.02|1.91|18.83|5.42|
|**claude-3-sonnet**|||||||
|InitialVersion|0.38|3.20|66.93|1.90|21.52|6.55|
|SOAP|0.32|2.65|66.95|1.91|18.30|5.43|

*Author Rebuttal Table 2: Evaluation results for five different LLMs on identical tasks verified by correct execution (210 tasks).*

|Steps|ET(s)|NET|MU(Mb)|NMU|TMU(Mb*s)|NTMU|
|-------|--------|-----|---------|-----|------------|------|
|**OpenCodeInterpreter-1.3B**|||||||
|InitialVersion|0.80|5.74|62.60|1.61|45.18|5.63|
|SOAP|0.52|3.73|48.73|1.25|29.93|3.73|
|**deepseek-coder-1.3b-instruct**|||||||
|InitialVersion|0.63|4.57|59.32|1.64|27.57|5.49|
|SOAP|0.43|3.10|45.80|1.27|17.44|3.47|
|**CodeLlama-7b-Instruct-hf**|||||||
|InitialVersion|3.98|23.93|103.78|2.19|370.74|28.06|
|SOAP|3.41|20.50|89.37|1.88|335.53|25.39|
|**CodeLlama-70b-Instruct-hf**|||||||
|InitialVersion|0.90|7.34|50.69|1.91|46.93|21.07|
|SOAP|0.63|5.18|41.44|1.56|34.90|15.67|
|**starcoder2-15b**|||||||
|InitialVersion|0.49|1.88|132.27|1.17|53.54|1.18|
|SOAP|0.22|0.86|66.18|0.59|25.24|0.56|
|**gpt-3.5-turbo-0301**|||||||
|InitialVersion|0.68|4.83|102.39|2.82|302.04|39.98|
|SOAP|0.39|2.77|42.67|1.18|18.57|2.46|
|**claude-3-sonnet**|||||||
|InitialVersion|0.74|5.20|64.02|1.76|56.85|7.29|
|SOAP|0.51|3.60|46.88|1.29|27.24|3.49|

*Author Rebuttal Table 3: Evaluation results of SOAP and baselines in PIE provided default simulator. We use the default simulator provided by PIE for our experiments.*

|Model|ExecutionTime(s)|TokenUsage(million)|PerIterationInput/OutputTaskTokenUsage(K)|
|-------|-------------------|----------------------|--------------------------------------------------|
|InitialVersion|76.81|1.1|1.1|
|UnsupervisedSelf-Refine|416.81|3.9|1.1602|
|Result-AwareSelf-Refine|419.24|3.9|1.1602|
|LineProfiler|555.68|4.7|1.4888|
|MemoryProfiler|536.26|4.7|1.4888|
|SOAP|566.67|5.4|1.7805|

*Author Rebuttal Table 4: Overhead of different code efficiency optimization prompt engineering methods for GPT-3.5-turbo.*

|Steps|CoV of ET(%)|CoV of NET(%)|CoV of MU(%)|CoV of NMU(%)|CoV of TMU(%)|CoV of NTMU(%)|
|-------|---------------|----------------|---------------|----------------|----------------|-----------------|
|OpenCodeInterpreter-DS-1.3B|2.5|2.5|2.5|2.5|2.5|2.5|
|deepseek-coder-1.3b-instruct|1.7|1.7|1.7|1.7|1.7|1.7|
|CodeLlama-7b-Instruct-hf|0.3|0.3|0.3|0.3|0.3|0.3|
|CodeLlama-70b-Instruct-hf|1.7|1.7|1.7|1.7|1.7|1.7|
|starcoder2-15b|1.6|1.6|1.6|1.6|1.6|1.6|
|gpt-3.5-turbo-0301|0.6|0.6|0.6|0.6|0.6|0.6|
|claude-3-sonnet|0.2|0.2|0.2|0.2|0.2|0.2|

*Author Rebuttal Table 5: Overall coefficient of variation from five executions of SOAP for each metric and model in EffiBench. We conducted the experiments five times, then calculated the mean and std for each metric and model.*

| Steps | ET (s) | NET | MU (Mb) | NMU | TMU (Mb*s) | NTMU |
| --- | --- | --- | --- | --- | --- | --- |
|**OpenCodeInterpreter-1.3B**|||||||
|0|1.60|1.52|38.91|1.00|89.16|1.11|
|1|1.60|1.51|38.91|1.00|88.77|1.11|
|2|1.59|1.51|38.91|1.00|88.16|1.10|
|3|1.59|1.50|38.91|1.00|87.85|1.09|
|4|1.59|1.50|38.91|1.00|87.77|1.09|
|5|1.29|1.23|38.91|1.00|70.63|0.88|
|**DeepSeek-1.3B-Ins**|||||||
|0|1.42|1.32|36.04|1.00|40.61|1.12|
|1|1.42|1.23|36.04|1.00|39.30|1.09|
|2|1.42|1.20|36.04|1.00|37.99|1.05|
|3|1.15|1.16|36.04|1.00|37.33|1.03|
|4|1.15|1.12|36.04|1.00|36.43|1.00|
|5|1.15|1.07|36.04|1.00|35.48|0.98|
|**CodeLlama-7B**|||||||
|0|4.70|3.68|46.76|0.99|212.41|1.93|
|1|4.59|3.58|38.94|0.82|165.16|1.50|
|2|4.56|3.55|39.11|0.82|162.16|1.47|
|3|4.56|3.56|39.03|0.82|161.98|1.47|
|4|4.56|3.55|39.03|0.82|161.57|1.47|
|5|4.52|3.54|38.67|0.82|157.76|1.43|

*Author Rebuttal Table 6: Evaluation results of SOAP for different optimization step.*

Meta-review: This paper has a very wide spread of scores from 3 (reject) to 8 (strong accept).
After reading the paper, reviews, rebuttal, and substantial discussions between authors and reviewers, I believe that the paper should be accepted.

There has been extended rebuttals by authors with numerous additional results as well as additional comments and questions by reviewers. I believe the key issues are:

1. Is the contribution novel enough? Adding profiling results to LLM prompt to improve the code efficiency is novel. Reviewers are split whether the novelty is significant. I would rate novelty by itself in the weak accept area.
2. Is the evaluation done well? I believe that authors have improved the evaluation significantly during the rebuttal process. I agree with reviewer VC11 that "the current evaluation of SOAP adequately demonstrates the effectiveness of the work". However, I acknowledge the reviewers who pointed out somewhat questionable results in rebuttal. I am also somewhat concerned on how the authors are going to integrate all the additional results from rebuttal into the paper and whether the evaluation will be presented well. I would rate the evaluation also at the level of weak accept (I might change it to higher if the evaluation was well integrated into the paper.)
3. Functional Correctness: Some reviewers expressed concerns about the impact on functional correctness arguing that the SOAP method might introduce semantic degradation in code correctness after refinement. Authors have shown minimal correctness degradation in the benchmarks suggested by the concerned reviewers. In my opinion the correctness degradation may occur, but it is not a blocker for adopting SOAP like techniques.

Based on my evaluation of these issues - taking into account all the reviews, rebuttal, and discussions - I propose we accept the paper.
Decision: Accept (poster)