EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization

Authors: dong  huang
jianbo  dai
han  weng
puzhen  wu
yuhao  qing
heming  cui
zhijiang  guo
jie m zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 14:25:28

Abstract: Large language models (LLMs) have shown remarkable progress in code generation, but their generated code often suffers from inefficiency, resulting in longer execution times and higher memory consumption. To address this issue, we propose EffiLearner, a self-optimization framework that utilizes execution overhead profiles to improve the efficiency of LLM-generated code. EffiLearner first generates code using an LLM, then executes it locally to capture execution time and memory usage profiles. These profiles are fed back to the LLM, which then revises the code to reduce overhead. To evaluate the effectiveness of EffiLearner, we conduct extensive experiments on EffiBench and two commonly used code generation benchmarks with 16 open-source and 6 closed-source models. Our evaluation results demonstrate that through iterative self-optimization, EffiLearner significantly enhances the efficiency of LLM-generated code. For example, the execution time (ET) of StarCoder2-15B for the EffiBench decreases from 0.93 (s) to 0.12 (s) which reduces 87.1\%  execution time requirement compared with the initial code. The total memory usage (TMU) of StarCoder2-15B also decreases from 22.02 (Mb*s) to 2.03 (Mb*s), which decreases 90.8\% total memory consumption during the execution process.

Main paper: 
 introduction: 
 Large language models (LLMs) have recently achieved significant advancements across various tasks [49, 4, 5, 42] . LLMs such as GPT-4 [50] and Copilot [44] have demonstrated considerable efficacy in code-related applications, including code completion [11, 6] , debuggingg [25, 12] , and translation [54, 1] . These innovative tools have been seamlessly integrated into popular development environments, significantly augmenting developer productivity by providing intelligent codeProblem: Given a non-empty array of integers nums, every element appears twice except for one. Find that single one. You recommendations based on natural language instructions. However, the primary focus of existing efforts has predominantly been on the correctness of the generated code [18] , ensuring it meets the functional requirements and adheres to syntactical norms.Despite advancements in ensuring code correctness, there remains a significant gap in the literature regarding the efficiency of code produced by LLMs. Efficiency is crucial as it translates to faster execution and lower memory and processing power consumption, which is especially important in resource-constrained environments such as mobile devices or embedded systems [18, 53, 13, 48, 17, 56, 9, 41, 52] . Recent studies [56, 48] reveal that LLM-generated code often exhibits lower efficiency in terms of execution time and memory usage when compared to canonical solutions. For instance, on a benchmark that evaluates efficiency, EffiBench [27] , even the most powerful LLM (e.g., GPT-4-Turbo) generates code with suboptimal efficiency. The average and worst-case execution times are 1.69 and 45.49 times longer than those of the canonical solutions, respectively. This inefficiency underscores the need for developing new methods focused on evaluating and improving the efficiency of code generated by LLMs, ensuring that they produce correct and highly efficient code.To bridge this gap, we draw inspiration from the methodology used by coders on coding platforms. When addressing a programming problem, coders typically write an initial program that is executable on the test cases. Next, they execute the code and obtain a profile of the execution time and memory usage overhead, as shown in Figure 1 . Based on this overhead profile, the coder optimizes the code to enhance its efficiency. During this process, the coder extracts key information (e.g., execution times and memory usage of each line) from the overhead profile, which helps identify lines or operators that require significant overhead (e.g., loops that execute multiple times or unnecessary variables being saved). This information assists the coder in optimizing their code.With this motivation, we propose EFFI-LEARNER to improve the efficiency of LLM-generated code. As shown in Figure 2 , EFFI-LEARNER first requires the LLM to generate code based on the task description. Then, EFFI-LEARNER executes the generated code locally and captures the execution time and memory usage profile. These overhead profiles are fed back into the LLM, which then revises the code to reduce the overhead. Through multi-iteration self-optimization, the efficiency of the LLM-generated code is improved. While it's true that the iterative process requires extra time, it's crucial to recognize the long-term advantages that come with this investment. By optimizing the code, we can enhance the overall efficiency once it's deployed.To evaluate the effectiveness of EFFI-LEARNER, we conduct extensive experiments on the EffiBench and two commonly used code generation benchmarks (i.e. HumanEval [11] and MBPP [6] ) with 16 open-source and 6 closed-source models. We compare the efficiency of the code generated by the LLM before and after applying EFFI-LEARNER. The experimental results demonstrate that EFFI-LEARNER significantly improves the efficiency of the LLM-generated code. For example, the execution time (ET) of StarCoder2-15B decreases from 0.93 (s) to 0.12 (s) which reduces 87.1% execution time requirement compared with the initial code. The max memory usage (MU) of DeepSeek-6.7B-Ins also decreased from 259.73 (Mb) to 36.97 (Mb), which reduces 85.8% max memory consumption for the code execution requirement. The total memory usage (TMU) of StarCoder2-15B also decreases from 22.02 (Mb*s) to 2.03 (Mb*s), which decreases 90.8% total memory consumption during the execution process.2 Related Work 
llms for code generation: 
 The increasing popularity of LLMs for code generation has coincided with the growing availability of open-source code repositories and the need to boost developer productivity. Initial efforts focused on training models specifically for coding tasks, such as CodeT5 [61] , AlphaCode [33] , CodeGen [47] , InCoder [19] , StarCoder [32] , SantaCoder [3] and DeepSeek Coder [15] . Contrastingly, models such as Codex [11] and CodeLLaMA [55] represent a subsequent stride, being finetuned from foundation models [8, 59] . These code LLMs have been applied to various tasks, including code generation [11, 14] , program repair [25, 28] , automated testing [31, 16] , code translation [54, 1] , type prediction [45, 64] , and code summarization [26, 2] . Among these, code generation, where models generate code snippets based on natural language descriptions or docstrings, has become a critical domain for evaluating LLMs. While LLMs have achieved impressive results in code generation tasks like HumanEval [11] and MBPP [6] , their efficiency has received less attention. Recent studies [56, 27, 48] have shown that LLM-generated code exhibits lower efficiency in execution time and memory usage compared to canonical solutions. These findings highlight the need for further research and development to improve the efficiency of LLM-generated code. In this work, we propose the first method that significantly improves the efficiency of code generated by a wide range of LLMs.A prevalent strategy for improving the behavior of LLMs is learning from feedback, mirroring human learning where individuals refine their actions through trial, error, and correction [7, 43] . Early efforts involve using human feedback to evaluate and refine models [30, 51, 21] . To minimize human intervention, another strategy focuses on automated feedback. These methods iteratively learn from automatically generated feedback signals, understanding the consequences of their actions and adapting their behaviors. The source of this automated feedback can be diverse, ranging from the LLM itself [40, 57] , external tools [22, 37] or verifiers [36] , external knowledge sources [20, 68] and even generation logits [67] . In code generation, the program executor is frequently used as a source of feedback for refining the model's initial code. For example, Self-Edit [69] and Self-Evolve [29] execute the initial program on example test cases and provide the execution results as feedback, prompting the LLM to refine the code. Self-Debug [12] explores using program explanation, unit tests, and program interpreters as feedback types. ALGO [70] employs a more fine-grained approach by generating a reference oracle program that solves the problem with an exhaustive search. Feedback is then collected by comparing the generated outputs with the oracle. While existing work primarily focuses on using feedback to edit the initial code to ensure correctness, our method explores using overhead profiles to improve the efficiency of the code. 
effi-learner: 
 Inspired by the optimization strategies employed by human coders on coding platforms, we propose a framework EFFI-LEARNER to enhance the efficiency of LLM-generated code. Human coders typically analyze execution time and memory usage profiles to identify bottlenecks and optimize their code. EFFI-LEARNER leverages this principle by integrating a self-optimization loop into the code generation process. As illustrated in Figure 2 , EFFI-LEARNER consists of three main components: Code Generation, Overhead Profiling, and Code Refinement, each playing a crucial role in the self-optimization process. Given a task description or code generation requirement, the LLM generates an initial version of the code. The LLM takes the task description as input and produces code that aims to solve the task.The generated code is executed locally to capture its execution time and memory usage overhead profiles. During this step, the code is run on a set of open test cases, and the execution time and memory usage for each line of code are recorded. This information forms the overhead profiles that provide insights into the efficiency of the generated code.Execution Time Profiling In this step, we measure the execution time of each line of code to identify potential bottlenecks and inefficiencies. To perform execution time profiling, we utilize the line_profiler library in Python. During the profiling process, we run the generated code on a set of open test cases provided by the dataset. The line_profiler library tracks the execution time of each line of code for all the test cases combined. This helps us assess the code's performance under different conditions and identify any performance bottlenecks. The execution time profiling results are reported based on the total consumption for all open test cases. The profiling output includes information such as the line number, the number of times each line is executed, and the total time spent on each line. These profiles serve as input for the subsequent code refinement step.Memory Usage Profiling Memory usage profiling is another essential aspect of the EFFI-LEARNER framework. It helps us understand how the generated code utilizes memory resources and identifies any memory-related inefficiencies or leaks. To profile memory usage, we employ the memory_profiler library in Python. During the memory usage profiling process, we run the generated code on the set of open test cases. The memory_profiler library monitors the memory usage of each line of code throughout the execution of all the test cases combined. It captures the memory usage at different points, such as before and after function calls, loop iterations, and memory allocation statements. The memory usage profiling results are reported based on the total consumption for all open test cases. The profiling output includes information such as the line number and the corresponding memory usage. These profiles provide valuable insights into the memory efficiency of the generated code and help identify areas for optimization.This component leverages the execution time and memory usage profiles to optimize the generated code. In this step, the LLM analyzes the overhead profiles to refine the code for better efficiency. To enable self-optimization, we feed the overhead profiles back into the LLM along with the generated code. The LLM analyzes patterns in the overhead profiles, such as high execution time or excessive memory usage, and correlates them with specific code segments. It then applies optimization techniques, such as loop unrolling, memorization, data structure optimization, algorithm substitution, and code simplification, to improve the efficiency of the identified code segments.During the self-optimization process, the LLM considers factors such as the impact of each optimization on the overall efficiency, the trade-offs between execution time and memory usage, and the preservation of code correctness. It aims to strike a balance between performance improvement and maintaining the functional integrity of the code. The LLM iteratively refines the code based on the overhead profiles, applying optimizations until a satisfactory level of efficiency is achieved or a predefined number of iterations is reached. The optimized code is then validated against the open test cases to ensure its functional correctness. By leveraging the execution time and memory usage profiles, the self-optimization step enables the LLM to improve the efficiency of the generated code.We carefully design prompts to guide LLMs in optimizing code efficiency while ensuring the optimized code passes predefined test cases. The prompt template (Figure 3 ) used in EFFI-LEARNER's self-optimization stage includes a task description, test case, initial code, overhead analysis, and optimization rules. The task description provides context and requirements, the test case ensures correctness, and the initial code is the starting point for optimization. The overhead analysis highlights performance metrics and areas for improvement, while the optimization rules focus the LLM on enhancing efficiency, encapsulating the optimized code, and excluding the test case from the code block. This comprehensive prompt equips the LLM with the necessary information to effectively optimize code, maintain consistency across models and tasks, and facilitate comparison of their code optimization capabilities, advancing the field of LLM-driven code optimization. Details of the template can be found in Appendex A.3. 
dataset and metrics: 
 We evaluate EFFI-LEARNER on EffiBench [27] . Following their settings, we use Execution Time (ET), Normalized Execution Time (NET), Max Memory Usage (MU), Normalized Max Memory Usage (NMU), Total Memory Usage (TMU), and Normalized Total Memory Usage (NTMU) as metrics. We provide a detailed definition of these metrics in A.5. Following the setup of EffiBench, evaluation metrics were only calculated on the tasks that generated code for both the initial version and EFFI-LEARNER optimized code that can pass all private test cases provided by the dataset For open-source models, we evaluate EFFI-LEARNER with OpenCodeInterpreter (1.3B, 6.7B, and 33B) [72] , DeepSeek-Instruct (1.3B, 6.7B, and 33B) [24] , CodeLlama (7B, 13B, 34B, and 70B) [55] , XwinCoder (7B and 34B) [58] , StarCoder (3B, 7B, and 15B) [32] , and WizardCoder-13B [38] , where the detailed versions of LLMs are demonstrated in supplementary file. For closed-source models, we use GPT-3.5-Turbo, GPT-4-Turbo, GPT-4 [50] , Claude-3-haiku, and Claude-3-sonnet 5 . These LLMs have achieved competitive pass@1 scores in various code generation tasks [11, 6, 35] .Setup We first collect the generated code from each LLM and evaluate its correctness using open test cases. Only the code that passes all test cases is considered for efficiency evaluation. This approach ensures consistency in the evaluated tasks across different self-optimization iterations, as EFFI-LEARNER focuses on improving the efficiency of initially correct code without altering its pass@1 score. By evaluating a diverse set of open-source and closed-source LLMs, we aim to provide a comprehensive assessment of the efficiency of LLM-generated code and the effectiveness of EFFI-LEARNER in improving code efficiency across different models and architectures.Open-source LLMs As shown in Table 1 , we observe that the efficiency metrics for all models have been increased in most experiments once we apply EFFI-LEARNER to optimize the efficiency of LLM-generated code. For example, in OpenCodeInterpreter-1.3B, the execution time for its generated code decreases from 1.60 (s) to 1.29 (s), a reduction of 19.4% in execution time. Additionally, the The improvements in efficiency metrics across both open-source and closed-source LLMs highlight the generalizability and adaptability of EFFI-LEARNER. By iteratively refining the generated code based on efficiency profiler feedback, EFFI-LEARNER enables LLMs to produce more efficient code without compromising the correctness of the generated solutions. The consistent improvements across various models and architectures demonstrate the potential of EFFI-LEARNER as a model-agnostic approach for optimizing the efficiency of LLM-generated code in real-world applications.To investigate the impact of the number of self-optimization steps on the efficiency of the EFFI-LEARNER-optimized code, we conduct an ablation study by varying the number of steps from 0 to 5. Table 2 for CodeLlama-70B and GPT-3.5-Turbo-0301 at different self-optimization steps. Table 2 demonstrates the significant impact of the number of self-optimization steps on the efficiency of the EFFI-LEARNER-optimized code. For both CodeLlama-70B and GPT-3.5-Turbo-0301, the first self-optimization step yields the most substantial improvements in code efficiency. As the number of steps increases, the efficiency metrics continue to improve, albeit with diminishing returns. By the fifth step, the efficiency metrics reach their lowest values, demonstrating the effectiveness of EFFI-LEARNER's iterative self-optimization approach in enhancing the efficiency of LLM-generated code. The evaluation results highlight that the majority of efficiency improvements occur in the first few steps, with subsequent steps contributing to further refinements of the optimized code.To show the effectiveness of the overhead profile in guiding LLMs to refine their generated code, we compare the performance of EFFI-LEARNER with two alternative approaches: Unsupervised Self-Refine and Result-Aware Self-Refine [40, 57] . Unsupervised Self-Refine uses a prompt that directly requires the LLM to refine the code without providing additional information. Result-Aware Self-Refine feeds the ET, MU, and TMU, then requires the LLM to refine the code based on these metrics. Table 3 presents the code efficiency metrics for CodeLlama-70B and GPT-3.5-Turbo-0301 using different code refinement approaches.CodeLlama-70B Unsupervised Self-Refine and Result-Aware Self-Refine result in significant increases in ET, memory usage (MU), and TMU compared to the initial version. These results highlight the importance of the overhead profile feedback in guiding LLMs to generate more efficient code. Without the overhead profile, the code refinement process using alternative prompts fails to improve code efficiency and even leads to significant performance degradation. The overhead profile provides valuable insights into the resource consumption of the generated code, enabling LLMs to make targeted optimizations and achieve substantial efficiency improvements. Impact on correctness We provide the pass@1 of LLM-generated initial code and EFFI-LEARNER optimized code for EffiBench in Table 6 . We observe that the pass@1 of EFFI-LEARNER optimized code may be lower than LLM-generated initial code. The key reason is that during the self-optimization process, EFFI-LEARNER only uses public test cases to guide code efficiency optimization for correct initial code. However, since public test cases may not cover all edge cases in the private test cases (test cases used to evaluate pass@1 of LLMs), this can cause the pass@1 of EFFI-LEARNER generated code to be lower than the initial code. Nevertheless, we observe that the pass@1 of EFFI-LEARNER only decreases by about 0% to 0.5%, which means that only a few of the codes will be incorrect. As shown in Table 1 , we can observe that the code efficiency is largely increased. We believe that this minor decrease in pass@1 is worthwhile considering the significant efficiency gains.Case study To illustrate how EFFI-LEARNER improves the efficiency of LLM-generated code, we provide a case illustration in Appendix Figure 4 -Figure 11 . As shown in Figure 6 , we can observe that the execution time of the initial code is 23.59 (s) while in the self-optimized code, the execution time decreases from 23.59 (s) to 3.36 (s). The key reason is that in the initial code, the algorithm uses a standard unidirectional Breadth-First Search (BFS), which explores all possible states level by level starting from the initial state. This method results in a large number of states to explore, leading to significant computational overhead. In contrast, the self-optimized code employs a bidirectional BFS, which simultaneously searches from both the initial state and the target state. This reduces the search space by meeting in the middle, significantly decreasing the number of states that need to be explored and thereby improving the execution time.Error Analysis We also provide a case illustration to explain why some code efficiency does not improve significantly when EFFI-LEARNER is applied to LLM-generated code. As shown in Appendix Figure 12 -Figure 18 , we observe that the initial code only requires 0.0012 (s) to execute, while in the optimized code, the execution time is still 0.0011 (s). The key reason for this minimal improvement is that both implementations already operate with the same theoretical time complexity of O(log(min(m, n))). Given the problem's constraints and small input sizes, the actual runtime differences are overshadowed by the inherent efficiency of the binary search algorithm. Additionally, the overhead of function calls and Python runtime operations can further minimize the observed performance gains. Therefore, while the optimized code may offer clearer partition management and slight improvements, the overall efficiency remains largely unchanged due to the already optimized nature of the initial approach. 
conclusion: 
 This paper focuses on the critical issue of efficiency in code generated by LLMs. While LLMs have shown impressive capabilities in code generation, their output often suffers from suboptimal efficiency, leading to slower execution and higher resource consumption. To tackle this challenge, we propose EFFI-LEARNER, a novel self-optimization framework that leverages execution overhead profiles to guide LLMs in improving code efficiency. Extensive experiments and analysis demonstrate that EFFI-LEARNER significantly enhances the efficiency of LLM-generated code, achieving substantial reductions in execution time and memory usage. For future work, we would like to investigate the application of EFFI-LEARNER to other programming tasks and languages, as well as explore the potential benefits of incorporating domain-specific knowledge into the optimization process. 
acknowledgment: 
 The  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cite all evaluated models and datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Limitations EFFI-LEARNER presents a compelling solution for enhancing the efficiency of code generated by LLMs. However, it's crucial to acknowledge several potential limitations. Firstly, the multi-iteration self-optimization process at the heart of EFFI-LEARNER can be time-consuming, particularly when applied to intricate programming tasks. Yet, it's important to note that this investment of time can yield significant long-term benefits, as the optimized codes, once deployed, can considerably improve efficiency. Moreover, the process of feeding overhead profiles to LLMs to prompt code optimization may consume more tokens. This is due to the fact that the length of the overhead profiles necessitates additional tokens. Lastly, the effectiveness of EffiLearner has been primarily evaluated on Python. Therefore, its performance in different programming languages or environments may vary, underscoring the need for further testing and validation of this approach in a diverse range of contexts.Positive Societal Impacts The proposed method EFFI-LEARNER has the potential to bring about several positive societal impacts. Primarily, it can significantly enhance productivity by enabling developers to complete tasks more quickly and efficiently, due to the faster execution times and lower memory and processing power consumption of the optimized code. This is particularly beneficial in resource-constrained environments, such as mobile devices or embedded systems, where conserving resources is crucial. Moreover, EFFI-LEARNER's focus on improving code efficiency can lead to more cost-effective solutions, as lower resource consumption translates to reduced operational costs.Negative Societal Impacts The increasing effectiveness of LLM-based tools like EFFI-LEARNER could also have negative societal impacts. For instance, it could lead to an over-reliance on these systems, potentially resulting in a decline in human coding skills and a lack of understanding of the underlying code. This could be problematic in situations where human intervention is necessary. Additionally, there is a risk of job displacement in the coding and programming industry, as the demand for human coders may decrease, particularly for routine coding tasks. Lastly, while EFFI-LEARNER can improve code efficiency, it does not necessarily address potential security vulnerabilities or privacy issues in the code, which could have significant societal impacts if not properly managed.A. To demonstrate EFFI-LEARNER's effectiveness with existing baselines and some prompt engineering methods that can refine the prompt from the correctness to the efficiency of LLM-generated code. In our work, we adopt the efficiency metrics proposed by EffiBench [27] to evaluate the effectiveness of EFFI-LEARNER in improving the efficiency of LLM-generated code.Execution Time (ET) Execution time (ET) measures the average time taken for code execution. Mathematically, ET is defined as:ET = 1 N N T codewhere ET is the execution time metric, T code is the execution time of the code (with all the test cases), and N is the number of codes generated by code generation models used for evaluation.Normalized Execution Time (NET) Normalized Execution Time (NET) measures the execution time required by generated code relative to that of a canonical solution. We define NET as:N ET = 1 N N T code T canonicalwhere T code is the execution time of the generated code, and T canonical is the execution time of the canonical solution. A NET value greater than 1 indicates that the generated code is slower than the canonical solution, while a value less than 1 suggests the generated code is faster.Max Memory Usage (MU) Max Memory Usage (MU) measures the average max memory consumption during code execution. Mathematically, MU is defined as:M U = 1 N N M codewhere M U is the memory usage metric, M code is the max memory consumption of the generated code among all the test cases, and N is the number of code instances generated by code generation models used for evaluation. This metric is critical for assessing the resource efficiency of generated code, particularly in environments with limited maximum memory capacity.Normalized Max Memory Usage (NMU) Normalized Max Memory Usage (NMU) quantifies how the max memory efficiency of the generated code compares to the canonical solution. We define NMU as:N M U = 1 N N M code M canonicalwhere N M U is the normalized max memory usage metric, M code is the max memory usage of the generated code, and M canonical is the max memory usage of the canonical solution. An NMU value less than 1 indicates that the generated code is more memory-efficient than the canonical solution, whereas a value greater than 1 suggests it is less efficient in terms of memory usage. This metric provides a relative measure of the memory optimization in the generated code in comparison to a standard baseline.Total Memory Usage (TMU) Total Memory Usage (TMU) assesses the efficiency of memory usage throughout the execution of code, taking into account both the magnitude and duration of memory utilization. To calculate TMU, first, monitor and record the memory usage at discrete time intervals during the execution, resulting in a memory usage profile M (t), where t represents time. Then, compute the area under the curve of M (t) over the total execution time, T total , using numerical integration methods such as the trapezoidal rule:T M U = 1 N N Ttotal 0 M (t) dtA lower TMU value indicates higher memory efficiency, reflecting an optimized balance between the amount of memory used and the duration of its usage.The Normalized Total Memory Usage (NTMU) offers a comparison of the dynamic memory efficiency between the generated code and the canonical solution. To determine NTMU, calculate the TMU for both the generated code and the canonical solution. Normalize the TMU of the generated code by dividing it by the TMU of the canonical solution:N T M U = 1 N N T M U code T M U canonicalwhere T M U code is the TMU of the generated code and T M U canonical is the TMU of the canonical solution. An NTMU value less than 1 signifies that the generated code manages dynamic memory more efficiently compared to the canonical solution, while a value greater than 1 indicates a less efficient management of dynamic memory. This metric provides insight into the relative use of dynamic memory of generated code compared to an established benchmark.Instruction Tuning for Code Instruction tuning has proven effective in enhancing the usability and overall performance of LLMs across various language tasks [51, 63, 60, 71] . This approach has been extended to the domain of code generation. The core challenge is the acquisition of high-quality instructional data, which is often labor-intensive. To address this, recent research has focused on developing methods to generate synthetic instruction data. Studies have shown that textbook-quality synthetic data alone can improve a model's coding and reasoning capabilities [23, 34] . One early effort was Self-Instruct [62] , which utilized LLMs to generate synthetic instruction-response pairs using carefully crafted prompts. The same LLM was then instruction-tuned on this synthetic data.Problem: You have a lock in front of you with 4 circular wheels. Each wheel has 10 slots: '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'. The wheels can rotate freely and wrap around: for example we can turn '9' to be '0', or '0' to be '9'. Each move consists of turning one wheel one slot. The lock initially starts at '0000', a string representing the state of the 4 wheels. You are given a list of deadends dead ends, meaning if the lock displays any of these codes, the wheels of the lock will stop turning and you will be unable to open it. Given a target representing the value of the wheels that will unlock the lock, return the minimum total number of turns required to open the lock, or -1 if it is impossible.Example 1: Input: deadends = ["0201","0101","0102","1212","2002"], target = "0202" Output: 6 Explanation: A sequence of valid moves would be "0000" -> "1000" -> "1100" -> "1200" -> "1201" -> "1202" -> "0202". Note that a sequence like "0000" -> "0001" -> "0002" -> "0102" -> "0202" would be invalid, because the wheels of the lock become stuck after the display becomes the dead end "0102". Example 2: Input: deadends = ["8888"], target = "0009" Output: 1 Explanation: We can turn the last wheel in reverse to move from "0000" -> "0009". Example 3: Input: deadends = ["8887","8889","8878","8898","8788","8988","7888","9888"], target = "8888" Output: -1 Explanation: We cannot reach the target without getting stuck.• 1 ≤ deadends.length ≤ 500 • deadends [i] .length == 4 • target.length == 4 • target will not be in the list deadends. • target and deadends[i] consist of digits only. solution = Solution() assert solution.openLock(["0201","0101","0102","1212","2002"], "0202") == 6 assert solution.openLock(["8888"], "0009") == 1 assert solution.openLock(["8887","8889","8878","8898","8788","8988","7888","9888"], "8888") == -1 • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Some LLMs may not generate correct initial code, we do not report the efficiency results for it.We provide results for more LLMs in Tablein Appendix.We do not include Claude-3-opus in our experiments due to limited resources. 


Decision: This paper has a very wide spread of scores from 3 (reject) to 8 (strong accept).
After reading the paper, reviews, rebuttal, and substantial discussions between authors and reviewers, I believe that the paper should be accepted.

There has been extended rebuttals by authors with numerous additional results as well as additional comments and questions by reviewers. I believe the key issues are:

1. Is the contribution novel enough? Adding profiling results to LLM prompt to improve the code efficiency is novel. Reviewers are split whether the novelty is significant. I would rate novelty by itself in the weak accept area.
2. Is the evaluation done well? I believe that authors have improved the evaluation significantly during the rebuttal process. I agree with reviewer VC11 that "the current evaluation of SOAP adequately demonstrates the effectiveness of the work". However, I acknowledge the reviewers who pointed out somewhat questionable results in rebuttal. I am also somewhat concerned on how the authors are going to integrate all the additional results from rebuttal into the paper and whether the evaluation will be presented well. I would rate the evaluation also at the level of weak accept (I might change it to higher if the evaluation was well integrated into the paper.)
3. Functional Correctness: Some reviewers expressed concerns about the impact on functional correctness arguing that the SOAP method might introduce semantic degradation in code correctness after refinement. Authors have shown minimal correctness degradation in the benchmarks suggested by the concerned reviewers. In my opinion the correctness degradation may occur, but it is not a blocker for adopting SOAP like techniques.

Based on my evaluation of these issues - taking into account all the reviews, rebuttal, and discussions - I propose we accept the paper.

Reviews: review: paper_summary: paper_summary: This paper presents SOAP, a new code generation approach that improves the efficiency of code generated by a LLM. SOAP adopts a self-refinement method that iteratively prompts the LLM to re-generate the code based on the profiling results. Specifically, it uses the line-profiler package in Pyhton to get the execution time profiling results and the memory_profiler package to get the memory usage profiling result. SOAP is evaluated on three benchmarks and 22 LLMs, including both 16 open-sourced and 6 closed-sourced models. The results show that SOAP can effectively improve the execution time and memory usage of code generated by most LLMs. While it slightly decreases the LLM's performance on functional correctness, the impact is small.
strength_weakness: strength_weakness: 1. The technical novelty of this work is limited. The self-refinement method proposed in this work is very similar to existing self-refinement methods such as Self-Edit and Critic. The only difference is that SOAP uses profiling results while existing methods use testing results, etc.  
2. The authors ignored existing work on using transformers or LLMs for code optimization. They should discuss these papers in the related work and also use them as comparison baselines. 
- Liu, Shengcai, et al. "Large language models as evolutionary optimizers." arXiv preprint arXiv:2310.19046 (2023).
- Chen, Zimin, Sen Fang, and Martin Monperrus. "Supersonic: Learning to generate source code optimisations in c/c++." arXiv preprint arXiv:2309.14846 (2023).
- Shypula, Alexander, et al. "Learning performance-improving code edits." arXiv preprint arXiv:2302.07867 (2023).
3. Section 4.3 compares SOAP with Self-Refine and Reflexion. It is unclear how Self-Refine and Reflexion were configured in this experiment. Did the authors use the original prompts from Self-Refine and Reflexion? Or did the authors modify their prompts to include instructions on code efficiency? If the former, there is a comparison fairness issue since the original prompts in Self-Refine and Reflexion are not designed for code optimization. If the latter, the authors should provide the prompts used in Self-Refine and Reflexion and justify the prompt design. 
4. The experiments on the impact of step-optimization steps (Section 4.2) only include two big models. The findings may not be generalizable to smaller models. Why not also include some smaller models?
5. The experiments on the impact of functional correctness were only done on EffiBench. Since EffiBench is not specifically designed for 
 functional code generation, the results may not be representative. The authors should also experiment with HumanEval and MBPP and report the pass@1 scores on these two benchmarks.
questions: questions: 1. What is the key technical novelty of this work?
2. How would you compare SOAP with existing transformer or LLM-based code optimization work?
3. What are the prompts used in the experiments with Self-Refine and Reflexion?
limitations: limitations: The discussion on limitations looks reasonable.
paper_summary: paper_summary: The paper introduces a novel method for code super-optimization by iteratively refining code using LLMs with profiling information. The focus is on enhancing the efficiency of LLM-generated code, addressing execution time and memory requirements. The proposed methodology involves generating code with an LLM and then iteratively feeding profiling results back to the LLM to optimize the code's efficiency. The profiling information includes per-line execution time and memory usage data, helping to identify specific lines that require significant overhead. Experimental results show that this approach outperforms existing unsupervised self-refine and result-aware self-refine methods on efficiency benchmarks like EffiBench, with supplementary experiments on HumanEval and MBPP.
strength_weakness: strength_weakness: - LLM self-refinement or self-repair techniques have been extensively employed in generating responses/codes [1] and debugging tasks [2], whereas utilizing code profiling information for optimization is also a long-established concept [3, 4], which raises concerns in the proposed method's technical novelty. 
- The paper seems to treat the number of optimization steps as a hyperparameter. While Section 4.2 covers the impact of self-optimization steps, the average number of iterations required to achieve the optimization results in Table 1 is not clearly disclosed. This information is critical for comprehending the efficiency and practicality of the proposed method. Automating the stopping criteria for iterative optimization is also important in practice, but it is not supported by the proposed method, thus limiting its applicability.
- The potential semantic degradation in code correctness after refinement, particularly in scenarios where test cases do not cover all edge cases (which are common in real-world applications), raises concerns about the reliability of the optimized code, particularly in safety-critical scenarios. 
- The methodology section, particularly section 3.4, is overly verbose. The explanation of each part of the prompt could be condensed to improve readability.
questions: questions: ## Questions & Suggestions:

1. What is the average number of iterations required for optimization? What are the stopping criteria for iterative optimization? Did you set it to 5 for all the experiments? How is it determined when to stop the refinement process?
2. How significant is the profiling overhead? Is there any analysis on the average number of extra tokens required for including the profiling information as prompts? How does this affect the overall efficiency and effectiveness of the optimization process?
3. The downgraded pass@1 results in Section 4.4 suggest that the proposed method cannot guarantee semantic equivalence after code refinement. This could impact the practicality and reliability of the code superoptimizer, especially in safety-critical applications. How can this issue be mitigated?
4. Analysis of the optimization techniques applied by the LLM is necessary. Are there certain types of optimization where the method may fall short?
5. The current approach relies on the LLM to analyze and identify hot spots from the profiling results. Given the standardized format of profiling results, could rule-based post-processing techniques that filter out hot spot information improve or accelerate the self-refinement process?
6. In line 189, only code that passed all test cases is considered in the evaluation. What if the refined code fails the test cases during optimization? How is this situation handled?
7. In Table 2, the optimization effect on memory usage appears limited compared to execution time. Also, in the ablation studies (Table 3), self-optimization with only execution time profiling feedback achieves significant memory usage reduction. Does this mean that memory optimization is relatively easier in the evaluated benchmark, or does execution time information play a more critical role in optimization?

---
[1] Madaan, Aman, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon et al. "Self-refine: Iterative refinement with self-feedback." Advances in Neural Information Processing Systems 36 (2024).

[2] Olausson, Theo X., Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. "Is Self-Repair a Silver Bullet for Code Generation?." In The Twelfth International Conference on Learning Representations. 2023.

[3] Chang, Pohua P., Scott A. Mahlke, and Wen‐Mei W. Hwu. "Using profile information to assist classic code optimizations." Software: Practice and Experience 21, no. 12 (1991): 1301-1321.

[4] Agakov, Felix, Edwin Bonilla, John Cavazos, Björn Franke, Grigori Fursin, Michael FP O'Boyle, John Thomson, Marc Toussaint, and Christopher KI Williams. "Using machine learning to focus iterative optimization." In International Symposium on Code Generation and Optimization (CGO'06), pp. 11-pp. IEEE, 2006.
limitations: limitations: The limitations related to the overhead of profiling and the semantic correctness of the optimized code are mentioned in the paper but not analyzed in detail. A more comprehensive cost-benefit analysis, including the extra tokens required for profiling information in prompts, would provide a clearer picture of the method's overall efficiency. The authors should also consider the broader implications of this optimization approach, including its potential impact on code reliability and safety in practical applications.
paper_summary: paper_summary: This paper studies an important and timely issue: the inefficiency often found in code generated by current Large Language Models (LLMs), which can result in longer execution times and higher memory consumption. To mitigate this issue, the paper proposes a self-optimization framework SOAP, designed to improve the efficiency of LLM-generated code. SOAP starts from the code generated by an LLM and then executes it locally to profile its execution time and memory usage. These profiles are fed back to the LLM, allowing it to revise the code and reduce overhead iteratively. The authors verify the effectiveness of SOAP through empirical experiments on the leading open-source and closed-source models, showing SOAP can substantially optimize both execution time and memory consumption.
strength_weakness: strength_weakness: 1) The multi-iteration optimization method requires large token overhead, which can be both resource-intensive and time-consuming.

2) Overhead profiles serve as a good indicator for local code optimization. Nevertheless, the framework may struggle with the scalability of extended code global optimization due to the limited context window of LLMs.
questions: questions: 1) In this paper, SOAP optimizes both execution time and memory usage simultaneously. However, in real-world scenarios, people might prioritize a single objective, either time or space. Do you think it is possible for SOAP to include optimization options (similar to -O3 or -Oz in GCC) to cater to different preferences?

2) The iterative self-optimization process might increase the risk that the final generated code is either non-executable or deviates from the original problem specifications. Do you have any mechanisms in place to prevent such deterioration?

3) How does the token usage in the SOAP profile optimization process compare to that in direct code generation (one-shot)?
limitations: limitations: The authors addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: The paper aims to address the inefficiency issue of code generated by LLMs in terms of execution time and memory consumption. The authors propose a framework called Self-Optimization based on OverheAd Profile (SOAP), which improves the efficiency of LLM-generated code by iteratively refining it based on execution overhead profiles. These profiles capture the code's execution time and memory usage, which are then fed back into the LLM to optimize the code. The paper demonstrates the effectiveness of SOAP through extensive experiments on various benchmarks and selected LLMs. Results show reductions in execution time and memory usage, highlighting SOAP's ability to enhance code efficiency without compromising correctness.
strength_weakness: strength_weakness: * The baseline selection can be largely improved. The current baseline in the paper is to ask LLMs to directly generate code -- it is not the models' fault to generate non-optimal code when they are not asked to. At least two reasonable and simple baselines should have been included: (1) ask the model to generate an efficient version of code, and (2) ask the model to generate code and then directly optimize it (similar to the ALGO approach, despite not using test generation and execution to soften the environmental assumptions). Furthermore, the PIE [1] paper mentioned various few-shot prompting methods to perform code optimization, which should also be used as baselines.
* To perform an "apple-to-apple" comparison over the models, Tables 1-4 should be better aligned. To understand which model is better at generating efficient code, they must be compared under the same set of tasks. While due to correctness inconsistency, the attributed coding tasks are not aligned, authors may compare the models under a set of mutually passing coding tasks.
* Efficiency evaluation can be noisy without carefully configuring the metrics and testbeds. The considered metrics such as execution time and memory usage are flaky and non-reproducible as they are measured in physical platforms. To mitigate these limitations, authors may follow PIE [1] to use architecture simulators or other better-defined metrics.
* While the author highlighted efficiency is critical in resource-constrained scenarios, the proposed technique is specialized for Python. It is unclear if Self-Optimization can work for performance-demanding languages such as C/C++ and related experiments are ignored.
* [1] is a closely relevant and published work that should be well discussed and compared.

[1] Shypula A, Madaan A, Zeng Y, Alon U, Gardner J, Hashemi M, Neubig G, Ranganathan P, Bastani O, Yazdanbakhsh A. “Learning Performance-Improving Code Edits.” ICLR 2024.
questions: questions: * How is SOAP compared to more reasonable baselines below:
  1. Prompting an LLM to generate efficient code in zero-shot
  2. Ask the LLM to refine its first generation for code performance
  3. Fewshot prompting mentioned in PIE
* Are profilers necessary? What if we just replace profilers with self-reasoning or self-reflection?
* How's the cost (in the form of token consumption and pipeline running time) of SOAP and baselines?
* How is the overall variation of mentioned metrics including execution time? For example, when running and measuring the code execution for N times, what is the overall coefficient of variation?
limitations: limitations: * Using a profiler and code executor in the pipeline is expensive with respect to the degree of improvement. In production, integrating code execution in a sandbox is already challenging, and integrating a profiler (for multiple languages) would take more engineering cost. Meanwhile, the performance gain (e.g., execution time) overall looks minimal compared to the baseline whose prompt does not even encourage code optimization. 
* Overall the technical contribution of the paper is incremental: compared to prior techniques such as ALGO and PIE, the major difference is the use of code profiler.

Comments: title: Response to Reviewer H3VZ
Response to the rebuttal
Friendly Reminder: Pending Reviewer Responses and Assessment Consideration
Friendly Reminder: Pending Reviewer Responses and Assessment Consideration
Friendly Reminder: Pending Reviewer Responses and Assessment Consideration
Response Reviewer 5c1j follow up questions
Results
Apple-to-Apple results
Clarification on rejection
Thank you for your response
Rebuttal for Reviewer 4 Follow Up
Thanks for the detailed reply!
Thank you for clarification and additional results
Response for AC question
Thank you for your response.
Response about global optimal
Question for authors
Author-Reviewer discussions (Aug 7 - Aug 13)
comment: Dear Reviewer H3VZ,

Thanks for your response before the discussion period deadline, which allows us to further clarify the potential misunderstanding and concerns that would affect your overall assessment. 

First, compared with self-refine, SOAP utilizes line_profiler and memory profiler to catch the execution time and memory usage of each line in the LLM-generated code.

Second, for the pass@1 of LLM-generated code of pre-SOAP and post-SOAP, we want to mention that in https://anonymous.4open.science/r/SOAP-FF6C/src/SOAP.py (Lines 177-Lines 185), we provide tolerance for SOAP that requires the SOAP optimized code can pass public test cases and then replace original code. If the optimized code can not pass the public test cases, SOAP optimized code will not replace the original code, and we will use the original code for the next step of optimization (Lines 184-185). In this step, the coverage of the public test cases would largely affect the correctness of the final SOAP-optimized code. After the empirical study, we observe that the code line coverage of EffiBench for public test cases achieves **99.24%**, which makes sure that the pass@1 after the self-optimization process (SOAP) only has a few decreases, and even in some models, the pass@1 would not decrease. 

Third, we need to mention that PIE already trained CodeLlama with its provided code, which means that the pass@1 of PIE+CodeLlama-7B may be different, and the tasks addressed by PIE and the original CodeLlama are different. Then, some tasks with higher execution time (e.g., 4.7s) may not addressed by PIE-CodeLlama, and the default (initial) ET and other metrics are different. We hope that Reviewer H3VZ can understand this difference between the original CodeLlama and PIE-CodeLlama. 

We hope that our clarification can address Reviewer H3VZ's concern about **I feel these results require another round of careful reviews, especially given that some results look quite surprising and counterintuitive.**
Thank you for the responses, and sorry for the late response due to a flurry of proposal and review dues. Regarding the novelty, would it be fair to say the essential difference to self-refine is the inclusion of the profiling results from the `line_profiler` library in the prompt?

While I appreciate the authors' effort in conducting new experiments quickly in the short rebuttal period, I feel these results require another round of careful reviews, especially given that some results look quite surprising and counterintuitive. Without knowing more details and experiment settings, it is hard to make a full assessment of the correctness of the results. 

In particular, it's surprising to see that the pass@1 rates of all these models barely change after applying SOAP. Note that compared with the original code generation prompt, SOAP adds a lot of line-level profiling results, which significantly deviates from the original prompt. Presumably, we should see some deviations in pass@1, as LLMs are pretty sensitive to prompt design and including many profiling results may distract the LLM from reading the original task description. How did the authors keep the pass@1 rates almost unchanged?

Furthermore, the evaluation results of PIE with CodeLlama7B look quite strange. The average execution time of code optimized by PIE is in the range of 0.4s to 0.9s. However, the execution time of CodeLlama7B in Table 1 is 4.7s before SOAP optimization and 4.52s after SOAP optimization. If the execution time after PIE optimization is correct, **does this mean PIE can achieve 10x speed up compared with the initial code and the code optimized by SOAP alone?** If that's the case, what's the point of applying SOAP on top of PIE given that PIE has already achieved such a big improvement?

Overall, I feel this paper requires another round of careful reviews before it can be accepted.
Dear Reviewer 5c1j,

Thank you for your review and comments on our paper on 13 Aug, we have provided the clarification and other experiment results to address your concern.

Since the discussion period will be closed in the next six hours, while we do not receive the message about whether we have addressed all of your concerns, we would greatly appreciate it if you could take the time to review these results and consider improving your overall assessment of our paper based on these revisions.

Thank you once again for your valuable feedback and for considering our request.
Dear Reviewer H3VZ,

Thank you for your review and comments on our paper.

Since the discussion period will be closed in the next six hours, while we do not receive the message about whether we have addressed all of your concerns, we would greatly appreciate it if you could take the time to review these results and consider improving your overall assessment of our paper based on these revisions.

Thank you once again for your valuable feedback and for considering our request.
Dear Reviewer pc3i,

Thank you for your review and comments on our paper, which you provided on August 12th.

To address your concerns regarding the "apples to apples" comparison, we have added all experiments in the last response thread with the same format for our paper. We believe this will facilitate a more accurate and fair evaluation of our work.

We would greatly appreciate it if you could take the time to review these results and consider improving your overall assessment of our paper based on these revisions.

Thank you once again for your valuable feedback and for considering our request.
Thank you for your response to our rebuttal.

We have conducted additional experiments and provided further clarification to address the concerns raised in the follow-up questions. We hope that these efforts have adequately addressed Reviewer 5c1j's concerns. If Reviewer 5c1j thinks that the issues have been satisfactorily resolved, we would greatly appreciate it if they could consider improving the overall assessment for SOAP, as the current average overall assessment is in a borderline scenario.

**Q1**


To address your concern, we first optimized the code based on the steps outlined in Paper Table 2 and then continued to optimize for an additional five steps to analyze the efficiency of LLM-generated code after a total of 10 optimization steps. Our evaluation results are shown below. We can observe that increasing the number of optimization steps decreases the overhead, but after five steps, the efficiency metric improvements are minimal. Therefore, in our paper, we mainly set the number of steps to 5. However, if developers want to optimize their code further, they can increase the optimization steps at the cost of some additional token usage. We believe this token usage is worthwhile, as the optimized code would be deployed in the software and run thousands to millions of times.

| Steps | ET (s) | NET | MU (Mb) | NMU | TMU (Mb*s) | NTMU |
|-------|--------|-----|---------|-----|------------|------|
| CodeLlama-70B |  |  |  |  |  |  |
| 0     | 0.36   | 2.50 | 91.25   | 2.45 | 157.50     | 19.75 |
| 1     | 0.33   | 2.35 | 36.09   | 0.99 | 13.70      | 1.81 |
| 2     | 0.31   | 2.18 | 36.09   | 0.99 | 13.04      | 1.72 |
| 3     | 0.29   | 2.06 | 36.08   | 0.99 | 12.57      | 1.66 |
| 4     | 0.29   | 2.03 | 36.08   | 0.99 | 12.50      | 1.65 |
| 5     | 0.28   | 2.01 | 36.08   | 0.99 | 12.43      | 1.64 |
| 6     | 0.27   | 2.00 | 36.07   | 0.99 | 12.40      | 1.63 |
| 7     | 0.27   | 1.98 | 36.06   | 0.99 | 12.37      | 1.62 |
| 8     | 0.27   | 1.98 | 36.05   | 0.99 | 12.35      | 1.61 |
| 9     | 0.27   | 1.97 | 36.04   | 0.99 | 12.35      | 1.60 |
| 10    | 0.26   | 1.96 | 36.03   | 0.99 | 12.35      | 1.59 |
| GPT-3.5-Turbo-0301 |  |  |  |  |  |  |
| 0     | 0.36   | 2.50 | 91.25   | 2.45 | 157.50     | 19.75 |
| 1     | 0.33   | 2.35 | 36.09   | 0.99 | 13.70      | 1.81 |
| 2     | 0.31   | 2.18 | 36.09   | 0.99 | 13.04      | 1.72 |
| 3     | 0.29   | 2.06 | 36.08   | 0.99 | 12.57      | 1.66 |
| 4     | 0.29   | 2.03 | 36.08   | 0.99 | 12.50      | 1.65 |
| 5     | 0.28   | 2.01 | 36.08   | 0.99 | 12.43      | 1.64 |
| 6     | 0.28   | 2.00 | 36.07   | 0.99 | 12.40      | 1.63 |
| 7     | 0.27   | 1.99 | 36.06   | 0.99 | 12.38      | 1.62 |
| 8     | 0.27   | 1.98 | 36.05   | 0.99 | 12.36      | 1.61 |
| 9     | 0.27   | 1.97 | 36.04   | 0.99 | 12.35      | 1.60 |
| 10    | 0.27   | 1.97 | 36.04   | 0.99 | 12.33      | 1.59 |

**Q2**

Response:

**The difference between Initial pass@1 v.s. SOAP pass@1 is only the SOAP-generated incorrect code.** While the address method discussed in Q6 does not reflect in W3 & Q3, which means that with our solution in Q6, the incorrect code generated by SOAP in SOAP pass@1 will be replaced by the Initial generated code.


**Q3.**

We understand your concern about filtering out failed code in our evaluation. However, our approach is consistent with existing works in this field [1-6]. The primary reason for focusing only on correct code is that incorrect code may cause test cases to fail and terminate the execution process prematurely. Including such cases would lead to inaccurate results when measuring the efficiency of LLM-generated code. 

[1] Huang, Dong, Jie M. Zhang, Yuhao Qing, and Heming Cui. "EffiBench: Benchmarking the Efficiency of Automatically Generated Code." arXiv preprint arXiv:2402.02037 (2024).

[2] Qiu, Ruizhong, Weiliang Will Zeng, Hanghang Tong, James Ezick, and Christopher Lott. "How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark." arXiv preprint arXiv:2406.06647 (2024).

[3] Shi, Jieke, Zhou Yang, and David Lo. "Efficient and Green Large Language Models for Software Engineering: Vision and the Road Ahead." arXiv preprint arXiv:2404.04566 (2024).

[4] Zheng, Jiasheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. "Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models." arXiv preprint arXiv:2407.11470 (2024).

[5] Waghjale, Siddhant, Vishruth Veerendranath, Zora Zhiruo Wang, and Daniel Fried. "ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?." arXiv preprint arXiv:2407.14044 (2024).

[6] Du, Mingzhe, Anh Tuan Luu, Bin Ji, and See-Kiong Ng. "Mercury: An efficiency benchmark for llm code synthesis." arXiv preprint arXiv:2402.07844 (2024).
Thank you for the detailed response and the supplementary experiment results. I have a few follow-up questions:

1. Could the optimization performance be further improved with steps greater than 5? Is there a way to determine convergence in self-refinement, especially considering that the number of iterations affects token usage and LLM inference costs (as in W2)?
2. In your response to Q6, you mentioned that the entire task is filtered out if the refined code fails any private test. Can I interpret the difference in your answer to W3 & Q3 (Initial pass@1 v.s. SOAP pass@1) as the proportion of filtered tasks?
3. I do not think it is reasonable to filter out failed code in your evaluation. In real-world scenarios, private cases are supposed to be invisible.
Thanks for your reply. We provide the distribution below. We can observe that GPT-4-turbo-preview has the highest distribution in Pre-SOAP efficiency results. GPT-3.5-turbo-0301 achieves the highest efficiency ratio in Post-SOAP efficiency results.


*Distribution of the most efficient code for each task of EffiBench. We provide each LLM ratio for the most efficient code and then report the ratio for the overall results.*


| Model                             | Pre-SOAP (%)| Post-SOAP (%)|
|-----------------------------------|----------|-----------|
| OpenCodeInterpreter-DS-1.3B       | 2.33     | 1.78      |
| OpenCodeInterpreter-DS-6.7B       | 4.66     | 2.88      |
| OpenCodeInterpreter-DS-33B        | 6.30     | 3.70      |
| deepseek-coder-1.3b-instruct      | 3.84     | 1.10      |
| deepseek-coder-6.7b-instruct      | 5.07     | 7.81      |
| deepseek-coder-33b-instruct       | 0.00     | 0.00      |
| CodeLlama-7b-Instruct-hf          | 1.23     | 0.55      |
| CodeLlama-13b-Instruct-hf         | 3.29     | 0.82      |
| CodeLlama-34b-Instruct-hf         | 4.52     | 1.51      |
| CodeLlama-70b-Instruct-hf         | 5.34     | 0.82      |
| XwinCoder-13B                     | 4.11     | 0.55      |
| XwinCoder-34B                     | 5.75     | 2.74      |
| WizardCoder-Python-13B-V1.0-GPTQ  | 1.10     | 0.41      |
| starcoder2-3b                     | 0.68     | 1.23      |
| starcoder2-7b                     | 1.37     | 0.41      |
| starcoder2-15b                    | 0.55     | 0.14      |
| gpt-3.5-turbo-0301                | 8.77     | 27.67     |
| gpt-3.5-turbo-0613                | 7.53     | 5.07      |
| gpt-3.5-turbo-1106                | 7.12     | 4.52      |
| gpt-4                             | 7.53     | 7.53      |
| gpt-4-turbo-preview               | 16.16    | 14.25     |
| claude-3-haiku                    | 1.37     | 5.89      |
| claude-3-sonnet                   | 1.37     | 8.63      |
***Reviewer pc3i Table 2 (common tasks)** Effect of the number of self-optimization steps in SOAP (46 tasks). The evaluation results are used to replace the results of Paper Table 2 with the same correct tasks addressed by two LLMs for all steps.*


| Steps | **ET (s)** | **NET** | **MU (Mb)** | **NMU** | **TMU (Mb*s)** | **NTMU** |
|-------|------------|---------|-------------|---------|-------|----------|
|||  | |  |  ||
| **CodeLlama-70b-Instruct-hf** ||  | |  |  ||
| 0| 0.40| 3.00 | 54.79| 1.76 | 20.05| 4.86|
| 1| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53|
| 2| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53|
| 3| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53|
| 4| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53|
| 5| 0.29| 2.20 | 54.90| 1.77 | 14.58| 3.53|
|||  | |  |  ||
| **gpt-3.5-turbo-0301** ||  | |  |  ||
| 0| 0.24| 1.78 | 31.12| 1.00 | 10.74| 2.60|
| 1| 0.22| 1.66 | 31.28| 1.01 | 7.36 | 1.78|
| 2| 0.22| 1.64 | 31.27| 1.01 | 7.02 | 1.70|
| 3| 0.22| 1.64 | 31.27| 1.01 | 6.98 | 1.69|
| 4| 0.22| 1.64 | 31.27| 1.01 | 6.98 | 1.69|
| 5| 0.22| 1.64 | 31.27| 1.01 | 6.98 | 1.69|






***Reviewer pc3i Table 3 (common tasks)** Effect of the number of self-optimization steps in SOAP (40 tasks). The evaluation results are used to replace the results of Paper Table 3 with the same correct tasks addressed by two LLMs for all steps.*


| Steps| ET (s) | NET| MU (Mb) | NMU | TMU (Mb*s) | NTMU |
| ------------------------- | ------ | ----- | ------- | --- | ---------- | ---- |
| **CodeLlama-70b-Instruct-hf**  | ||  |  |  ||
| Initial  | 1.82| 14.78 | 25.85| 1.00| 36.32| 16.89|
| Unsupervised Self-Refine  | 0.34| 2.80  | 25.82| 1.00| 6.99| 3.25 |
| Result-Aware Self-Refine  | 0.33| 2.67  | 25.96| 1.00| 6.85| 3.19 |
| memory_profiler | 0.26| 2.08  | 25.88| 1.00| 4.38| 1.97 |
| time_overhead_profiler | 0.19| 1.57  | 25.88| 1.00| 4.03| 1.89 |
| SOAP | 0.19| 1.55  | 25.87| 1.00| 3.98| 1.85 |
| **gpt-3.5-turbo-0301** | ||  |  |  ||
| Initial  | 0.24| 1.95  | 25.91| 1.00| 5.01| 2.33 |
| Unsupervised Self-Refine  | 0.26| 2.14  | 29.99| 1.16| 7.65| 3.56 |
| Result-Aware Self-Refine  | 0.23| 1.87  | 25.95| 1.00| 5.12| 2.38 |
| memory_profiler | 0.19| 1.53  | 25.88| 1.00| 3.84| 1.79 |
| time_overhead_profiler | 0.18| 1.47  | 31.26| 1.21| 4.98| 2.32 |
| SOAP | 0.18| 1.48  | 25.88| 1.00| 3.72| 1.73 |


***Reviewer pc3i Table 4 (common tasks)** Effect of the number of self-optimization steps in SOAP (3 tasks) for CodeLlama family.*

| Steps | **ET (s)** | **NET** | **MU (Mb)** | **NMU** | **TMU (Mb*s)** | **NTMU** |
|-------|------------|---------|-------------|---------|-------|-----|
||   ||||| |
| **CodeLlama-7b-Instruct-hf** |   ||||| |
| Initial | 0.78 | 5.61 | 30.82 | 1.00 | 13.89 | 5.50 |
| SOAP (step 5) | 0.23 | 1.65 | 54.69 | 1.77 | 7.80 | 3.09 |
||   ||||| |
| **CodeLlama-13b-Instruct-hf** |   ||||| |
| Initial | 0.33 | 2.38 | 54.66 | 1.76 | 11.44 | 4.53 |
| SOAP (step 5) | 0.25 | 1.76 | 54.69 | 1.77 | 8.46 | 3.35 |
||   ||||| |
| **CodeLlama-34b-Instruct-hf** |   ||||| |
| Initial | 0.31 | 2.25 | 54.59 | 1.76 | 10.72 | 4.25 |
| SOAP (step 5) | 0.25 | 1.76 | 54.64 | 1.76 | 8.42 | 3.33 |
||   ||||| |
| **CodeLlama-70b-Instruct-hf** |   ||||| |
| Initial | 0.32 | 2.31 | 54.62 | 1.76 | 10.97 | 4.34 |
| SOAP (step 5) | 0.23 | 1.61 | 54.71 | 1.77 | 7.65 | 3.03 |
Thank you for your reply. SOAP is effective for both scenarios (i.e., improving LLMs' own tasks and improving common correct tasks).

To address the concern of the "apple-to-apple" results of our experiments in the Paper's main flow, in this thread, we provide results based on common tasks for Table 2 - Table 4's results, which provide consistent results for the two configurations.
(*Table 1's results are already demonstrated in a previous thread.*)

For Paper Table 2, where we discuss how SOAP's effectiveness would be affected by the optimization results, we provide the evaluation results in **Reviewer pc3i Table 2 (common tasks)**. First, we find the tasks addressed by both CodeLlama-70B-Ins and GPT-3.5-turbo-0301 with the Initial version (Step 0) and SOAP optimization process (Step 1-5). Then, we collect 46 tasks and measure the efficiency of LLM-generated code in these tasks. The evaluation results in **Reviewer pc3i Table 2 (common tasks)** show that CodeLlama-70B-Ins achieves SOAP efficiency results with one-step SOAP optimization, while GPT-3.5-turbo-0301 continuously improves the efficiency for the evaluated 46 tasks from Step 0 to Step 3. Then, the efficiency of GPT-3.5-turbo-0301 maintains efficiency for the other two optimization steps.


Next, for Paper Table 3, where we discuss how different prompt optimization steps affect the efficiency of LLM-generated code, we provide the evaluation results in **Reviewer pc3i Table 3 (common tasks)**. First, we find the tasks addressed by both CodeLlama-70B-Ins and GPT-3.5-turbo-0301 for six different prompts and then get 40 tasks. Then, we provide the efficiency results in **Reviewer pc3i Table 3 (common tasks)**, where we can observe that SOAP also achieves SOTA efficiency results compared with other baselines. For example, SOAP achieves 3.72 (Mb*s) for TMU, while baselines only achieve 3.84 (Mb*s) in the TMU metric for GPT-3.5-turbo generated code.


Finally, for Paper Table 4, where we discuss the differences in the efficiency of SOAP and the initial version of the CodeLlama family, we provide the evaluation results in **Reviewer pc3i Table 4 (common tasks)**. First, we detect correct tasks addressed by both SOAP and the Initial version for four CodeLlama models and then get 3 tasks. Next, we provide the efficiency results of CodeLlama family-generated code in **Reviewer pc3i Table 4 (common tasks)**. We can observe that for four CodeLlama models, SOAP continuously improves the efficiency of LLM-generated code.


We hope that our newly provided results can address the reviewer's concern. 

*Note: We provide the Tables in the next thread.*
Thanks for the reply. I am sorry for not explaining my decision enough and here are my detailed reasons for my suggestion of decision:

1. First, the preliminary experimental results from the authors quickly cleared my concerns about approach effectiveness regarding weaknesses in #1, #4, and #5.
2. I won't reject the paper just for #1, #4, and #5 because those results are complementary to what the paper already had, meaning that they could be easily added to the camera-ready version if we end up accepting it.
3. Yet, weakness #2 still holds as existing results of the paper are mostly not represented in an "apple-to-apple" fashion. It is not even right to compare the efficiency of LLM codegen over different sets of tasks, making the efficiency score unreasonable to compare. For an extreme case, two LLMs may solve completely different sets of solutions, and computing efficiency scores over two totally different task sets cannot lead to clean and reasonable results regarding code efficiency in general.
4. Because it is problematic to compare LLM efficiency over different sets of tasks, ideally the author should globally correct the evaluation methodology, i.e., removing results made by inconsistent comparison and enforcing correctness alignments. This is crucial before accepting this paper because if not future work might inherit such a misevaluation methodology, leading to more inconsistent comparison in general.
5. Yet, doing reason 4 in rebuttal is quite hard objectively speaking. Meanwhile, given it's a major revision, it might be more rigorous to apply and reorganize the experiments and resubmit a clean version for another review, rather than accepting the paper through some rough views over the set of preliminary results that we are unsure if those can be systematically adopted in the camera-ready.

To summarize, I deeply appreciate the efforts of the authors in the rebuttal; however, for soundness, the required revision goes beyond "convincing the reviewers through proof-of-concept experiments" that it is crucial to globally correct the evaluation methodology for positively leading future research in this thread. Last but not least, I strongly feel this work can be accepted after doing a major revision.
Thank you for your thorough responses. The additional experiments and explanations have satisfactorily addressed my concerns. As a result, I am pleased to elevate my confidence from 4 to 5 and advocate for the paper's acceptance.

I also read comments from the Area Chair and other Reviewers. I like the idea of evaluating N code variants x M LLMs, as it promises to provide a comprehensive performance spectrum across various LLMs. However, it is clear that conducting such an intensive experiment within the rebuttal period is impractical. Could you share the individual model performances (or percentages) from your experiment in the AC thread? It would be intriguing to see the expertise demonstrated by each individual model.
**Specifically, in the demonstrated comparison from the preliminary results SOAP outperforms other baselines, the ratio of improvements is largely different from table to table when using different measurements and task set alignments. For example, improvements in Table 3 are way larger than that of Table 2 and this is likely due to the impact of task correctness over efficiency. That being said, while I appreciate the various new preliminary experiments from the authors, I still want to hold a rejection before an optimal experimental setting is figured out and applied globally. This would largely improve the consistency and soundness of the conclusion, which is crucial for efficiency evaluation.**


Response:

Thank you for your appreciation of the new preliminary experiments we provided during the rebuttal process. However, we are disheartened that reviewer pc3i wishes to reject our paper based on the claim, **"I still want to hold a rejection before an optimal experimental setting is figured out and applied globally"**. To address reviewer pc3i's concerns about w2 and w3, we have provided new results to reviewer pc3i's requirements. Specifically, for w2, we want to clarify that we conducted experiments on **all** closed-source models. However, **we did not conduct experiments on open-source models due to their low pass@1 and the lack of consistent correct code across all open-source models**. 


Next, for the PIE provided Gem5 default simulator, we also included seven different LLMs: smaller open-source LLMs (OpenCodeInterPreter-1.3B, DeepSeekCoder-1.3B-Ins, CodeLlama-7B-Ins-hf, StarCoder2-15B), a larger open-source model (CodeLlama-70B-Ins-hf), and two closed-source models (GPT-3.5-turbo-0301 and Claude-3-sonnet). We believe that these models are representative of our experiments.




**The variations in Table 5?**


Response:

To address the reviewer’s concern, we provide the source code in our anonymous GitHub. We first use run_source_code_five_times.py to run the LLM-generated code five times. Next, we use report_variant.py to first calculate mean and std. Then we report coef_of_var with (std_dev / mean) * 100 for all experiments. 


**Configuration of Result-Aware Self-Refine compared to SOAP?**

Response:

The configuration of Result-Aware Self-Refine is shown in our anonymous GitHub prompts.py, where the only difference between SOAP and Result-Aware Self-Refine is that the overhead analysis part of Result-Aware Self-Refine in Lines 19 only contains the ET, MU, and TMU (See Lines 54-58). The overhead analysis part of SOAP also contains the execution time profiler and memory profile (See Lines 28-35).


**Follow-up suggestions:**

**S1: Given the large token overhead of SOAP, what if we just do optimization sampling and profile to select the best one using the same token budget?**

Response:

Thank you for your suggestion. We have provided evaluation results for your suggested method and SOAP in GPT-3.5-turbo-0301 below. It's observable that this method can enhance the efficiency of code generated by the LLM as compared to the initial version. For instance, the TMU of GPT-3.5-turbo-0301 decreases from 157.50 (Mbs) to 27.99 (Mbs). This significant reduction is primarily due to the suggested method of selecting the most efficient route from multiple codes. However, we also noticed that the suggested method is still less efficient than SOAP. The primary reason for this is that the suggested method primarily aims to generate multiple efficient solutions for a given task. However, without the profile information provided by SOAP, the LLM is unable to optimize the code and generate a more efficient version than the initial code, resulting in the suggested method being less efficient than SOAP.

| Strategy | **ET (s)** | **NET** | **MU (Mb)** | **NMU** | **TMU (Mb*s)** | **NTMU** |
|----------|------------|---------|-------------|---------|----------------|----------|
|Initial|0.36|2.50|91.25|2.45|157.50|19.75|
| Reviewer pc3i suggested | 0.31 | 2.17 | 36.14 | 1.00 | 27.99 | 2.94 |
| SOAP | 0.28 | 2.01 | 36.08 | 0.99 | 12.43 | 1.64 |


*Effectiveness of SOAP with Reviewer pc3i proposed optimization sampling and profile to select the best one for GPT-3.5-turbo with similar token usages. For each method, we use **int(token usage of SOAP/token usage of baselines such as direct generation with sampling) + 1** to ensure that the token usage of SOAP should be lower than the baselines.*

**S2: In general, it is encouraged to simplify the metrics for clarity.**

Response:

Thanks for your suggestion. We will revise our manuscript in the final version.
First, I'd like to thank the authors for their reply. It is impressive and surprising to add this volume of preliminary experiments within such a short period.

While I think weaknesses #1, #4, and #5 can be mostly addressed by adding new tables in the paper, weaknesses #2 and #3 require a more thorough global revision/update. Specifically, while in the demonstrated comparison from the preliminary results SOAP outperforms other baselines, the ratio of improvements is largely different from table to table when using different measurements and task set alignments. For example, improvements in Table 3 are way larger than that of Table 2 and this is likely due to the impact of task correctness over efficiency.

That being said, while I appreciate the various new preliminary experiments from the authors, I still want to hold a rejection before an optimal experimental setting is figured out and applied globally. This would largely improve conclusion consistency and soundness of evaluation, which is very crucial for efficiency evaluation.

Regarding the responses to my questions, I have a few concerns:
1. The variations in Table 5 are surprisingly small and the variations across different metrics are surprisingly equivalent. Can the authors double-check their data?
2. How do the authors configure Result-Aware Self-Refine compared to SOAP?

Follow-up suggestions: 
1. Given the large token overhead of SOAP, what if we just do optimization sampling and profile to select the best one using the same token budget?
2. In general, it is encouraged to simplify the metrics for clarity.
Thank you for clarification and additional results.
Dear Area Chair gHRy,

Clarification for Previously Provided Results:

Thank you for your response. We would like to address your concerns and clarify our setup as follows:

In the experiment for optimal ET, MU, and TMU, we first gather solutions from all LLMs for each task in our EffiBench. Subsequently, we calculate the ET, MU, and TMU for each solution from the collected solutions and rank them based on these three metrics. For instance, considering ET, we select the most efficient solution (that is, a solution with the lowest ET among all model-generated codes) as the pre-SOAP solution. This represents the most effective solution in terms of ET in the pre-SOAP stage. Similarly, for SOAP, we follow the same strategy to obtain the most efficient ET solution.

Through this process, we do not directly select the 0.29 (s) OpenCodeInterpreter-33B's pre-SOAP as the optimal ET since some tasks might not be addressed by OpenCodeInterpreter-33B. Other LLMs that may address these tasks are also used to calculate pre-SOAP. Lastly, we compute the efficiency for both the pre-SOAP most efficient code and the post-SOAP most efficient code, but only for correct codes in the experiments.

Indeed, the method you proposed is excellent. Based on your suggestion, we have conducted the following experiments. As can be seen, in this setup, SOAP also reduces the overhead of the initial version across all metrics. For instance, the ET decreases by 58.6%, reducing the average execution time from 0.29 seconds to 0.12 seconds. In our final version, we will also provide the evaluation results of SOAP for the N code variants x M LLMs.


| Steps   | **ET (s)** | **NET** | **MU (Mb)** | **NMU** | **TMU (Mb*s)** | **NTMU** |
|---------|------------|---------|-------------|---------|----------------|----------|
| Initial | 0.29       | 1.25    | 24.31       | 0.93    | 11.84          | 1.11     |
| SOAP    | 0.12 (58.6\%)| 1.03 (17.6\%)| 24.28 (0.1\%)| 0.82 (11.8\%)| 2.03 (82.9\%)| 0.88 (20.7\%)|
I appreciate your response.
I think this is a good metric.

However, could you clarify the numbers in your table? 

I would think that "Optimal ET" would have initial 0.29 and after-SOAP 0.25 (DeepSeek-33B-Ins and OpenCodeInterpreter-33B are most efficient based on ET pre-SOAP, I picked DeepSeek-33B-Ins since SOAP improves its code more), "Optimal MU" would have initial 24.31 and after-SOAP 24.28 (StarCoder2-3B is most efficient based on MU), "Optimal TMU" would have initial 13.06 and after-SOAP 11.54 (OpenCodeInterpreter-33B is most efficient based on TMU). Am I misunderstanding your metric? (I did not look up other columns in the table just the ones corresponding to the Optimal metric in your subheading)

Thank you again.
Dear AC,

Thanks for your efforts to remind reviewers to participate in the discussion process.

To address the concern about the global efficiency results, we conduct experiments with our previously generated source code and SOAP-optimized code. We follow the below instructions to calculate the overall efficiency. First, we collect all correct LLM-generated code for each task and then select the most efficient code generated by LLMs based on ET, MU, and TMU. Then, we calculate the overall efficiency metrics for each metric-guided selected code for the initial code and SOAP-optimized code. The evaluation results for EffiBench are shown below. We can observe that in this scenario, SOAP largely improved the efficiency of LLM-generated code. For example, in the Optimizal ET results (i.e., we collect source code for both initial and SOAP optimized code for each task), SOAP decreases the average ET from 0.97 (s) to 0.23 (s). In Optimizal MU, SOAP decreases the MU from 59.16 (Mb) to 21.13 (Mb). In Optimial TMU, SOAP also decreases the TMU from 137.18 (Mb*s) to 10.14 (Mb*s). We hope our experiments can address your concern.

| Steps   | ET (s) | NET   | MU (Mb) | NMU  | TMU (Mb*s) | NTMU |
|---------|--------|-------|---------|------|------------|------|
| Optimal ET |        |       |         |      |            |      |
| Initial | 0.97   | 7.55  | 59.26   | 1.75 | 153.39     | 27.01|
| SOAP | 0.23   | 1.59  | 29.74   | 1.03 | 13.71      | 1.59 |
| Optimal MU |        |       |         |      |            |      |
| Initial | 1.33   | 10.43 | 59.16   | 1.75 | 169.51     | 29.85|
| SOAP | 0.74   | 4.91  | 21.13   | 0.83 | 21.22      | 1.93 |
| Optimal TMU |        |       |         |      |            |      |
| Initial | 1.38   | 10.95  | 59.25   | 1.75 | 137.18     | 26.13|
| SOAP | 0.47   | 3.41  | 31.17   | 1.16 | 10.14      | 1.37 |
I appreciate your submission, detailed reviewer reviews, and your extensive rebuttal.

One question that I did not see discussed yet that might be important for readers and reviewers:
In Table 1, large percentage improvement is mostly achieved for very inefficient code especially for memory metrics. (StarCoder2-15B  ET is impressive exception).

You have a number of code solutions of various efficiency. It would be interesting to see the performance of SOAP for the code (not necessarily generated by the same LLM) that ranges from bad to "optimal" (the best available) solution.

Since the above would require a lot of experiments (N code variants x M LLMs), an alternative would be to provide ratio of best code metric after SOAP (irrespective of LLM) to best code before SOAP (irrespective of LLM). E.g. for ET in Table 1, this would be 0.12 / 0.29. This is cruder metric, since there is no guarantee that 0.12 can be achieved by running SOAP on 0.29 code, but it gives some measure of how much SOAP can improve good code. For example, for MU this metric shows very low improvement of 24.28 / 24.31.
Dear Submission5230 reviewers,

We appreciate your reviews as we move into the Author-Reviewer discussions phase (Aug 7 - Aug 13).
Please read all reviews and author responses carefully. 
Please address any remaining questions or concerns with the authors and respond to their rebuttal as needed. Authors need time to respond to your messages, so please post your responses as soon as possible, so there is time for back-and-forth discussion with the authors. At a minimum, please acknowledge that you have read the author rebuttal. Based on the material provided, please adjust your scores and reviews as necessary.

Dear Submission5230 authors,

We appreciate your rebuttal. Please continue the dialogue with the reviewers during this discussion phase.

This message and thread are visible to both reviewers and authors. If you need to respond privately, adjust the "Readers" setting accordingly.