More Than Just Functional: LLM-as-a-Critique for Efficient Code Generation
Authors: Derui Zhu
Dingfan Chen
jinfu chen
Jens Grossklags
Walter Pretschner
Weiyi Shang
Venue: NeurIPS 2025 poster ()
Keywords: Code Generation
Performance
Software Engineering
AI4SE
Primary area: other

Abstract: Large language models (LLMs) have demonstrated remarkable progress in generating functional code, leading to numerous AI-based coding program tools. However, their reliance on the perplexity objective during both training and inference primarily emphasizes functionality, often at the expense of efficiency—an essential consideration for real-world coding tasks. Perhaps interestingly, we observed that well-trained LLMs inherently possess knowledge about code efficiency, but this potential remains underutilized with standard decoding approaches. To address this, we design strategic prompts to activate the model’s embedded efficiency understanding, effectively using LLMs as \textit{efficiency critiques} to guide code generation toward higher efficiency without sacrificing—and sometimes even improving—functionality, all without the need for costly real code execution. Extensive experiments on benchmark datasets (EffiBench, HumanEval+) across multiple representative code models demonstrate up to a 70.6\% reduction in average execution time and a 13.6\% decrease in maximum memory usage, highlighting the computational efficiency and practicality of our approach compared to existing alternatives.

More Than Just Functional:
LLM-as-a-Critique for Efficient Code Generation
Derui Zhu
Dingfan Chen
Jinfu Chen
Jens Grossklags
Alexander Pretschner Weiyi Shang'
Techhicaa University of Munich
Max Planck Institute for Intelligent Systems
Wuhan University
*University of Waterloo
Abstract
Large language models (LLMs) have demonstrated remarkable progress in generat- ing functional code, leading to numerous Al-based coding assistant tools. However, their reliance on the perplexity objective during both training and inference primar- ily emphasizes functionality, often at the expense of efficiency- essential con- sideration for real-world coding tasks. Interestingly, we observed that well-trained LLMs inherently possess knowledge about code efficiency, but this potential re- mains underutilized with standard decoding approaches. To address this, we design strategic prompts to activate the model's embedded efficiency understanding, effec- tively using L.MSs as efficiency critiques to guide code generation toward higher efficiency without sacrificing- and sometimes even improving-fuuctioone functionality, all without the need for costly real code execution. Extensive experiments on benchmark datasets (EffiBench. HumanEval+, COFFE, Mercury) across multiple representative code models demonstrate up to 70.6% reduction in average exe- cution time and a 13.6% decrease in maximum memory usage, highlighting the computational efficiency and practicality of our approach compared to existing alternatives.
Introduction
Over recent years, large language models (LLMs) have made significant advances in understanding and generating programming code. These models have enabled wide range of practical applications, including code generation from natural language descriptions, code completion to assist in real-time development, code translation between programming languages, and code analysis and debugging for identifying and resolving issues in software. Numerous models, such as GPT-4 [1], Claude-3 [2], StarCoder [28, 34), CodeLlama [19], DeepSeck- -Ceder [20], and Opencoder [24], have showcased strong capabilities in code understanding. These models have become integral components of popular integrated development environments (IDEs), significantly enhancing developer productivity by providing intelligent, context-award code suggestions. Most existing research and development efforts have predominantly focused on ensuring the correctness of the generated code, often emphasizing functional accuracy (ensuring the generated code meets the intended behavior) and syntactical validity (ensuring adherence to language-specifie syntax rules) [21, 27, 40, 31, 32, 9].
In contrast, efficiency- crucial yet underexplored factor-plays plays pivotal role in determining the practicality and sustainability of generated code, as inefficiency can lead to significantly higher computational costs, increased latency, elevated energy consumption, and failures to meet the demands of real-world software development. However, standard LLM training and inference processes lack explicit supervision signals for efficiency, often resulting in generated code that,
39th Conference on Neural Information Processing Systems (NeurIPS 2025).
Equal contribution
Corresponding author

---

while functionally correct, lags in execution time and memory usage compared to human- solutions [43, 37, 22]. For instance, recent results from the EffiBench leaderboard [23] reveal that, on average, L.L.-generaa
code requires 2.59-3.44 times the execution time of human-writted solutions, with worst-case execution times up to -68 times longer. These findings underscore the urgent need for approaches that prioritize efficiency alongside functionality in L.LM-generated code.
To fill this gap, natural solution is to explicitly incorporate knowledge of code execution time and memory complexity into LLM generation and training process. Existing methods include leveraging execution overhead profiles for refinement during inference [22, 9, 38] or fine-tuning models with datasets enriched by aefformance-improve
edits from human programmers [44]. While these techniques have shown promise, they come with inherent limitations: generating execution overhead profiles requires running code in controlled environments, which can be resource-intensiig and obtaining sufficient human-curated data for fine-tuning is challenging.
In response, we propose enhancing the default inference process of gode-genneating. LLMs by explicitly incorporating awareness of code efficiency. Specifically, we introduce strategically prompted secondary LLM as an efficiency critique, which evaluates the static structure of the abstract syntax tree (AST) of generated code snippets and assigns efficiency-based scores guide the generation towards producing more efficient outcomes. Unlike existing methods, our approach eliminates the need for additional efficiency-annen annottaedd datasets, execution environments, or actual code execution. This makes our solution highly practical, lightweight, and easy to implement.
Despite its simplicity, our proposed method demonstrates significant improvements over existing state- of-the-art approaches. Through extensive experiments on standard code generation benchmarks [23, 7], leveraging various representative language models and diverse metrics, we comprehensively evaluate time and memory efficiency from multiple perspectives. For example, our method achieves an average execution speedup of approximately 17x and -19% reduction in normalized execution time compared to existing state-of-the-ar results [22].
2 Related Work
LLMs for Code Generation & Refactoring. The increasing availability of open-source code repositories and advances in training techniques for language models have fueled the rapid growth of code-generating LLMs. Progress in this field spans from models designed specifically for coding tasks. such as AlphaCode [29]. CodeGen [36], StarCoder [28, 34], CodeT5 [49], Opencoder [24], DeepSeek-Code -ooder [20]. and Qwen2.5-
[25], to general-purpose foundation models with code understanding capabilities, such as GPT-4 [1] and Claude-3 [2]. These LLMs have supported various code-related applications, including code generation from description, program repair, automated testing, code translation, type inference, and code summarization. Among these tasks, code generation from natural- natural-language description has emerged as key area for evaluating these models. Although LLMs have achieved notable success on benchmarks such as HumanEval [7] and MBPP [3], their efficiency in terms of execution time and memory usage has received comparatively less attention. Recent studies [43, 37, 22] have revealed that L.MM-generatdd code often lags behind written solutions in efficiency, underlining the need for further development. To address this, our work presents a practical approach that significantly enhances the efficiency of LLM -LM-grrertted code, achieving superior performance and broader applicability compared to existing methods. This focus complements LLM- refactoring systems, which primarily optimize internal code quality (c.g.. readability, modularity, and structural maintainability measured by coupling, cohesion, modularity, and cyclomatic complexity [13, 11, 12]) while restricting algorithm or data-structure changes. In contrast, our setting permits functionality-presstt algorithmic changes and optimizes for measured runtime and memory usage.
Inference-time Scaling. Inference-time scaling can effectively enhance LLM performance by strategically allocating computational resources during test time [4, 42, 15, 35, 45, 52]. A variety of methods leverage this principle, which can be broadly categorized into structured reasoning, diverse candidate exploration, and iterative refinement approaches. Structured reasoning techniques. such as thoighh [50] and tree-of-thought [52, 33] prompting, guide models to articulate intermediate steps, enhancing logical reasoning and interpretability. Similarly, lookahead search [3] explores future outcomes to guide current decisions, improving planning and anticipatory capabilities. Diverse candidate exploration methods, such as best-of-N [10, 48], produce multiple independent outputs

---

and select the best one, offering simple and effective strategy for improving performance. Beam- search [16] also fits into this category, maintaining a fixed number of promising candidates each step to balance exploration and efficiency. Iterative refinement focuses on incrementally improving initial outputs, as in refinement- -hasennt--addddde approached [35, 9]. which make localized adjustments to achieve better results, particularly for tasks that require step-by-step corrections. In our work, we focus on diverse candidate exploration for its simplicity, efficiency, and alignment with our objective of maintaining a large exploration space that encompasses diverse potential solutions.
Code Efficiency & Performance Analysis. Performance analysis aims to evaluate the efficiency of the code under various conditions to ensure the code meets the specified performance requirements in real-world applications. Traditionally, performance analysis is categorized into static and dynamic approaches. Static performance analysis detects inefficiencies without executing code by analyzing its structure using techniques such as AST analysis to identify performance anti-patterns [8]. Dynamic performance analysis, on the other hand, measures actual runtime behavior by executing the code, using unit tests [41, 6] and profiling techniques [51] to assess actual runtime behavior and memory usage. Our work mainly leverages LLMs to act code performance critique, which eliminates the need for actual execution while providing more expressive and insightful analysis than conventional static methods.
3 Method
Notation. Let Se denote the target code-generating language model parameterized by 0. We consider standard code-generation task where the goal to generate a sequence of code tokens = (y1,82,.... V2..... VL) given a task description æ. The model is trained on large dataset of code and descriptions by maximizing the conditional likelihood: fe(y)x) II fo (J (<t, æ), where fo y<t. .) predicts the token probabilities given the previous ones and the task description. During standard inference, the model iteratively samples new tokens with ÿ fo(yuly<i, 2). To enable controlled diversity, the model's output logits are scaled using temperature parameter T, transitioning the standard likelihood fo a)) to a temperature-scale probability distribution
fØ (y)ly<1,2) x)
fo (/muuly...))// Ev'ev Je(r"Iyec where V is the vocabulary. Starting with the initial token y1.
the model feeds each newly sampled token Yi back into itself to generate the subsequent token 71+ continuing this process until predetermined stopping criterion is met.
3.1 Efficiency-Aware Critique for Decoding
While the standard maximum likelihood objective effectively enables models to fit the distribution of real-world program data, allowing LLMs to mimic human-written code given specific descriptions, it does not explicitly address code efficiency. Although training datasets often include abundant unsupervised or weakly paired examples supporting functional correctesss and descriptive alignment, explicit supervision for efficiency is rare. This lack of efficiency-focusee signals during training results in limited emphasis on computational or algorithmic optimization. As a result, standard code-generating L.LSs frequently underperform in producing efficient code.
To address this limitation, we propose incorporating an efficiency-guided critique into the generation process via a reward function r: U?= V' + R. This function evaluates the efficiency of a generated code segment of length from vocabulary Integrating this reward function allows the model to prioritize not just functional correctness but also computational efficiency. In the following, we define multiple reward functions, each providing a relatively accurate approximation of code segment efficiency, enabling performance evaluation without the need for time- and resource-intensive code execution.
and outputs a scalar value reflecting its efficiency.
Static AST Pattern Matching. We leverage practical performacce-mmrooeennee insights at the AST level define our reward function, explicitly identifying 12 common performance-relater pat- terns from existing software engineering literature [5, 18, 26]. These patterns include "nested loops", "redundant function calls (inside loops)", "redundant function calls (memorization)". "inefficient use of data structures", "excessive function calls in loops", "unnecessary recursion", "deeply nested conditional statements", "inefficient string concatenation". "inefficient file/database operations", "large functions", "inefficient loop terminology". and "potential syntax errors" (see Appendix B for
3

---

details). To compute the reward for a code segment yzt. we first parse the segment into an AST and apply pattern matching detect the presence of these inefficiencies, applying a predefined penalty for each matched pattern. The final score is then normalized within [0, 1] as follows,
TAST (AST(y=1) = EpEpo 6p 1 1 (EA((()s))
(I)
E, 6,
where 6p represents the penalty associated with pattern p and 1 is the indicator function that equals 1 pattern is present in the AST of the code segment, and O otherwise. This penalization mechanism encourages the model to avoid these common performance pitfalls and guides the decoding search toward more efficient code.
Efficiency Critique. While AST pattern matching reward effectively improves generated code efficiency compared to the vanilla LLM baseline (see Table 4), it is limited by its inherent reliance static feature engineering. As more scalable alternative, we propose using trained I.IMs (themselves) as efficiency critics, leveraging their embedded knowledge of code understanding. Although these models may not autonomously generate desired efficient code without supervision, evaluating the efficiency of code segments and providing critique signals during inference is comparatively simpler task that they can handle effectively. Specifically, we strategically prompt the model to output a scalar value quantifying the efficiency of given code segment. We ask the critique LLM to assess factors such as time complexity, space complexity. runtime performance, memory usage efficiency, syntax correctness, and optionally AST analysis and perplexity (see Appendix A). Formally, the reward is defined as:
Prompting LLMs
(LIM q) = formge (VS1. q)
(2)
where foainue denotes the critique LLM, which directly outputs the scalar reward FLLM, and q represents the fffciency-focused prompt.
General Formulation. Building upon our consideration of various possible reward functions, we define a general formulation that enhances expressiveness through a linear combination of reward signals:
r(ysi. q) a FAST(( + B. FLLM(ySi q) PP(ygi)fo)
where Pa( denotes the perplexity (exponentiated average negative log-likelihood. of code sequence Vşi given the model fe. and a; 3, y hyperparameters that modulate the contribution of different reward components. Notably, this formulation flexibly integrates diverse evaluative criteria, enabling the incorporation of future insights into code efficiency for further improvement.
3.2 Diverse Candidate Exploration
For decoding search, we adopt widely used inference-time -imee scaling methods that prioritize diverse candidate exploration. This aligns with our objective of generating code that is not only probable also efficient, as discovering the most optimal solution requires navigating a sufficiently large output space that might otherwise be restricted or biased by standard perplexity-driven objective. Specifically, we perform perplexity- based ancestral sampling the token level until encountering line break symbol (e.g., \n, \r), which marks the end of a code statement. We then evaluate the reward of each segment at the statement level and apply beam search (with beam width >1) and greedy search (i.e., beam search with beam width =1) to expand the search space, following common practice [10, 47, 30, 46, 52]. Finally, we repeat the sampling process multiple times to generate a diverse set of candidates and select the final program by choosing the highest-rewarded one from the candidate set C, formally expressed as: y² = arg ea r()(() q).
4 Experiments
4.1 Setup
Datasets, Models, and Hardware. We conduct experiments on four recent standard code bench- mark datasets: EffiBench [23], a benchmark comprising 1,000 efficiency-critical LeetCode coding problems paired with human-written written canonical solutions, filtered to 988 samples with verified correct

---

test cases; Hum+ [31]. an extension of HumanEval [7] with 164 Coman-written Python programming tasks with expanded test coverage for rigorous functional correctness evaluation; Mercury [14], dataset of 1,889 Python tasks with test case generators and difficulty annotations derived from solution runtimes; and COFFE [39], code generation benchmark with 398 and 358 problems for function-level and file-level code generation, respectively. Our evaluation includes the following recent open-source code-generating large language models (LLMs) with varying sizes and configurations: OpenCoder [24] (the OpenCoder-8b checkpoint') DeepSeekCodes [20] (i.e., the DeepSeek-6, 7b2 and DeepSeekCoder-<2-
checkpoints), StarCoder [28] (i.e., StarCoder2-15b checkpoint). To
checkpoint"), ) and Owen2.5-Codee [25] (i.e., the Qwen2.5- Swnen..-CCoder-222 Coder-32B-Instruue measure performance, we profile execution time and memory usage using Line Profiler and Memory Profiler'. The code generation experiments were conducted
LLURM-manage' computing cluster equipped with 16 NVIDIA A100 Tensor Core GPUs (80GB memory each), interconnected via NVLink 3.0 technology. Each compute node featured 512GB memory. For code performance evaluation, all measurements were conducted in isolated environments. Each test was run on a separate virtual machine instance with identical configurations to minimize system-level variability: a dedicated CPU-only node was deployed containing dual Intel Xeon ES-2695 v4 processors (36 threads total @ 2. 2.1GHz base frequency) with 512GB DDR4-2400 memory. All experiments followed deterministic computing practices with fixed random seeds to ensure reproducibility.
Metries. In line with existing benchmarks [23, 31. 22], we evaluate the efficiency and correctness of the generated code using several key metrics. Execution Time (ET) measures the total runtime of the generated code in seconds, while Normalized Execution Time (NET) represents the execution time of the generated code divided by that of the canonical briteewwitten solution. Max Memory Usage (MU) captures the peak memory consumption (in MB) during execution, with Normalized Max Memory Usage (NMU) normalizing this value against the reference solution to assess memory efficiency. Correctness evaluated as the proportion of test samples successfully passing all test cases. When computing execution time and memory usage, we exclude generated code that fails to pass test cases, ensuring that efficiency metries are not skewed by severely incorrect programs.
Baselines & Method Implementation. We compare our approach against the following base- lines, which, to the best of our knowledge, represent the current state-of-the-arts the standard "Perplexity"-basse generation of LLMs with nucleus (top-p) and best-of-n sampling and selec- tion; "Self-Debug" [9], a self-refinement approach aimed at improving code generation correctness; "EffiLcarner" [22] and "PerfCodeGen" [38], which both optimize the efficiency of generated code by incorporating runtime feedback during generation. Before evaluating correctness and performance, apply a post-processing step to extract the generated code from the model's response without modifying its content. For implementing our methods, we use the prompt shown in Appendix A.2 as default input to the critique LLM, and use the prompt in Appendix A.1 for code generation. For the search strategies, we set a default search space of 50 (n=50 for best-of-n) and p=0.95 for nucleus (top-p) sampling following common practice [36]. See supplementary materials for more details.
4.2 Comparison to Baselines
Quantitative Results. In Table (and the supplementary materials), we demonstrate that our method consistently improves both efficiency and correctness metrics across different model ar- chitectures with varying model sizes, various datasets and all metrics. Compared to default LLM decoding (i.e., "Perplexity"),
approach reduces the average execution time (ET Avg) by around 92.0%-96 96.6%, while the median execution time (ET Median) improves by at least 28.2% and up to 58.9%. Additionally, our method lowers memory usage (NMU) by 6.9% to 39.9%. Compared to the previous
methods, i.e., EffiLearne and PerfCodeGen, our approach achieves substantial reductions in resource usage. Specifically, it reduces average execution time (ET Avg) by
https://huggingte httpp/ee...ciiioteeeoooeeee htttr Instruct
2
http://cw..oeeeeeeeeeaeeeeeeeppe soceeeepppeeeeecceeeeeeeee coder-6 7. instruct https://huggingfee
https://huggingte
httpe///eeee.ppeeaeaeeeeeeeeeeee Coder- Lite- Instruct
4 h/app//:///gigaaaceeeemm http::://nggnngte....////gggdd// htt.a//g.......ee/ggggot//rrrrrr
https: /tthh/ggggg.acc..oo/////ee//eeee
5
6
https://github
7
5

---

Datasets
Models
Methods
ETiAgg))
KT(Median)
NRTIANE
NKT(Mediam)
NMU
Correctuess
Perplesity (Best-of-n)
Perplxity (Top.))
1.51
1.23
1.39
50.05%
DeepSeek-6.n 6.76
Self-Debog
EfiLeamer
0.68
0.74
1.11
102
0.71
0.89
0.72
0.09
0.94
1.07
12,60%
51.17%
Pen
0.65
1.07
0.89
0.71
1.14
56.63%
Perplesity (Best-af-n)
1.38
52.415 18
Perplexity (Top-p)
OpenCoder-68 8b
Self Debug
Efflzomeer
0.72
0.08
1.41
1.02
0.71
0.87
0.89
0.69
0.95
1.16
59.78%
12:415
PerfCodeeGen
0.64
0.87
0.69
0.71
1.01
62.88%
Ours
0.04
0,36
11.2
0.59
0.89
64.17%
Perpleiity (Bet-af-nn
0.75
1.17
0.81
0.91
1,02
41,518
(Top-p)
1:17
0.81
23.233
EfäBBncch StarChder2-150
Debog
BliLeamer
1.04
0.71
13.29
PerfCideeean
0.71
1.11
0.75
0.73
1.08
S0 9180
Ours
0.06
0.84
0.62
0.56
0.95
53.81% %
Perplesity (Best-af-n)
1.39
1.41
0.79
0.83
1.05
45.51%
Perpletiry (Top-p)
1.37
1.42
01.81
0,81
1.13
42, 42.2255
DeepsassCCoder--21166
Seff-Debog
0.74
0.81
1.04
52,5756
EffLeamer
PerfCodeeGen
0.77
Ours
0.05
0.58
0.63
0.59
0.88
57.681%
Perplexity (Best-of-n)
1.45
1.35
1.11
0.96
1.48
53.121%
Perplxiity (Top-p)
144
1.33
1.02
0.91
1.39
$0.77%
Qwen2.5-Coder-2h
BfSLeamer
Self-Dehug
0.77
0.73
0.88
0.76
0.91
0.83
1.12
1.08
64 656
13.23%
Per[CodeGen
Ours
66.42%
Perplexity (Best-af-n)
3.71
2.84
2.98
1.11
26. 14%
Perplexity (Top-p)
3.48
3.68
2.72
2.83
1.14
27.4256
DeepScck 7b
EffLenamer
Self-Debog Debug
3.74
3.86
3.54
2.91
3.09
30,01
8.80%
PerfCodeGen
Ours
0.12
2.61
2,06
2.14
0,91
34.23%
Perplexity (Best-af-n)
27 22.2
Perplexiry (Top-p)
3.69
3.77
2,85
2.04
1.13
28 81
OpenCoder-Sb
Debog
3:45
3.86
3.91
3.69
2.79
3.03
2.83
3.18
1.03
1.03
31.13%
8. 8.79%
PerfClddeGion
Perplesity (Best-of-n)
Perplexity (Top-p)
Mereury
StaCCoder2-19
Self Deff-Debgg
3.13
3:32
2.35
2.58
0.90
32.17%
EfiLzcmer
2.70
3.02
2,08
2:23
0.88
9.321
PerfChdeeien
2.63
2.70
2,01
2.08
0.84
34 64.6256
Ours
2.13
37.02%
Perplexiry (Best-af-n)
3.61
27.9856
Porpesity (Top-p)
Deepoe&Coker-22118
Self-Dehog
3.77
EfäLeamer
3:17
3:16
2.55
2.71
1.03
8.21%
Per[CodeGen
3.09
3.13
2,43
2.51
0.99
32 29.69%
Ours
0.11
2,47
1.92
1.97
0.89
35.01 5)
Perplexity (Best-af-n)
3.28
3.42
2.61
2.77
1.08
35, 33.9155
Perplesity (Top.p p)
3.33
3.45
2.53
2.67
36. 88%
Qwen2.5 Sonder--C
EfflLaamer
Debog
39.
PerfCodeGen
3.01
2.97
2.34
2.43
0.95
31.14%
Ours
0.10
3,53
1.97
2.02
0.89
43.77%
Table 1: Comparisons of the generated code efficiency across different datasets. Methods that explic-
itly optimize for code efficiency are highlighted with a shaded background across all methods is indicated in bold.
The best performance
up to 94.1% %, median execution time (ET Median) by up to 24.3%, and normalized memory usage (NMU) by up to 20.7%. These improvements are consistent across models and datasets, with average reductions over all five models on EffiBench of 92.0% (ET Avg), 13.8% (ET Median), and 9.4% (NMU) when compared to EffiLearner, and 92.2% (ET Avg). 15.3% (ET Median), and 12.9% (NMU) when compared to PerfCodeGen. More importantly, while EffiLearner struggles with correctness (achieving below 15% accuracy on many models), our method improves correctness by up 13.3%, demonstrating its ability to generate not only more efficient but also functionally correct code. Overall, our approach consistently achieves substantial gains all benchmark datasets in our experiments, surpassing both naive decoding and prior efficiency-optimizz methods, providing more effective solution for optimizing execution time, memory usage, and correctness simultaneously.
Computation Overhead of Each Method. Computation overhead during inference is critical factor for the practical deployment of Sfficiency-optimizze methods. As shown Table 2, our method introduces significantly less overhead than existing inference-time scaling approaches, with

---

Perplexity (Best-of-n)
Perplexity (Top-p)
Models
Self-Dehug
EffiLcamer
PertCodeCien
Ours
DeepSock-6.7b
3.08
2.51
4088.59
3212.49
4144.29
28.39
OpenCoder-8h
2.83
2.31
3887.27
3009.78
3987.53
26.79
StarCoder2- 15b
4.52
6910.46
5316.53
6985.29
121.32
DeepseekCodder-2211e
6.34
5,02
8177.13
6242.98
8288.58
56.78
Qwen2.5-Cuder-32h
11.41
9.18
21661.09
17145.45
22180.94
101.34
Table 2: The median processing time (in seconds) of each method the EFFIBENCH dataset.
processing times that are two to three orders of magnitude lower than those of baseline methods such Self-Debug. EffiLearner, and Perf CerfennGnn In contrast these baselines- which incur significant computational costs due to repeated and extensive execution of generated code during inference, well as the considerable engineering effort required maintain real-time execution environments- method remains lightweight and efficient, For instance, while Eff fearnerrerr and PerfCodeGen require several thousand seconds refine the generated outputs from large models like S and 5- our method completes inference in 121.32s and 101 .34s, respectively. Notably, expected that standard Perplexity- -bsedd decoding (the shaded region marked in Table 2) exhibits minimal overhead, as it does not involve any additional evaluation or optimization steps. However, among all efficiency-awant methods, our approach stands out as significantly more practical, delivering strong performance gains without incurring the substantial computational costs typically associated with code execution-basee evaluation loops.
0.5
Purplaxity (Best-of-i)
Mathoda
0.5
Perploxity ofet
Malhoda
Perpleaily (Tap-P)
Peppeccly (Top-P)
Set Deoug
0.4
Set-Debug
FMl wxnner
11 wameer
PortCodetiin
PortoodoGen
Danak
0.3
0.
0.1
0.1
0.0
0.0
Expcution Time
Exccution Time
EffiBench
Mercury
Mwthods
Perlexitty (Best ot
Perplesity (Best otn)
Parplettty (Tp-P))
Parpeedy (Top-P)
Debug
Debug
EffLeeme
EfiLavamer
0.16
ParfCode
PorfCode Gon
0.10
0.075
0.00
0.050
0.025
0.00
0.000
Eseration Time
(c) COFFE
(d) HumanEval+
Figure Execution time (ET) distribution of generated code by OpenCoder-S across EFFIBENCH. MERCURY, COFFE and HUMANEVAL+
Execution Time Distributions of Generated Code. We further analyze the execution time distri- butions of the generated code across different methods on the test tasks from various benchmarks. This provides more comprehensive view of execution efficiency beyond the average and median statistics presented Table As shown in Figure 1, our method, alongside LffiLearner and PerfCodeGen, exhibits a substantial reduction in overall execution time compared to the default (Perplexite (Perplexity-based) decoding, highlighting the benefits efficiency-aware approaches. Notably, while refinement-bas based methods like Self -Dllf--ebggg adopt similar post-hoc correction procedures to

---

EffiLearner and PerfCodeGen, they do not explicitly model efficiency an optimization objec- tive. As result, Self shows little improvement execution efficiency, underscoring the necessity of explicitly incorporating efficiency considerations into the generation process. Moreover, while default Perplexity- decoding often leads to low-efficiency code with a heavy-tailed distribution, our efficiency-aware method concentrates the majority of the generated code within shorter execution times, effectively reducing the occurrence of excessively slow outputs. This sug- gests that the improvements observed in average execution time are not merely superficial artifacts of outliers but reflect a consistent and systematic enhancement in efficiency. Furthermore, our approach demonstrates more concentrated and effective improvement compared to the Eff Efiinerrer and PerfCodeGen baseline, reinforcing its ad advantages in optimizing execution performance.
cel Fireeedaaneeeeeeeee
nass: starftoe), sarez (Ser(sre]] Plase:
car ffrrtteetardorreeeerrria nanss, nansa)s
frsure russ is the sealler
rum sersellmmml numsi)
Jentrums))> 10nimm22:
Ien2rume)
rumzi, rume rumz,, manii Iann/aama)) lan(mamz))
langth 8:
return (1um) Blngth 2 13 + Ebegghh // (3) /
runghlenghh // 21
TLLM = 0.89, TAST = P, = 9.05, ET:
1.64 x 10 4s
eussffertiiciers
Figure 4: Our generated solution.
minkighe floet! lif')
partitient
cmm[parttiiien]]
moleety "lows!' let')
pertitign was nomtjarrtitier
winkighey flaatt partittow sise sad[artttsen]]
motefoo einnight and masseftr eistigtts
it o yi 5.2 0:
return Imarimarcetts, meccettr) mistiittg (ddtiget)) /
alsa:
cef fialeciaaourtseeeeeeenii mesdi Listfine], meszi Listfing] Floeti [e, 4) (ee((uu))), (er(Hems3))
return mad (eaciefts, ascuster)
elff eante winhiight
high partitiont 1
if mumo, +)
F,
rasse Valuetrron
Iow partittons 1
[1.rte, s,mas, half (0, (x+in (/:3)
IIL = 0.81, FAST 1, 1, 1.2P ET:
(1_eze a (Laan)) W/2
6.45 x 10-
3. -haff
1,nts +
Figure 2: OpenCoder generated solution.
elaf munstg 1] rom2[]]
[_eax
+lee:
def findeecandrtteeeeeeedoo nara1: List[inc), nuns;: Levt[ind]) -a floot:
B:
Jength Jeninues3) (ee((ums)))
nes_of_left numill
elif 3
nuec appeederinatl "ar"")
nax of - mestts 13
(eid,, mid?; (total Zength // 2, (totel langth 11 // 2)
maxuet Left mectnumals 11, mm2 13) if 4 n) 5 st anceffleft
[]
Pangrtteral__nrge Teaghh)
nett[:)] (ea.. 1 [+:
ndn of rtatt mass711
siit 3 nir_of_rigtt messEs] else:
menget (1po)t((ew0115))
ate right sintrowd]1) rum))13))
reture (mergestate)) + mergedesss11 12
right)
TLLM 0.43, TAST 1. P ET: 1.52 x 10-3s.
FLLM 0.78, TAST PP 8.57, ET: 5.97 x 10
Figure3 3: Alternative candidate solution 1.
Figure 5: Alternative candidate solution 2.
4.3 Analysis Studies
Qualitative Examples. We show in Figures 2-5 a concrete example of the generated candidate solutions for the "Median of Two Sorted Arrays" problem. Specifically, illustrate the correspond- ing reward values and the final execution time alongside each code snippet. As observed, the default OpenCoder decoding prioritizes the solution with the lowest perplexity (PP), which often leads to suboptimal efficiency. In contrast, our critique-base based LLM reward (FLLM) demonstrates a strong correlation with actual execution time, providing more effective signal for efficiency evaluation. While the vanilla AST-based reward (rAST) can be useful, it may not always be informative, as seen in this case where the predefined patterns do not directly apply. To address this, we incorporate AST analysis into the LLM critique prompts, enabling a more adaptive and expressive evaluation that benefits from both structural insights and learned assessments.
Varying Critique LLM. We analyze the impact of using different critique LLMs within our framework, while keeping OpenCoder-8h fixed as the target generation model. As shown Table 3, larger critique LLMs generally yield better overall performance-achienr NMU values, along with the highest correctness score of 67.73% This result expected, as larger models typically encode more domain knowledge, allowing them to more effectively evaluate the
lower ET, NET, and
8

---

Crinque LLMs
ET(Agg)
NET/Arg)
NMU
Conrectness
Prompt Stneogies
(Mediua)
NMU
Conrectiess
DeepSesk-6.b
0.62
0.95
87.87%
Default (Agpendix A.2]
0.73
0.91
63,89%
OpenCoder-Sh
0.04
0.59
0.89
64. 17%
who AST (Appendis A.3)
0.75
0.94
63.03%
DeppeekCokas2-110
0.04
0.56
0.88
66.49%
weo perplexity (Appendix A
0.74
0.94
61.21%
Qwen2.5 3.deer-36
0.03
0.53
0.88
67.73%
wo AST and perplesity (Appendix A.5)
0.9
62.88%
Table 3:
Comparison of different critique LLMs of our method with OpenCoder-8i as the target generation LLM on EFFIBENCH.
Comparison of varying prompting Table 4:
strategies for critique LLMs on EFFIBENCH.
efficiency of generated code based deeper understanding of program structure and execution behavior. Nonetheless, even when smaller or architecturally different models are used as the critique LLM, the overall trend remains consistent: our method continues to significantly outperform all baselines across key metrics. Although absolute performance may decline slightly- pat--putttuullyy in memory-related metrics-when adopting small critique models, the relative gains over default decoding and previous approaches remain substantial. This indicates that while more powerful critique LLMs offer additional benefits, our method is not strictly dependent model size or architectural alignment, and generalizes robustly across different critique configurations.
Varying Prompting Strategies. We investigate the impact of different prompting strategies for critique LLMs in Table 4. The results indicate only minor variations in performance, with all variants achieving substantial improvements over the baselines. The rest-performing strategy our default prompt (Appendix A.2), which integrates both AST analysis and perplexity. achieving the lowest NET, lowest NMU, and highest correctness at 63.89%. Removing AST analysis (w/o AST) or perplexity (w/o perplexity) results in slightly worse performance, while omitting both (w/o AST and perplexity) leads to the most noticeable decline. These findings validate the reasonableness of our default choice, suggesting that incorporating multiple perspectives structural insights from AST analysis and likelihood estimation via perplexity-enhane Furthermore, the relatively stable results across different prompting strategies highlight the inherent expressive power of LLMs, allowing them to adaptively balance different aspects without requiring fragile or intensive hyperparameter tuning, This flexibility enables the critique LLM emphasize useful patterns while mitigating potential weaknesses individual components, making our approach more robust across various settings.
the critique LLM's effectiveness.
Reward vs. Runtime Alignment. We analyze the alignment between the LLM-derived reward used at decoding time with real execution efficiency. Concretely, for each program í we form paired observation (riLM- At'), where r "iLM is the scalar reward assigned by the critique LLM to our method's final generated code, and At, = tours thaseline is the per-sample runtime difference relative to the corresponding reference bascline (i.c., "Perplexity (best-of-n)" rather than raw times controls for program-specinn difficulty and isolates efficiency changes at- tributable
Using differences
decoding. We then compute the Pearson correlation coefficient across the set of pairs, p= cor (ri (rLLM), ), (At')) which represents the correlation between the centered, standardized vectors of {ri.mM} and [. to quantify how closely the reward signal tracks actual performance gains. Here, values closer to indicate stronger alignment, since lower At, means faster execution. Averaged over samples from multiple datasets and models, we observe strong negative correlations: -0.78 on EFFIBENCH, -0.69 on MERCURY, and -0.74 on COFFE, suggesting that higher LLM rewards are generally associated with greater reductions in execution time.
5 Discussions & Limitations
Inference-time Scaling. Our approach builds on the premise that trained LLMs already possess substantial knowledge about code efficiency and have been exposed to diverse programming patterns during pre-training. This enables them to understand code efficiency and generate diverse solutions, including highly optimized ones. This aligns with recent findings that inference-tima scaling be more effective than training- time scaling (i.e., fine-tuning), as fully utilizes the model's pre-trained knowledge without requiring costly retraining.
In the context of our task, pre-training corpora frequently contain recurring efficiency-related patterns and anti patterat that LLMs can internalize. Moreover, many online sources included in the training

---

data, such as developer forums and technical blogs, explicitly discuss runtime behavior or algorithmic complexity alongside code examples. Consequently, models may implicitly learn correlations between code structure and efficiency. While final runtime depends partly on hardware, the dominant determinants of efficiency are algorithmic and structural (which can be captured by the program's computational graph), and can be analyzed through static code representations.
Nonetheless, inference-time scaling remains bounded by the model's pre-existing knowledge and may underperform in domains where such information is scarce. In these cases, fine-tuning on curated datasets of highly optimized code could provide a complementary path adapt the model toward domain-specific efficiency requirements.
Practicality, Scalability, and Generality. Existing decoding strategies prioritize perplexity, which reflect code frequency in training data but does not correlate with execution efficiency, often leading to suboptimal results. We address this by introducing an efficiency-aware ranking mechanism, achieving substantial improvements in execution time, memory usage, and correctness, making the generated code more practical for real-world deployment. At the same time, our method improves the practicability of the generation pipeline by avoiding the computational overhead of executing code during inference, a limitation existing efficiency- in Appendix D.2.) While invoking LLM as critique introduces some cost, it offers favorable trade-off by climinating the need for direct execution and environment setup. Moreover, preliminary experiments suggest potential efficiency gains by using high-temperature perplexity-basee token generation for diversity, followed by parallelized best-of-n selection with efficiency-awan rewards, reducing inference overhead to parallel critique LLM calls fixed set of sequences while ensuring efficiency-aware optimization and practical scalability.
approaches. (See additional results
Generality across programming languages is also important aspect of practicality. In principle, our method can be extended to other languages, as high-level AST patterns are often similar and the LLM-based reward component is designed to be sanguage-agnoste given appropriate training. While some engincering effort required to adapt the AST extraction module for each target language, this does not present a fundamental obstacle.
Finally, building on its generality, our method also demonstrates practical scalability. The overall processing time grows approximately linearly with code length, which allows it to handle reasonably large code snippets without a significant increase computational cost. In more complex, multi- file coding problems, additional engineering may be needed to enable cross-file coordination and dependency tracking. Nevertheless, the approach remains feasible up to the model's maximum input capacity, indicating that its scalability is constrained primarily by the backbone model's context size rather than by the design of our generation pipeline.
6 Conclusion
In summary, our work enhances the practicability of code-generating LLMSs by improving both generated code efficiency and the generation pipeline. We introduce an efficiency-awar sampling mechanism that improves execution time, memory usage, and correctness without requiring costly code execution during inference. Additionally, our analysis of critique LLM variations and prompting strategies demonstrates the flexibility and robustness of our approach across different settings. These findings pave the way for more efficient and scalable LLM-based code generation, with potential for further optimizations and broader real-world adoption.
Acknowledgments
We thank the anonymous reviewers for their valuable comments and constructive feedback, which have significantly improved the quality of this work. This work is partly supported by the National Natural Science Foundation of China (Grant Nos. 62302347 and U2436205).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4
10

---

technical report. arXiv preprint arXiv:2303.08774, 2023.
[2]
Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07722 2021.
[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional Al: Harmlessness from Al feedback. arXiv preprint arXiv:2212.08073. 2022.
[5] Boyuan Chen, Zhen Ming Jiang, Paul Matos, and Michael Lacaria. An industrial experi- ence report on performance-aware refactoring on a database-centric web application. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 653- 664. IEEE, 2019.
[6] Jinfu Chen, Zishuo Ding, Yiming Tang. Mohammed Sayagh, Heng Li, Bram Adams, and Weiyi Shang. IoPV: On inconsistent option performance variations. In Proceedings of the 3 ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 845-857. ACM, 2023.
[7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.0033
2021.
[8] Tse-Hsun Chen, Weiyi Shang, Zhen Ming Jiang, Ahmed E Hassan, Mohamed Nasser, and Parminder Flora. Detecting performance anti-patterns for applications developed using object- relational mapping. In Proceedings of the 36th International Conference on Software Engineer- ing (ICSE), pages 1001-1012. ACM, 2014.
[9] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.144 14168. 2021.
[11] Jonathan Cordeiro, Shayan Noei, and Ying Zou. An empirical study the code refactoring capability of large language models. arXiv preprint MiKiv:2411.023222 2024.
[12] Jonathan Cordeiro, Shayan Noei, and Ying Zou, Llm-driven code refactoring: Opportunities and limitations. In 2025 IEEE/ACM Second IDE Workshop (IDE), pages 32-36. IEEE, 2025,
[13] Kayla DePalma, Izabel Miminoshvili, Chiara Henselder, Kate Moss, and Eman Abdullah AlOmar. Exploring ChatGPT's code refactoring capabilities: An empirical study. Expert Systems with Applications, 249:123602, 2024.
[14] Mingzhe Du, Luu Anh Tuan, Bin Ji, Qian Liu, and See-Kiong Ng. Mercury: A code efficiency benchmark for code large language models. Advances in Neural Information Processing Systems, 37, 2024.
[15] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning (ICML), 2024.
[16] Markus Freitag and Yaser Al-Onaizan, Beam search strategics for neural machine translation. In Thang Luong, Alexandra Birch, Graham Neubig, and Andrew Finch, editors, Proceedings of the First Workshop on Neural Machine Translation. Association for Computational Linguisties, 2017.
[17] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding. Forty-first International Conference on Machine Learning (ICML), 2024.

---

[18] Micha Gorelick and Ian Ozsvald. High Performance Python: Practical Performant Program- ming for Humans. O' O'Reilly Media. 2020.
[19] Wenhan Xiong Grattafiori, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Llama: Open foundation models for code. arXiv preprini urXiv:2308. 12950, 2023.
[20] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, YK Li, et al. DeepSeek-Codet When the large language model meets programming- The rise of code intelligence. arXiv preprint arXiv:2401.1419 2024.
[21] Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adversarial testing. In Proceedings of the ACM SIGSAC Conference on Computer and Commu- nications Security (CCS), pages 1865-1879. ACM, 2023.
[22] Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Qing Yuhao, Heming Cui, Zhijiang Guo, and Jie Zhang. EffiLeamer: Enhancing efficiency of generated code via self-optimizztion. In eight Annual Conference on Neural Information Processing Systems (NeurIPS), 2024,
[23] Dong Huang, Yuhao Qing, Weiyi Shang. Heming Cui, and M. Zhang Jie. EffiBench: Bench- marking the efficiency of automatically generated code. In Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[24] Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, et al. OpenCoder: The open cookbook for top-tier code large language models. In Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics. Association for Computational Linguisties, 2025.
[25] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-Coder technical report. arXiv preprint MXXi::440..12188 2024.
[26] Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. Understanding and detecting real-world performance bugs. ACM SIGPLAN Notices, 47(6))77-88, 2012.
[27] Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, and Baba Mamadou Camara. How secure is code generated by ChatGPT? In IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2445-2451. IEEE, 2023.
[28] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Monteiro, Oleh Shliazhko, Nicolas Gontier, et al. StarCoder: May the source be with you! Transactions on Machine Learning Research (TMLR), 2023.
Joao
[29] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser. Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 378(6624): 10922-2-71999997 2022.
[30] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[31] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Thirty-seventh Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[32] Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. No need to lift a finger anymore? Assessing the quality of code generation by ChatGPT. IEEE Transactions on Software Engineering, (066):154-1155 2024.
[33] Jieyi Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.0822. 2023.
12

---

[34] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar. Jiawei Liu, Yuxiang Wei, et al. StarCoder 2 and The Stack v2: The next generation. arXiv preprint arXiv:2402. 19173, 2024.
[35] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self- Self-Refine: Iterative refinement with selfbfeed In Thirty-seventh Annual Conference on Neural Information Processing Systems (NeurlPS), 2023.
[36] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. CodeGen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations (ICLR), 2023,
[37] Changan Niu, Ting Zhang, Chuanyi Li, Bin Luo, and Vincent Ng. On evaluating the efficiency of source code generated by LLMs. In Proceedings of the 2024 JEEE/ACM First International Conference on Al Foundation Models and Software Engineering (FORGE), pages 103-107,
ACM, 2024.
[38] Yun Peng, Akhilesh Deepak Gotmare, Caiming Xiong, Silvio Savarese, Michael Lyu, and Doyen Sahoo. Perfeodegen: Improving performance of Ilm generated code with execution feedback. In Proceedings of the 2025 IEEE/ACM Second International Conference on Al Foundation Models and Software Engineering (FORGE), pages 1-13. IEEE, 2025.
[39] Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. COFFE: A code efficiency benchmark for code generation. arXiv preprint arXiv:2502.0288 2025.
[40] Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. Do users write more insecure code with Al assistants? In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2785-2799. ACM, 2023.
[41] David Georg Reichelt, Stefan Kühne, and Wilhelm Hasselbring. PeASS: A tool for identifying performance changes at code level. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1146-1149, IEEE, 2019.
[42] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.0m 055002, 2022.
[43] Jicke Shi, Zhou Yang, and David Lo. Efficient and green large language models for software engineering: Vision and the road ahead. ACM Transactions on Software Engineering and Methodology, 34(5):1-22, 2024.
[44] Alexander G. Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R. Gardner, Yiming Yang, Milad Hashemi, Graham Ncubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improvinn code edits. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[45] Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute op- timally can be more effective than scaling model parameters. arXiv preprint 4rXiv:2408.03314. 2024.
[46] Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first International Conference Machine Learning (ICML), 2024.
[47] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-Shepherd: A label-free step-by-step verifier for LLMS in mathematical reasoning. arXiv preprint arXiv:2312.08935, 2023.
[48] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math- Sheh-Shhpphed: Verify and reinforce LLMs step-by-step without human annotations. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2024.
13

---

[49] Yue Wang, Weishi Wang. Shafiq Joty, and Steven CH Hoi. CodeT5: Identifier- aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2021.
[50] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thouge prompting elicits reasoning in large language models. Thirty-Sixth Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
[51] Lingmei Weng, Yigong Hu, Peng Huang, Jason Nich, and Junfeng Yang. Effective performance issue diagnosis with value-assisted cost profiling. In Proceedings of the Eighteenth European Conference on Computer Systems (EuroSys), pages 1-17. ACM, 2023.
[52] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Thirty- eighth Annual Conference on Neural Information Processing Systems (NeurlPS), 2024.
[53] Yao Zhao, Zhitian Xie, Chen Liang, Chenyi Zhuang, and Jinjie Gu. Lookahead: An inference acceleration framework for large language model with lossless generation accuracy. In Proceed- ings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 6344 6355. ACM, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We clearly describe our work's contribution and scope in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
We have a section to discuss the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do conduct theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide details in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the implementation at the following link: https://githie com/ hi tittum
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /ttpsscccppbblcc////eeccppcpppcp wwwubblccguuideee guwdee/ /CodeSubmisi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provide details in the Experiments section and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provide details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information in the Experiments section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttprr////eee.eeeeeeee cc cctppclcc///eteeeeceec wtwtee/ttehhhhheeeeeeieeeeeeeeee
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include discussion section in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our work does not involve misuse-related research.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the source of the open-sourced models used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 'ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We include the implementation details in the Appendix.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the detaset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments or experiments with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve the described research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification:
We investigate the applications of LLMs in code generation research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https : htt////erripppp http///uip..c.oooeeeeeeeeeeeeeee for what should or should not be described.
21

---

Appendix
These supplementary materials include the prompt templates (SA), the details of the static AST patterns ($B), the implementation details (&C), and additional results (SD).
A Prompt Templates
A.1 Generation LLM
You software engineer with Python expertise, and your task is complete the code with the given prefix. Your generated code should be the optimal in time efficiency and memory usage.
During each generation step, you need to rethink step by step whether your generation is optimal in time efficiency. You need to self-evaluate whether the generated code the optimal in time efficiency, if it is not optimal, you need to reflection and regenerate it.
There are test examples included in the prompt and you need to analyze it. The completed code needs to included in a code block. {task}
A.2 Critique LLM (default)
{code snippet: ""code"
Please rate the above code snippet based on the following performance-rllate criteria:
1. Time Complexity (Big-O Notation): Assess the time complexity of the code and assign a score reflecting its efficiency in terms of how the time complexity scales with input size. A score closer to indicates highly efficient code (e.g.. O(log n). O(n)), while score closer to indicates inefficient code (e.g.. O(n2), O(2)).
2. Space Complexity (Big-O Notation): Evaluate the space complexity of the code, con- sidering memory usage for variables and data structures. score closer to represents minimal space usage (e.g., O(1), Onn)), while a score closer to 0 reflects high memory consumption (c.g.. (n)).
3. Running Time Performance: Provide estimate of the expected running time for typical input sizes (small, medium, large). Assign a score between 0 and based on the speed execution, with being the fastest and () being the slowest.
4. Memory Usage Efficiency: Evaluate how effectively the code uses memory resources, including variable allocations and data structures. score closer to 1 indicates optimal memory usage, while a score closer to indicates inefficiency or excessive memory consumption.
5. AST Analysis for Performance: Perform an Abstract Syntax Tree (AST) analysis to assess the efficiency of the code's structure and operations. The analysis should consider factors like loop depth, redundant expressions, and operator usage. Provide performance score based on how well-optimized the AST is for execution. score closer represents an optimized AST structure, while score closer to O indicates structure with potential inefficiencies.
6. If the code contains syntax error, the final score is O.
7. The perplexity of the code snippet is as low as better.
Output: single numerical value between 0 and I that represents the overall performance score based on the above criteria. No additional text or analysis should be provided. Just the final performance score.
22

---

A.3 Critique LLM (without AST)
{code snippet: "code")
Please rate the above code snippet based on the following performance-relane relattd criteria:
1. Time Complexity (Big-O Notation): Assess the time complexity of the code and assign score reflecting its efficiency in terms of how the time complexity scales with input size. A score closer indicates highly efficient code (e.g.. O(log n.,, ((n)), while score closer to indicates inefficient code (e.g., ( O(2"))
2. Space Complexity (Big-O Notation): Evaluate the space complexity of the code, sidering memory usage for variables and data structures. score closer to represents minimal space usage (e.g., O(1), O(n)), while score closer to 0 reflects high memory consumption (e.g., O(n)).
3. Running Time Performance: Provide an estimate of the expected running time for typical imput sizes (small, medium, large). Assign a score between 0 and I based on the speed of execution, with being the fastest and 0 being the slowest.
4. Memory Usage Efficiency: Evaluate how effectively the code uses memory resources, including variable allocations and data structures. score closer to indicates optimal memory usage, while score closer to O indicates inefficiency or excessive memory consumption.
5. If the code contains syntax error, the final score is O.
6. The perplexity of the code snippet is as low as better.
Output: single numerical value between 0 and I that represents the overall performance score based on the above criteria. No additional text or analysis should be provided. Just the final performance score.
A.4 Critique LLM (without perplexity)
{code snippet: "coode")
Please rate the above code snippet based on the following performance-related criteria:
1. Time Complexity (Big-O Notation): Assess the time complexity of the code and assign score reflecting its efficiency in terms of how the time complexity scales with input size. A score closer to indicates highly efficient code (e.g.. O n.,, O(n)). while score closer to indicates inefficient code (e.g., O(n?), O(2)))
2. Space Complexity (Big-O Notation): Evaluate the space complexity of the code, sidering memory usage for variables and data structures. score closer to I represents minimal space usage (e.g., O(1), O(n)), while score closer to 0 reflects high memory consumption (e.g.. (n)).
3. Running Time Performance: Provide an estimate of the expected running time for typical input sizes (small, medium, large). Assign a score between 0 and I based on the speed of execution, with being the fastest and 0 being the slowest.
4. Memory Usage Efficiency: Evaluate how effectively the code uses memory resources, including variable allocations and data structures. score closer to 1 indicates optimal memory usage, while score closer to O indicates inefficiency or excessive memory consumption.
5. AST Analysis for Performance: Perform an Abstract Syntax Tree (AST) analysis to assess the efficiency of the code's structure and operations. The analysis should consider factors like loop depth, redundant expressions, and operator usage. Provide performance score based on how well-optimized the AST is for execution. score closer to represents an optimized AST structure, while score closer to 0 indicates structure with potential inefficiencies.
6. If the code contains syntax error, the final score is 0.
Output: A single numerical value between 0 and I that represents the overall performance score based on the above criteria. No additional text or analysis should be provided. Just the final performance score.
23

---

A.5 Critique LLM (without AST, without perplexity)
(code snippet: "code")
Please rate the above code snippet based on the following performance- relatte criteria:
1. Time Complexity (Big-O Notation): Assess the time complexity of the code and assign a score reflecting its efficiency in terms of how the time complexity scales with input size. A score closer indicates highly efficient code (e.g.. (log nz). (n)), while score closer to 0 indicates inefficient code (e.g.. O )(2")).
2. Space Complexity (Big-O Notation): Evaluate the space complexity the code, con- sidering memory usage for variables and data structures. A score closer to I represents minimal space usage (e.g., O(1), (n))), while a score closer to 0 reflects high memory consumption (e.g.. (n)).
3. Rumning Time Performance: Provide an estimate of the expected running time for typical input sizes (small, medium, large). Assign score between 0 and 1 based on the speed of execution, with being the fastest and ( being the slowest.
4. Memory Usage Efficiency: Evaluate how effectively the code memory resources, including variable allocations and data structures. score closer to indicates optimal memory usage, while score closer indicates inefficiency or excessive memory consumption.
5. If the code contains syntax emor, the final score is O.
Output: A single numerical value between 0 and I that represents the overall performance based on the above criteria. No additional text or analysis should be provided. Just the final performance score.
B Static AST Patterns
1. Nested Loops: The program detects nested loops, which are potential source of inefficiency due increased time complexity.
# I. Detect Fested Loops
nested_loopa_ppe hot._pppppppplltyy..eeoeppa 10 Larger penalty for nested loops
for node in ast valk(tree):
isinstance ast For) ininstanca (noda, ast. While): for other in ast. valk(t))))))e
Fort ast. While)) node other_noder if (sinstance ast. and (sihe mode, ast For): scora nested_loopa_penat
Lainstanca
2. Redundant Function Calls (Inside Loops): Checks for repeated calls to functions like inpensive_ffuction within loops and suggests moving them outside the loop.
Datect Redundost Pusction Catla Insáde Loops
redundant_ cotunddat__fncctionn sunction_ccall_pmmee for node in vale(tre)):
Koderste penalty for redundant calls
if isinstance (noda, Por, ast. while))
for stat in bode:body:
if isinstance(stmt, ast Expr) and isinstance (stnt, value, ast ast.Call Call): func w_w_pnneesttt... value. id isinstance (etnt value value.fune, func, Xane) else if func_ "expensive, Function": "rrpessive_uucction"..
redundant_fuuccte uut.t/effccttoolllleeeaallg
3. Redundant Function Calls (Memorization)): Flags redundant calls to functions with identical arguments and suggests memorization.
# 3. Detect Redundant Punction Calls (Nomoization Opportunitses) redundant, functton___ooooono cawle,alleeeeeeeeeee wwt.mmiieeiiioppeallllll function_ calle (
node set valk(tree);
if isinstance (node, ast. set.E)pr) and Lsinstance (node, value, ast Call):
value. func. if isinetance (node value func, Kane) if Tunc_pane - "expensive functioonn"
func_name
n
func_ function_calll caw.saallnaancccccoooo [ane]] set()
function_calle.
arge taple(ast. wanasttttuppeee.. darg) arg node args)
ir args in redundant Aaansiant_function.ccllll wtw.lleeemnoizatioooooooooe
24

---

Function_calll callo [func_pame] (arge))))
4. Inefficient Use of Data Structures: Identifies inefficient data structures, like using list for membership testing.
4. Detect Inefficient Use of Data Structures (list for nembership tect) inefficient, att.aasattaasssuurree... wtw.eerccccuueeeeallll... for node in oalk(trea):
if Viietannce(oode sst. set.Expr) and Leinstance (node. value, aet. Compare): if score
valse. Name) isinatance (node. value. comparatora [o] ,ast.
5. Excessive Function Calls in Loops: Detects function calls in loops that might be expensive and suggests optimization.
Defect Excessive Function Catta is Loops
excessive, function_ceete node valk(tres):
sucessive.function... cacection_call__ppee calle_penalty -
if isinstance (node, (ast For, set. Mhile)): stat if isinstance (stat, aet Expr) and ieinstance value, ast Call): ininstance value. func, Mane):
body:
if stat. value func, id in ("expensive function", ""expeacive_fuactioo",,,, suuctiinncalll_ppe...
6. Unnecessary Recursion: Finds unnecessary recursion and suggests efficient iterative approach.
# 6. Defect Onnecessary Recursson
for node in ast valk(trea):
isinstance :unctionnn)
if any (isinatance(n, ast. and isinstance (n. func, ast ast. Nane)
\
and funuce.id node. for in vall((e))):
score ameccssary.rrcsarsoo......
7. Deeply Nested Conditional Statements: Warns when the conditional logic is too deeply nested, which can affect readability and efficiency.
# Defect Deeply Nested Conditional Statements
deeply_ seelyy_eetee___eeeeee ceeeted__ondittaee wtt.ttiioneeeeellleeee for node in vtre(t if isinetance(uode, aet.If):
depth =
parent node
while isinstance (parent, ast.If):
depth
parent parent.parent if hasattr hasattr(parent, 'parent') else None
if depth htwply/.eattd_cooottttooo....e
acore
8. Inefficient String Concatenation: Detects inefficient string concatenation inside loops and suggests using the join() method.
Detect Inefficient String Concatesation
maefficint__ttrnn___ wtt.:cccttttttoooooaaaaoeae valeee))))
if isinstance aor):
body:
if Sitsstance(stmt. ast Expr) and isinstance value, ast act.BinDp) \ ieinstance value. op, adt:
if isinstance value. ast. and isinstanco(stntt value rstht, ast.Sr) score saefficient_etring... www.iiggccccttaattiio..eeeea
9. Inefficient File/Database Operations: Flags file and database operations inside loops, which could be optimized by batching or caching.
# 9. Defect Ine/ficient File/Dalabase Operatione
inefficient_ www.ooopraation.ppmalll.l
for nade in ast. ast. valk(tra)):
isinstance (node, aithvith):
for stat in noda. body:
Expr) and isinstance value, Call): isinstance(stmt,
func junc_nane stat.value wal.ervaaa........... func.id if isinstance (stnt, value. func, ast. Kano) else if func_pame ["open", "execute", "query"]:
score inefficient, inafficcentt.oooperatt..o 1t..ttoooorrattoo__mmmaaa..
25

---

10. Large Functions: Identifies large functions that could benefit from refactoring for clarity and performance.
10. Dotect Large Functions (Aefactoring Opportansty) farge,.unction peniion,,.ettt for node in set valk(tree);
if ieinetance ast. Functionnnn:::
function_eise len (inode.bod))
if fuaction_size 20: Arbésrery threshotd for Large Junctions
score large_ function.pemallyy
Inefficient Loop Terminology: (leng(aadddta))
Identifies inefficient
loop constructs
like
II. Detect Inefficient Loop Terminology (e.g., rasge(len(data) vs for sten ss data)
inefficient_les how.ott_oooottrrrooooooeeeeee for node in ast. valk((e))))
isinstanca ast.Por):
if isinatance (node. iter, call) and isinatance (node moodd.1tez iter func, ast, Name)
and node. itter func.id "range":
isinstance(ndde.iees iter arga [0] Call)
and isinstance itar arge[0] ast Name) and nade. iter www.i.trrrrrrraao. a0] func. "en"" score
teg.
12. Potential Syntax Errors: Identifies syntax errors incomplete code.
try:
(code
except SyntazError as
iseues (f SyataxErrer detected:(e). Code night incouplete. Analyzing partial AST.")
scoze -- 20
Penalsze sacomplete code
C Implementation Details
We employ the official open-source implementations of EffiLearner and PerfCodeGen' for evaluation. Since no official implementation available for Self-Debug we re-implemented based on the descriptions provided in the original paper. For the standard vanilla Perplexity-basee decoding, we evaluate commonly used sampling strategies, including top-p and best-of-n for fair and stable comparison. We set maximum limit of new tokens to 256 to enforce stopping criterion to by default. Furthermore, we use regular expressions
for token generation. The temperature to extract the code portion from the model's response. If multiple code implementations are contained, we only extract and test the first one.
Our method is implemented using beam search and best-of-n selection, with a default beam width b and number trials n 50. Table summarizes the default hyperparameter settings for the different configurations of our method used in the ablation study. The default configuration, corresponding the main paper results, uses the composite scoring function a a + PP with b = 1 and 50.
3
PAST 3
1, 50)
1.2
0.4
FLLM B-PP
1, 50)
1.0
0.4
PAST 3- FLLM +7.
n 50)
1.3
0.4
1..3.PP PP (b 50,
1.5
0.5
a 3- PP (6 50. - 1)
0.9
0.5
a- + LLM (6 50, n=
1.2
0.8
Table 5: Weight Configuration for Different Reward Functions.
26

---

D
Additional Results
D.1 Experiments HumanEval+ and COFFE dataset
We present the detailed quantitative results in Table 6, comparing different methods on the Hu-
MANEVAL+ and COFFE datasets. This serves as supplementary analysis to Table 1 in the main paper.
Datasets
Models
Methods
ET(Agii
ETiMedian).
NET(Agg)
NET(Mediam)
NMU.
Correctness"
Perplexity (Best-uf-n)
1.39
1.26
0.89
0.85
1.25
49.895
Perplexity (Top-p)
144
1.06
50.12%
DeepSeok-677
EtilLcamer
SelE-Debug
0.61
1.03
0.75
0.75
1.01
13.15%
Per
0.72
0.84
0.73
0.72
0.97
55.815 Ft
Ours
0.05
0.82
0.63
0.62
0.91
62.20%
Perplexity (Best-ul-n)
2.51
1.38
2,02
1.56
1.15
4K.77%
Perplesity (Top-p)
2.55
2.41
2.14
47.955%
Self Debug
CodeLlamm-7b
EtiLeamerr
1.70
1.10
1.81
1,80
1,06
15.94%
Gen
1.78
1.91
1.79
1.75
1.04
54 2916
Ours
1.54
L.99
1.71
1.68
0.95
60.37 %
Perplexity (Best-of-n)
1.33
1.11
0.83
51.72%
Perplesity (Top-p)
1.57
1.23
0.98
0.82
52.0456
OpenCoder-Sh
SelE-Debug
EfiLeamerr
0.50
0.98
0.68
0.70
0.95
11.08%
Perf Gen
0.68
0.81
0.70
0.68
0.94
58,0296
Ours
0.84
0.75
0.59
0.60
0,87
64.89%
Perplexity (Best-of
1.65
1.33
0.96
0.90
48.12%
Perplesity (Top-p)
1.50
1.34
48.3955
lamo-133
Self-Debag
EffLeamer
0.68
1.08
0.79
0.78
1,03
14,87%
Perf Gen
0.75
0.89
0.77
0.73
1.01
$4.9398
Ours
0.17
0,53
0.44
0.51
0.93
60.08 %
Perplexity (Best-af-n)
1.30
1.18
0.82
0.80
1,15
52.46%
Perplexity (Top-p)
0.95
SturCoder2-19b
Self-Dehng
Etileaamer
0.61
0.96
0.66
0.93
10.58%
17.1
PereCodleGen
0.66
0.79
0.69
0.67
0.92
5% 944
Ours
0.03
0.73
0.58
0.59
1.86
65.31 %
Perplexity (Best-aF-n)
1.21
L109
0.76
0.74
1.04
$4.51%
Perplexity (Top-p)
1.26
DeepwraCCvke--2116
Self-Dehig
EtiLeammer
0.55
0.91
0.78
0.62
0.64
0.89
9.62%.
PerCCodeGen
0.61
0.75
0.65
0.63
0.88
61 .1.24
Ours
0.82
0.67
0.51
0.56
IL82
68.47 %
Perplexity (Best-oF-n)
1.10
039
0.68
56.78
Perplesity (Top-p)
1.14
1.01
0.79
$7.25%
Qwen2 5-Colen-32b
SelE-Debug
EtiLeamer
0.89
0.85
0.58
0.60
0.84
8.21
PerlCodeGen
0.56
0.71
0.60
0.58
0.83
63 95%
Ours
0.01
0.61
0.47
0.52
0.78
69.83 %
Perplesity (Best-af-n)
2.15
2.31
1.64
1.78
12%
Perplexity (Top
41.75%
DecpSeek 6.7b
Self-Dehig
EffLeamer
2.26
192
2.05
1.49
1.53
1.10
11.34%6
PerCCodeGen
1.75
L.89
1.38
1.45
1.06
$4.561
Ours
0.18
1,69
1.21
1.30
0.91
47.90 56
Perpleuty (Best-aF-n)
2.08
1.48
11.385
Perplexity (Top-p)
1.55
43,50%
OpenCoder-Sh
Self-Dehig
1.63
Effleamer
1.83
1.96
1.38
1.40
1.08
10.9796
PernCodeGen
1,69
1.81
1.32
1.39
1.03
46 22
Ours
0.17
1.60
1.15
1.22
ILAN
48. 95
Perpisury (Best-af-n)
1.51
Perpexiit (Top-p)
COFFE
SturCoder2-15b
Self-Dehug
Efil comer
2:20
1.78
2,30
1.93
1.69
1.56
.41
1:05
PerCCodeGen
1,66
1.77
1.29
1.34
1.00
43 66.66%
Ours
0.16
1.57
1.10
1.18
1.86
46.93 %%
Ferpiciry (Best-al-n)
2.47
Perplesity (Top-p)
Dodpeekkooder-2--1
Soli-Debug
2.34
2.49
1.79
40.845
EfiLeamer
1.94
2.09
1,45
1.52
1.12
10.11%
PerCoodeGen
1,82
L.94
1.39
1.46
1.06
42. 11.11%
Ours
0.19
1.65
1,23
1.29
0.90
45.88 %
Perpesuty (B
Perplesity (Top.p)
Qwen2 5- Coder 32b
Solí-Debug
EffLeamee
2.03
2.46
2.15
2.59
1,87
1.53
1.00
1.95
1,24
1.14
10.4246
99.96%
PerCoodeGen
1.91
2.01
1.43
1.50
1.08
41.02%
Ours
0.20
1.70
1,28
1.35
0.91
44.66 %
Table 6: Comparison of the generated code efficiency on HUMANEVAL+ and COFFE. Methods that
explicitly optimize efficiency shaded; best results in bold.
27

---

D.2 Additional Results on Method Processing Time
To further support the findings presented in the main paper, we report the median processing time during the decoding phase for each method the Mercury, HumanEEal+, and COFFE datasets in Tables 7-9. These results serve as supplement to Table the main paper.
Consistent with our earlier findings, our method demonstrates a significant advantage in decoding efficiency compared to baseline methods specifically designed to optimize the generated code beyond the perplexity, e.g., Self-D EffiLearner and PerfCodeGen. These baselines achieve improved runtime performance or correctness in the generated outputs, but do so at the expense of substantially higher decoding latency in some cases, one two orders of magnitude slower than our method. In contrast, our approach achieves superior code efficiency, competitive or superior functional correctness, while maintaining fast generation speeds. It is worth noting that the vanilla baselines not explicitly optimized for code efficiency (i.e.. Perplexity) remain faster, but they do not offer the runtime benefits in the generated code approach.
Perplexity Best-of-n)
Perplexity (Top-p)
Models
Self-Debug
EfiiLeamer
Per[CodeGen
Ours
DeepSeck-6.7h OpenCoder-8h
2.89
2.22
3998.65
3555.88
3979.42
25.17
2.91
2.05
4117.52
4242.41
4884.11
23.36
StarCoder2-150
6.06
5.13
7111.11
5989.32
7373.07
153.32
DeepseekCudder-2-1666
6.27
5.84
7215.69
6004.49
7517.46
65.59
Sven2.5-Cuder-326
13.33
11.03
25543.87
18787.55
20089.51
115.58
Table 7: The median processing time (in seconds) of each method on the MERCURY dataset.
Perplexity (Best-of-n)
Perplexity (Top-p)
Models
Self-Debug
EffiLeamer
PerfCodeGen
Ours
DeepSoek-6.7h
3.18
2.45
4212.84
3763.19
4189.66
27.14
OpenCoder-8h
3.21
2.28
4328.75
4463.52
5078.94
25.36
StarCoder2- 15b
5.52
7358.99
6224.08
7613.42
159.47
DeepseckCoder-v2...
6.21
7432.77
6230.55
7735.28
69.18
5.nn2.5-Coder326
14.02
11.59
26438.73
19420.86
20833.11
121.46
Table 8: The median processing time (in seconds) of each method the HUMANEVAL.+ dataset.
Perplexity (Best-of-n)
Perplexity (Top-p)
Models
Self-Debug
PerfCodeGien EffiLeamer
Ours
DeepSeek-6.7b
2.08
3895.42
3450.31
3870.15
24.38
OpenCuder-8b
1.95
4012.67
4129.86
4775.33
22.41
StarCoder2-155
5.83
4.90
6982.76
5855.24
7232.58
149.11
DeepseekCoder--2--
6.01
5.60
7099.28
5883.20
7400.38
63.45
Owen2.5-Coder-322
Coder-32b
12.95
10.67
25032.15
18345.77
19675.28
112.76
Table 9: The median processing time (in seconds) of each method on the COFFE dataset.
D.3 Hyperparameter Tuning
As part of preliminary study, performed hyperparameter tuning determine effective configu- rations for sampling strategies in code generation.
This tuning models on the HumanEval+ dataset. The results of these experiments, detailing the average execution time (ET) of generated code under various hyperparameter settings, presented in Table 10.
performed using the CodeLlama-77 -aBaaaaa------ 7B-Instruct--HF and
D.4 Impact of Different Critique LLMSs
We present the detailed quantitative results of varying critique LLMSs Table 11. This serves as supplementary analysis to Table 3 in the main paper.
28

---

CobeLlaama----nstruu--ii
13b- 13b-mm-uc---I--tttt--------
0.5
1.61
0.21
1.55
0.19
1.3
0.4
1.54
0.17
1.5
0.5
1.49
0.19
1.5
0.7
0.3
1,56
0.18
Table 10: The ET (Avg) of the generated code on HUMANEVAL+ dataset across different settings.
Critique LLMs
ET (Avg)
ET (Median)
NET (Avg)
NET (Median)
NMU
Correctness
DeepSeek-6.76
0.04
0,92
0.66
0.63
1.42
63.89%
CodeLlama-7b
0.04
0.97
0.68
0,66
1.22
61.13%
OpenCoder-8b
0.04
0.86
0.62
0.59
0.89
64.17%
DeepseckCoder-222100
0,04
0.85
0,61
0,59
0.92
64.79%
Dwwn2.5-Coder-32h
0.04
0.83
0.60
0.58
0.88
65.02%
Table 11: More metrics on the impact of different critique LLMs of our method with OpenCoder-8b the target generation LLM ERFIBENCH.
D.5 Impact of Intermediate Per-statement Selection
We present additional results comparing two variants: one that performs selection only at the end using standard perplexity-driven token generation during intermediate decoding our default setting that applies selection after every statement ("Ours""). The results in Table 12 show that our method consistently outperforms "End-Selection', suggesting that intermediate feedback more effectively guides generation and promotes greater diversity in the outputs. We hypothesize that applying the reward at the statement level (as the default of our method) provides fine-grained control during decoding, helping the model better explore its learned code manifold and construct efficient solutions incrementally, rather than relying solely on post hoc refinement.
and
Models
Methods
ET(Avg) |
Correctness
End-Sclection
0.43
33.79%
DeepSeck-6.77
Ours
0.12
34.23%
End-Selection
1.12
31.19%
OpenCoder-8b
Ours
0.13
32.88%
End-Selection
0.78
33.15%
StarCoder2-155-innttn
Ours
0.09
37.02%
End-Selection
1.08
30.03%
DeepseekCodderv2-1666
Ours
0.11
35.01%
End-Selection
0.79
30.08%
Qwen2.5-33b
Ours
0.10
33.77%
Table 12: Comparison between End- And-Selection and Ours across different models on MERCURY.
D.6 Impact of Different Combinations of Configuration Components
We further investigate how different combinations of configuration components influence the ef- ficiency of generated code across the COFFE and HUMANEVAL+ benchmarks. Figures and present the results. Notably, while individual components vary in their contributions, combining all different rewards generally yields the most effective outcomes terms of code efficiency. Among the components, the AST reward appears contribute the least when used in isolation.
29

---

Ouu (0.-1.111 --)
Ful 110-50)
0.8
Cost + PP 10
Fasr + (b m. 50))
Fro PP -0)
FuL (b- 50)
OpenCodur-Bb
OpanCodo-BB
Models
Models
(a) Correctness
(b) (Avg)
2,00
Cuee (A 1, 50)
Ruv (b - 1,n-
Fast 11
1.2
Fast (0=1,n= 50)
Fax Dxsr PP 1, - 50)
(b 1, 50)
1.0
Fax (b=1,
(b-
NET 000
1,25
0.8
0.6
0.4
0.50
OpenCoder-bb
DeepSeek-6.R
Codellama-7b
CodaLlame-7o
Models
(c) NET (AVG)
(d) NMU
Figure 6: Comparison across different configurations on HUMANEVAL+ with OpenCoder-8B so-
(b 1, n0)
Ful 11n
0.5
Ger 11 50)
+ PP (b 1,
Fuw (b n, 50)
0.4
PP
(AVO)
0.3
0.2
OpanCCdor-Bb
DaspSeak-6.7
CosoLlama-7b
OparCoddr-Bb
DaspSeak-6.77
CodeLiame-70
(a) Correctness
(b) (Avg)
144
(b = 1, = 50)
(b 1, n
Cor PP (b 1, 50)
Taur PP (b 1
fuu 1b=11:
Fie (b=
1,0
PP
NET (NN
NMU
0.8
0.4
0.2
OpenCoder-St
DeesSeok-6.7
CodeLlama-To
OpenCoder-Sb
DecpSeak-677
CodeLlama-7b
(c) (AVG)
(d) NMU
Figure 7: Comparison across different configurations COFFE with OpenCoder-8Pi fo.
30


Reviews:
[{"review_id": "YXEWiNImq6", "reviewer": "Reviewer_jQdC", "summary": "The paper targets the challenging problem of efficient code generation by presenting a new method. The main motivation is to integrate the LLM during the decoding processing as a critique for code efficiency. Experimental results on several benchmarks like EffiBench, HumanEval+, COFFE, and Mercury, as well as different coding models validate the effectiveness and efficiency of the proposed algorithm.", "questions": "Please mainly address the questions in the weakness section, especially, the questions related with the novelty of the paper.", "limitations": "The paper addressed the limitations in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strength\n- The problem of efficient code generation is of great importance to the industry community. Thus,                        the improvement of the code generation efficiency should be repreciated. \n- The design of the proposed algorithm is simple to implement. During the decoding process, it introduces several reward functions like AST pattern matching, LLM-prompting, and perplexity as the critique to improve code efficiency. \n- Extensive experimental results have been evaluated on several benchmarks as well as different coding models. As shown in Table 1, the proposed algorithm obtains superior performance gain over the baseline. \n\n\n# Weakness\n- The novelty of the paper is limited. It introduces a set of reward functions as the critique to speed up the code efficiency. However, as it is only conducted on the decoding stage, it cannot boost the performance of the original coding model. If the coding model itself cannot generate both effecient and effective codes, the proposed algorithm will fail as well. \n- The proposed algorithm will obviously increase the decoding time of the coding model. How about a comparison with the COT/RL model like Deepseek R1 for both inference computational cost as well as the code efficiency. \n- How about the setting of the hyperparameters (\\alpha, \\beta, \\gamma) in General Formulation reward signals (there should have an equation number in the paper)? Is there any ablation study to analyse the setting of these hyperparameters?", "was_revised": true, "final_justification": "The rebuttal well addressed most of my concerns in the previous round of review. Considering the simple design of the proposed approach as well as the extensive evaluations, I would suggest to borderline accept the paper.", "rebuttal": "We sincerely thank the reviewers for their constructive feedback! We are pleased that most reviewers have a clear positive view of our submission, praising it as a **\"​​Novel Approach\"** (Reviewer RQoh), of  **\"great importance to industry community\"** (Reviewer jQdC), is **\"​​lightweight and easy to apply\"** (Reviewer 4J1w), and **\"presents a solid execution of a straightforward idea\"** (Reviewer pDaa),  while also highlighting the **\"thorough empirical study across multiple established benchmarks\"** (Reviewer RQoh). \n\nWe below address the individual concerns of **Reviewer jQdC**. All minor points will be directly incorporated into the revised manuscript.\n\n**Q1. [The novelty of the paper is limited. It introduces a set of reward functions as the critique to speed up the code efficiency. However, as it is only conducted on the decoding stage, it cannot boost the performance of the original coding model. If the coding model itself cannot generate both efficient and effective codes, the proposed algorithm will fail as well.]**  \n   \nTo our knowledge, we are the first to move away from the dominant paradigm of relying on labor- and time-intensive real execution for refining code generation, and instead propose an alternative that uses a set of lightweight, efficiency-oriented reward functions directly during decoding. \n  \nWhile it is difficult, if not impossible, to make definitive claims about the inherent capabilities of a code generation model, what we do observe is that even when a model has the capacity to produce correct and efficient code, it often fails to do so (likely due to the nature of autoregressive generation being guided by token-level perplexity rather than functional objectives). Our method effectively steers the generation process toward more efficient and reliable outputs without modifying the base model or relying on ground-truth execution. We believe this shift in methodology offers substantial practical value, particularly in scenarios where real-time execution feedback is costly or infeasible.\n\n\n**Q2. [The proposed algorithm will obviously increase the decoding time of the coding model. How about a comparison with the COT/RL model like Deepseek R1 for both inference computational cost as well as the code efficiency.]**   \n  \nWhile DeepSeek-R1 (with a model size exceeding 600GB) is beyond the hardware constraints of typical academic research environments, we conduct experiments using the distilled variant (the DeepSeek-R1-Distill-Qwen-32B) on the EffiBench dataset. As shown in the results below, although the COT/RL model improves correctness over the base Qwen2.5-Coder-32B model (~59% vs. ~53%, see Table 1 in the main paper), our method achieves further gains in both correctness and efficiency. Also, the computational cost comparison is in line with the trends and analysis reported in Table 2 of the main paper.\n\n\n| Methods                              | ET(AVG) | ET(Median) | NET(Avg) | NET(Median) | NMU  | Time Cost (Median) | Correctness  |\n|--------------------------------------|---------|------------|----------|-------------|------|--------------------|--------------|\n| Default (temperature=0.6, top-p=0.95) | 1.49    | 1.35       | 1.31     | 1.01        | 1.51 | 10.02              | 59.33%       |\n| Perplexity (Best-of-n, n=50)           | 1.51    | 1.32       | 1.22     | 1.02        | 1.53 | 12.12              | 61.22%       |\n| Perplexity (Top P)                     | 1.47    | 1.32       | 1.31     | 0.98        | 1.49 | 9.21               | 58.76%       |\n| EffiLearner                          | 0.63    | 0.68       | 0.83     | 0.73        | 1.32 | 16421.22           | 11.11%       |\n| Self-Debug                           | 0.79    | 0.91       | 0.95     | 0.84        | 1.22 | 22312.09           | 52.55%       |\n| PerfCodeGen                          | 0.68    | 0.78       | 0.89     | 0.84        | 1.01 | 23076.52           | 56.24%       |\n| Ours                                 | 0.11    | 0.64       | 0.71     | 0.63        | 0.93 | 121.84             | 64.12%       |\n\n\n**Q3. [How about the setting of the hyperparameters (\\alpha, \\beta, \\gamma) in General Formulation reward signals?]**  \n  \nThe hyperparameter values used in our main experiments are provided in Appendix C. Below, we present detailed results on hyperparameter tuning using CodeLlama-7B-Instruct on the HumanEval+ dataset. While it is challenging to draw definitive trends given the interplay of three correlated reward components, we generally observe that incorporating all terms leads to the best overall performance. Among them, slightly emphasizing the LLM-based reward ($\\beta$) over the perplexity-based term ($\\gamma$), e.g., using a weight ratio of 3:2 to 2:1 tends to yield better results. Additionally, modestly increasing the weight of the AST term ($\\alpha$) also leads to slight improvements in final performance.  Notably, a broad range of hyperparameter configurations already outperform all baselines, indicating the robustness of our approach. For the final evaluations across different datasets and models, we fix the combination $\\alpha = 1.3$, $\\beta = 1$, and $\\gamma = 0.4$, which achieved the best results in this setting.\n|  $\\alpha$ |  $\\beta$  |  $\\gamma$  | ET (Avg) of | Correctness |\n|:---:|:---:|:---:|:-----------:|:-----------:|\n|  1  |  0  |  0  |     4.89    |    38.27%   |\n|  0  |  1  |  0  |     1.68    |    50.24%   |\n|  0  |  0  |  1  |     2.78    |    45.43%   |\n|  0  |  1  |  1  |     1.67    |    52.55%   |\n| 0.5 |  1  |  1  |     1.64    |    54.49%   |\n|  1  | 0.5 |  1  |     1.72    |    52.25%   |\n|  1  |  1  | 0.5 |     1.61    |    56.63%   |\n|  1  |  1  |  1  |     1.65    |    49.95%   |\n|  1  |  1  | 1.5 |     1.71    |    47.78%   |\n|  1  | 1.5 |  1  |     1.57    |    53.38%   |\n|  1  |  2  |  1  |     1.58    |    55.55%   |\n| 1.5 |  1  |  1  |     1.63    |    51.13%   |\n| 1.4 |  1  | 0.5 |     1.59    |    57.77%   |\n| 1.4 |  1  | 0.4 |     1.62    |    55.58%   |\n| 1.3 |  1  | 0.5 |     1.55    |    59.12%   |\n| 1.3 |  1  | 0.4 |     1.54    |    60.37%   |\n| 1.3 |  1  | 0.3 |     1.57    |    59.08%   |\n| 1.3 |  1  | 0.2 |     1.62    |    58.88%   |\n| 1.3 | 0.7 | 0.4 |     1.62    |    51.21%   |"}, {"review_id": "yciYLsZ5ta", "reviewer": "Reviewer_4J1w", "summary": "Based on my understanding, this paper introduces a clever and surprisingly practical approach to improving the efficiency of code generated by large language models (LLMs). While most prior work and current coding assistants focus mainly on ensuring the code is functional and syntactically correct, this work points out a critical yet often overlooked problem: LLM-generated code can be very inefficient, with longer runtime and higher memory usage compared to human-written code. The interesting point is that LLMs actually do have implicit knowledge about what makes code efficient, but standard training objectives (like perplexity) and decoding strategies fail to activate this knowledge. So, the authors propose a simple solution, using a second LLM, strategically prompted, as an \"efficiency critic\" to guide the code generation process. This efficiency-aware critique is based on the structure of the code (like the abstract syntax tree), and it doesn't require any actual code execution or expensive annotations, which making it  lightweight and easy to apply. (However, the authors proposed \"time complexity, space complexity,\nruntime performance, memory usage efficiency, syntax correctness\" could be easily assessed by simple **speed test scripts**, I really not sure the meaning why introduce the LLM to do this.)", "questions": "The authors propose that \"time complexity, space complexity, runtime performance, memory usage efficiency, and syntax correctness\" can be assessed by simple speed test scripts. I am not convinced of the rationale behind introducing an LLM to perform this evaluation. Others see weaknesses. I am open to discussion about this key point.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nThe paper reveals that LLMs already possess implicit knowledge about code efficiency (e.g., runtime/memory complexity), even though they are typically trained for correctness. The method taps into this latent knowledge without modifying the model. Instead of retraining or fine-tuning the LLM, the method simply prompts a secondary LLM to act as an efficiency critic. \n\nWeaknesses:\n\n1. While using LLMs as critics or evaluators is an emerging trend, applying them to assess fine-grained, precision-sensitive attributes (e.g., efficiency, memory use) raises concerns. The paper assumes LLMs can act as reliable efficiency evaluators, but this assumption lacks empirical validation or calibration against ground truth measurements. Unlike prior works that rigorously validate LLM-as-judger setups, this paper does not justify the trustworthiness of its critique model with sufficient rigor.\n\n2. The proposed method hinges on the assumption that LLMs can consistently and accurately judge code efficiency without actual execution. However, inspection of examples and appendix results reveals that the judgments may be inconsistent or even contradictory across similar samples. For a critique system to guide generation, its correctness must be established—something not thoroughly validated here.\n\n3. The decoding configuration described in lines 212–214 (“best-of-n=50 with top-p=0.95 sampling”) is claimed to follow [33], but [33] in fact uses different sampling parameters. Moreover, the search space size (n=50) is unusually large and can significantly impact performance, especially under resource constraints. This discrepancy is not analyzed or ablated, making it difficult to disentangle gains from prompt-based critique versus aggressive sampling.", "was_revised": true, "final_justification": "The core assumption for this task is not convinced.", "rebuttal": "We sincerely thank the reviewers for their constructive feedback! We are pleased that most reviewers have a clear positive view of our submission, praising it as a **\"​​Novel Approach\"** (Reviewer RQoh), of  **\"great importance to industry community\"** (Reviewer jQdC), is **\"​​lightweight and easy to apply\"** (Reviewer 4J1w), and **\"presents a solid execution of a straightforward idea\"** (Reviewer pDaa),  while also highlighting the **\"thorough empirical study across multiple established benchmarks\"** (Reviewer RQoh).   \n\nWe below address the individual concerns of **Reviewer 4J1w**. All minor points will be directly incorporated into the revised manuscript.\n  \n**Q1. [While using LLMs as critics or evaluators is an emerging trend, applying them to assess fine-grained, precision-sensitive attributes (e.g., efficiency, memory use) raises concerns. The paper assumes LLMs can act as reliable efficiency evaluators, but this assumption lacks empirical validation or calibration against ground truth measurements. Unlike prior works that rigorously validate LLM-as-judger setups, this paper does not justify the trustworthiness of its critique model with sufficient rigor.]**   \n  \nFirst, we would like to clarify that our approach does **not** rely on LLMs as the final evaluators for measuring code efficiency. All evaluation results reported in the paper are based on *real code execution*, ensuring objectivity, verifiability, and immunity to potential biases inherent in LLM-as-judge evaluations. Therefore, our findings are grounded in actual runtime behavior, not subjective estimations.  \n  \nIn addition, we provide supplementary empirical evidence showing that the LLM-based reward used during decoding aligns well with real execution efficiency. Specifically, we report the average Pearson correlation coefficients between the LLM reward scores and measured execution times across samples from multiple datasets for the studied models, achieving correlations of -0.78, -0.69, and -0.74 on Effibench, Mercury, and COFFE datasets, respectively. These results indeed indicate a strong and consistent alignment between the LLM-generated reward signal and ground truth performance.\n  \n\n\n**Q2. [The proposed method hinges on the assumption that LLMs can consistently and accurately judge code efficiency without actual execution. However, inspection of examples and appendix results reveals that the judgments may be inconsistent or even contradictory across similar samples. For a critique system to guide generation, its correctness must be established—something not thoroughly validated here.]**  \n  \nWe appreciate the reviewer’s concern and would like to clarify that the examples provided in Figures 2–5 actually demonstrate a consistent alignment between the LLM reward signal and real code execution performance. Specifically, higher LLM rewards (0.89, 0.81, 0.78, 0.43) closely correspond to lower execution times (1.64E-4, 6.45E-4, 5.97E-4, 1.52E-3), indicating the LLM’s ability to reasonably infer code efficiency. The slight variance between some scores (e.g., 0.81 vs. 0.78) is expected, as the LLM reward captures a combination of both time and space complexity, which may not align perfectly with a single metric like execution time.  \n  \nTo our knowledge, Figures 2–5 are the only examples in the paper that include both LLM rewards and real execution metrics, and we are not aware of any other results that could suggest inconsistency or contradiction.  \n  \nFurthermore, as elaborated in our response to Q1, we provide supplementary empirical validation showing that the LLM reward signal indeed strongly correlates with ground-truth execution time across multiple datasets.  \n  \n**Q3. [The decoding configuration described in lines 212–214 (“best-of-n=50 with top-p=0.95 sampling”) is claimed to follow [33], but [33] in fact uses different sampling parameters. Moreover, the search space size (n=50) is unusually large and can significantly impact performance, especially under resource constraints. This discrepancy is not analyzed or ablated, making it difficult to disentangle gains from prompt-based critique versus aggressive sampling.]**  \nIn [33], the authors indeed claim to use top-p = 0.95 in Section 3 (page 4). In our experiments, we set best-of-n = 50 to ensure a fairer comparison and to isolate the impact of our proposed method from potential gains that could result simply from using a large n.   \n  \nBelow, we present results on Qwen2.5-Coder-33B using our approach under varying values of n. The results show that increasing n leads to improved generated code efficiency and accuracy, albeit with higher computational cost. Overall, we choose n = 50 as our default setting, as it provides a strong balance between generation performance and computational efficiency.  \n  \n|  n | ET(AVG) | ET(Median) | NET(Avg) | NET(Median) |  NMU | Time Cost (Median) | Correctness |\n|:--:|:-------:|:----------:|:--------:|:-----------:|:----:|:------------------:|:-----------:|\n|  1 |   1.38  |    1.34    |   1.11   |     0.94    | 1.35 |        8.79        |    55.33%   |\n|  5 |   1.23  |    1.31    |   1.24   |     0.94    | 1.27 |        17.65       |    55.58%   |\n| 10 |   0.59  |    0.98    |   1.01   |     1.18    | 0.95 |        23.33       |    57.52%   |\n| 25 |   0.08  |    0.87    |   0.82   |     1.12    | 0.95 |        59.43       |    59.91%   |\n| 50 |   0.05  |    0.61    |   0.66   |     0.58    | 0.89 |       101.34       |    66.42%   |\n| 70 |   0.05  |    0.59    |   0.65   |     0.57    | 0.91 |       185.58       |    67.26%   |\n\n\n**Q4. [The authors propose that \"time complexity, space complexity, runtime performance, memory usage efficiency, and syntax correctness\" can be assessed by simple speed test scripts. I am not convinced of the rationale behind introducing an LLM to perform this evaluation.]**  \n  \nFirst, as noted above, we do *not* use the LLM for *final evaluation*. All final assessments are performed via actual code execution, which is straightforward and typically requires only one or a few runs per sample. In contrast, the LLM is used during the *refinement stage*, which often involves **repeated** sampling and evaluation to guide the model toward generating correct and efficient code. Replacing such **repeated** ground-truth executions in this stage with a lightweight LLM-based proxy results in substantial computational savings.\n  \n  \nAdditionally, the human effort and infrastructure overhead involved in setting up execution environments are far from negligible. In this regard, our method presents a practical and scalable alternative “for free”: it enables iterative improvement toward efficient and correct code using only the **same** LLM, without the need for repeatedly executing the code. We believe this makes our approach particularly valuable in resource-constrained or rapid-development settings."}, {"review_id": "bLEVtUcdnh", "reviewer": "Reviewer_pDaa", "summary": "The paper proposes a method for better utilizing the existing capabilities of code-generating LLMs to produce more efficient code. The two (2) key ideas in this paper are to (1) introduce an efficiency-guided critique as an extension to the code-generation reward function and (2) to make that reward function a linear combination of directly measurable static AST features and LLM-generated estimates of other performance metrics, including time and space complexity, runtime performance and memory efficiency. The paper experiments with various configurations of the reward function, including both direct and LLM measurement of the AST analysis. Rewards are evaluated at the statement level and generation is organized using a beam search method. The approach has the advantages of improving generated code efficiency without requiring model retraining (or the associated augmented datasets required for retraining) and not requiring runtime evaluation (or standing up the environments needed for runtime evaluation). Since the approach does not involve retraining or code execution, it generalizes across existing codegen models. The paper claims a 17x average execution speedup vs SotA results (Effilearner, PerfCodeGen) measured using EffiBench, HumanEval+, Mercury and COFFE against exiting code generation models, including DeepSeekCoder, OpenCoder, StarCoder and Qwen-2.5-Coder.", "questions": "Questions are contained within the strengths and weaknesses section of the review.\n\nIt summary, the paper demonstrates a strong result from a straightforward idea of combining static and LLM-as-judge metrics into a reward function. As a research paper, I would like to come away with a better understanding of where the performance gains are coming from. This could take the form of any or all of (1) more quantitative information that shows the impact of various design choices (varying alpha/beta, comparison with post-generation LLM-driven refactoring), (2) some additional examples that show why the method finds performance gains that a system like Effi-Learner misses, or (3) some analyis on the accuracy of the LLM-provided estimates used to drive the reward function and if the accuracy of these estimates correlate with reported gains.\n\nThe result is solid and I would be open to improving the rating if the weaknesses can be addressed, even as an addition to the Appendix.", "limitations": "The paper includes a section of limitations. The premise that \"a trained LLM already possesses the ability to understand code efficiency\" seems like something that could have been measured by checking how accurate the various critique estimates actually are. The limittations section does not comment on likely applicablity to larger code bases or other languages.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength - The paper presents a solid execution of a straightforward idea and demonstrates how it leads to an improvement on basic coding benchmarks. Prompts and AST metrics used and other details needed to support reproducibility are provided in the appendix.\n\nStrength - The approach is practical at least in so far as it does not require additional training and does not require setting up a runtime environment or sandbox for code execution.\n\nStrength - The paper leverages existing code benchmarks and tests a reasonable selection of code generation models as described in the summary. Comparisons against recent code-efficiency improvements systems, including Effi-Learner, are good.\n\nStrength/Weakness - The related work section is adequate, but I would have expected more discussion of refactoring systems. That would seem to the biggest comparable to this work - systems that reduce the complexity of working code after the solution is fully formed, but still without requiring code execution. What was the advantage of baking code-efficiency into the reward function and applying it at the statement level?\n\nWeakness - While example outcomes are provided in Figures 2-5, it would be helpful to have more information about what is driving the improvement in the method. While the paper provides a few different prompting strategies, I did not see a breakdown of the impact of varying the alpha and beta parameters in the general formulation. Is having the secondary LLM consider AST factors redundant with the empirical AST measure? There is some description of these tradeoffs beginning on line 291, but it would be helpful to understand in more detail what the tradeoff space looks like.\n\nWeakness - The paper lacks much introspection into understanding why the method is more effective than alternatives. Why is correctness improved? Is shorter code generally more likely to be correct? What is the advantage of performing multi-dimensional analysis after every statement? How well do the individual estimates made on predicted performance compare to the actual results? While there is some commentary starting around line 250, there is only a minimal attempt at explaining why the approach achieves better results.\n\nWeakness - There is essentially no commentary on how the method might generalize to other programming languages or larger code problems.", "was_revised": true, "final_justification": "I appreciated the author's rebuttal and their addressing my feedback with additional results. I have also looked at the material provided as response to other reviewer's comments. On the basis of these clarifications and improvements I have increased my overall rating to 5.", "rebuttal": "We sincerely thank the reviewers for their constructive feedback! We are pleased that most reviewers have a clear positive view of our submission, praising it as a **\"​​Novel Approach\"** (Reviewer RQoh), of  **\"great importance to industry community\"** (Reviewer jQdC), is **\"​​lightweight and easy to apply\"** (Reviewer 4J1w), and **\"presents a solid execution of a straightforward idea\"** (Reviewer pDaa),  while also highlighting the **\"thorough empirical study across multiple established benchmarks\"** (Reviewer RQoh). \n\n\nWe below address the individual concerns of **Reviewer pDaa**. All minor points will be directly incorporated into the revised manuscript.\n\n**Q1. [The related work section is adequate, but I would have expected more discussion of refactoring systems. What was the advantage of baking code-efficiency into the reward function and applying it at the statement level?]**  \n  \nWe thank the reviewer for highlighting this point. We agree that automatic refactoring systems are worth discussing. Traditional refactoring tools (such as those in IDEs or static analyzers) aim to improve code quality or structure while preserving behavior. While recent work has explored learning-based approaches, most existing methods remain largely template-driven and rely on executability and test cases to validate behavior-preserving transformations. As a result, they often focus on stylistic or structural improvements and are less effective at producing efficiency-oriented rewrites, particularly without access to execution or profiling.  \n  \nIn contrast, our approach incorporates efficiency considerations directly into the sampling objective through an inference-time reward function. By applying this at the statement level, it enables fine-grained control over decoding, helping the model explore its learned code manifold and generate efficient solutions as part of the construction process, rather than just refactoring them post hoc.    \n    \nWe also believe that insights from refactoring systems could inform future improvements to our refinement stage, and we will revise the related work section to reflect this connection more clearly.  \n  \n**Q2. [Breakdown of the impact of varying the alpha and beta parameters in the general formulation. Is having the secondary LLM consider AST factors redundant with the empirical AST measure?]**   \n  \nWe present detailed results on hyperparameter tuning using CodeLlama-7B-Instruct on the HumanEval+ dataset. While it is challenging to draw definitive trends given the interplay of three correlated reward components, we generally observe that incorporating all terms leads to the best overall performance. Among them, slightly emphasizing the LLM-based reward ($\\beta$) over the perplexity-based term ($\\gamma$), e.g., using a weight ratio of 3:2 to 2:1 tends to yield better results. Additionally, modestly increasing the weight of the AST term ($\\alpha$) also leads to slight improvements in final performance.  \n  \nRegarding the AST term, our design leverages AST signals in two complementary ways. First, they provide knowledge guidance to the LLM reward model via prompting, as incorporating AST information helps the LLM reason about structural aspects of efficiency—something we find empirically beneficial. Second, because AST alignment is closely tied to structural correctness, where deviations may compromise functionality, it serves a role akin to a *hard* constraint. This contrasts with the LLM-based critique, which reflects a *soft*, semantic preference. This separation motivates us to modulate their respective influences via the $\\alpha$ and $\\beta$ parameters, ensuring that structural correctness is explicitly prioritized alongside more flexible semantic signals.\n\n\n|  $\\alpha$  |  $\\beta$  |  $\\gamma$  | ET (Avg) | Correctness |\n|:---:|:---:|:---:|:-----------:|:-----------:|\n|  1  |  0  |  0  |     4.89    |    38.27%   |\n|  0  |  1  |  0  |     1.68    |    50.24%   |\n|  0  |  0  |  1  |     2.78    |    45.43%   |\n|  0  |  1  |  1  |     1.67    |    52.55%   |\n| 0.5 |  1  |  1  |     1.64    |    54.49%   |\n|  1  | 0.5 |  1  |     1.72    |    52.25%   |\n|  1  |  1  | 0.5 |     1.61    |    56.63%   |\n|  1  |  1  |  1  |     1.65    |    49.95%   |\n|  1  |  1  | 1.5 |     1.71    |    47.78%   |\n|  1  | 1.5 |  1  |     1.57    |    53.38%   |\n|  1  |  2  |  1  |     1.58    |    55.55%   |\n| 1.5 |  1  |  1  |     1.63    |    51.13%   |\n| 1.4 |  1  | 0.5 |     1.59    |    57.77%   |\n| 1.4 |  1  | 0.4 |     1.62    |    55.58%   |\n| 1.3 |  1  | 0.5 |     1.55    |    59.12%   |\n| 1.3 |  1  | 0.4 |     1.54    |    60.37%   |\n| 1.3 |  1  | 0.3 |     1.57    |    59.08%   |\n| 1.3 |  1  | 0.2 |     1.62    |    58.88%   |\n| 1.3 | 0.7 | 0.4 |     1.62    |    51.21%   |\n\n\n**Q3. [Understanding why the method is more effective than alternatives. Why is correctness improved? Why the method find performance gains that a system like Effi-Learner misses?]**   \n  \nWe believe that incorporating AST patterns into the decoding-time reward plays an important role in guiding the model to be aware of both correctness and efficiency during generation. In contrast, previous refinement-based methods typically rely on passively re-feeding profiling signals and simply filtering out incorrect code that fails test cases for further refinement. However, this process lacks active guidance and can lead to instability—new errors may emerge when attempting to fix earlier ones, as seen in approaches like Self-Debug and Self-Refinement. Without an explicit feedback signal guiding the generation holistically, these methods risk oscillating between incorrect variants rather than converging on efficient and correct solutions.\n\n \n**Q4. [Is shorter code generally more likely to be correct? What is the advantage of performing multi-dimensional analysis after every statement?]**  \n  \nRegarding code length: We compute the Pearson correlation between average code length (in lines) and correctness across our experiments and find a coefficient of 0.06, indicating a negligible relationship between the two. The observed code lengths are likely an artifact of our example selection for visualization purposes, as we deliberately chose shorter examples to fit within space constraints.\n\n  \nRegarding the advantage of performing analysis *after every statement*: We present additional results comparing two variants: one that performs selection only at the end using standard perplexity-driven token generation during intermediate decoding (**End-Selection**), and our default setting that applies selection after every statement (**Ours**). The results show that our method consistently outperforms End-Selection, suggesting that intermediate feedback more effectively guides generation and promotes greater diversity in the outputs. As briefly mentioned above, we hypothesize that applying the reward at the statement level provides fine-grained control during decoding, helping the model better explore its learned code manifold and construct efficient solutions incrementally, rather than relying solely on post hoc refinement.\n\n\n\n\n|Models|Methods|ET(AVG)|Correctness|\n|:-----|:------|:------:|:----------:|\n|DeepSeek-6.7b|End-Selection|0.43|33.79%|\n||Ours|0.12|34.23%|\n|OpenCoder-8b|End-Selection|1.12|31.19%|\n||Ours|0.13|32.88%|\n|startcoder2-15b-instruct|End-Selection|0.78|33.15%|\n||Ours|0.09|37.02%|\n|deepseekcoder-v2-16B|End-Selection|1.08|30.03%|\n||Ours|0.11|35.01%|\n|qwen2.5-33b|End-Selection|0.79|30.08%|\n||Ours|0.10|33.77%|\n\n**Q5. [How well do the individual estimates made on predicted performance compare to the actual results?]**  \n  \nWe provide supplementary empirical evidence showing that the LLM-based reward used during decoding aligns well with real execution efficiency. Specifically, we report the average Pearson correlation coefficients between the LLM reward scores and measured execution times across samples from multiple datasets for the studied models, achieving correlations of -0.78, -0.69, and -0.74 on Effibench, Mercury, and COFFE datasets, respectively. These results indeed indicate a strong and consistent alignment between the LLM-generated reward signal and ground truth performance.\n\n\n**Q6. [There is essentially no commentary on how the method might generalize to other programming languages or larger code problems.]**  \n  \nWe appreciate the reviewer’s comment and will include a discussion on generalization in the revised version. To our knowledge, there is no fundamental limitation that prevents our method from extending to other programming languages. High-level AST patterns are typically identical or highly similar across languages, and the LLM-based reward component is designed to be language-agnostic and should remain effective, assuming the LLM is appropriately trained. While the AST extraction module would need to be adapted for each target language, this is a relatively straightforward engineering task. Additionally, the processing time of our method scales linearly with code length and, in principle, should not form a bottleneck for larger code problems."}, {"review_id": "L98M6OzEVu", "reviewer": "Reviewer_RQoh", "summary": "This work introduces a novel approach for generating efficient code that leverages the LLM itself as an execution-free efficiency critique.\nBy prompting LLMs to analyze code static features like algorithmic complexity and AST structure, it generates an efficiency reward signal that navigates the decoding process toward code generations that are not only functionally correct but also computationally efficient. \nExtensive experiments show that this approach significantly reduces execution time and memory usage across multiple benchmarks and models, outperforming costly execution-based refinement methods while improving functional correctness at the same time.", "questions": "- In Section Prompting LLMs as an Efficiency Critique, have you tried to gauge *runtime performance* and *memory usage efficiency* the real overhead? How well do the predicted values ​​match the actual values?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- **Novel Approach:**\nThis work tackles code efficiency, a critical yet often overlooked problem in LLM code generation. It leverages LLM itself as a static effiency critique to bypass the costly code execution stage. This approach significantly reduces the inference overhead in generating efficient code comparing with existing approaches.\n- **Comprehensive Empirical Evaluation:**\nThe authors conduct a thorough empirical study across multiple established benchmarks and a diverse set of state-of-the-art code LLMs. The proposed method consistently demonstrates a clear advantage over several efficient code generation baselines, achieving significant reductions in both execution time and memory usage.\n\n**Weaknesses**\n- **Dependence on Capable Critique LLMs:**\nThe method's performance is highly tied to the capabilities of the critique LLM as shown in Table 3. While predicting code performance without real execution is a promising direction, it remains unclear how well these predictions align with ground-truth performance metrics.\n\n- **LLM Critique Generalization:** \nCurrent evaluation is performed on self-contained and established benchmarks. It is highly probable that this data was involved in the training corpora of most LLMs. Therefore, it raises a concerns about whether the LLM's ability to critique code efficiency stems from truely code comprehension or merely from memorization of the benchmark solutions.", "was_revised": true, "final_justification": "I thank the authors for their thorough rebuttal. \n\nTheir responses have addressed my main concerns, and I have finalized my score at 4.", "rebuttal": "We sincerely thank the reviewers for their constructive feedback! We are pleased that most reviewers have a clear positive view of our submission, praising it as a **\"​​Novel Approach\"** (Reviewer RQoh), of  **\"great importance to industry community\"** (Reviewer jQdC), is **\"​​lightweight and easy to apply\"** (Reviewer 4J1w), and **\"presents a solid execution of a straightforward idea\"** (Reviewer pDaa),  while also highlighting the **\"thorough empirical study across multiple established benchmarks\"** (Reviewer RQoh). \n\nWe below address the individual concerns of **Reviewer RQoh**. All minor points will be directly incorporated into the revised manuscript.  \n\n**Q1. [Dependence on Capable Critique LLMs: The method's performance is highly tied to the capabilities of the critique LLM as shown in Table 3. While predicting code performance without real execution is a promising direction, it remains unclear how well these predictions align with ground-truth performance metrics.]**  \n\n  \nFirst, while the capabilities of the critique LLM do influence performance, we emphasize that all standard evaluation results in our paper use the **same** model for both code generation and critique. This design ensures that our improvements do not rely on access to a stronger or external model, and thus impose no additional requirements.    \n  \nWe provide supplementary empirical evidence showing that the LLM-based reward used during decoding aligns well with real execution efficiency. Specifically, we report the average Pearson correlation coefficients between the LLM reward scores and measured execution times across samples from multiple datasets for the studied models, achieving correlations of -0.78, -0.69, and -0.74 on Effibench, Mercury, and COFFE datasets, respectively. These results indicate a strong and consistent alignment between the LLM-generated reward signal and ground truth performance, supporting the reliability of our critique mechanism.\n\n\n**Q2. [LLM Critique Generalization: Current evaluation is performed on self-contained and established benchmarks. It is highly probable that this data was involved in the training corpora of most LLMs. Therefore, it raises a concerns about whether the LLM's ability to critique code efficiency stems from truly code comprehension or merely from memorization of the benchmark solutions.]**  \n  \nWe acknowledge the concern regarding potential data contamination in benchmark evaluations, which is a challenge that is nearly impossible to eliminate entirely, given that LLMs are typically trained on large-scale web-scraped corpora. However, our results suggest that even when LLMs possess the necessary knowledge to generate correct and efficient code, they often fail to do so. This is likely due to limitations in autoregressive decoding and the reliance on perplexity as the primary generation objective. In this context, our approach of steering the LLM at inference time using feedback signals (derived from the *same* model) should offer meaningful practical value by promoting generation that better aligns with both correctness and efficiency."}]

Author rebuttal:

Meta-review: This paper addresses the important but underexplored problem of generating efficient code with LLMs by introducing an execution-free critique mechanism that guides decoding toward runtime- and memory-efficient solutions. Reviewers agree on its practicality, simplicity, and broad applicability, with consistent performance gains over recent baselines. One common weaknesse raised is the reliance on the accuracy and consistency of the critique LLM. Overall, the consensus is that the paper makes a timely and useful contribution to efficient code generation, and I recommend acceptance.
Decision: Accept (poster)