Title: WEDGE: Synthesizing Performance Constraints for Evaluating and Improving Code Efficiency
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Test Generation

Code Analysis

Performance Analysis

Code Optimization

Fuzzing
Abstract: Large Language Models (LLMs) have been increasingly used to optimize code efficiency. Evaluating their effectiveness and further suggesting optimization opportunities often rely on high-quality tests to demonstrate the performance bottlenecks presented in the program. However, existing approaches rely on a limited set of hand-curated inputs or LLM-generated uninteresting length-stressing tests, failing to reveal more nuanced optimization opportunities. We present WEDGE, a framework for generating performance-stressing input given the program under test. WEDGE synthesizes explicit performance-characterizing constraints in the form of branch conditions to partition the programs’ execution space into performance-specific regions. When integrated with the coverage-guided fuzzer, reaching different regions introduces explicit rewards for test generation to explore inefficient implementations. Our evaluation shows that WEDGE introduces a significant slowdown compared to the tests in CodeContests and those claimed to be optimized by existing approaches. From the utility perspective, integrating our tests substantially improves the existing code optimization approaches that rely on test-driven execution feedback. We release PERFFORGE, the performance tests generated by WEDGE, to benchmark future approaches for efficient code generation at https://github.com/UChiSeclab/perfforge.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a two-step pipeline for generating program inputs that cause slow performance. The first step uses an LLM to generate and instrument the target program with formulas that, when evaluated as true, should result in slow program execution. The second step uses an existing fuzzer to generate inputs that satisfy the formulas. The authors empirically evaluate their pipeline and find that, on average, they can generate program inputs that lead to longer execution times (in terms of the number of CPU instructions) than those generated by EVALPERF. Even better, they show that code optimizers can use their pipeline to generate more performant code.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Generating performance constraints to guide fuzzing is a nice idea.\n- The paper is easy to read; the figures and tables are informative.\n\n## Weaknesses\n- Makes strong claims about related work without evidence to support them. For example, “However, they are known to suffer from scalability issues such as path explosion, fine-grained performance profiling overhead, and the lack of oracles to precisely capture inefficient behaviors or symptom” is a sweeping dismissal of a large and diverse body of related work that the authors present without evidence. See question 2 below.\n- Questionable baselines for the empirical evaluation. See, e.g., question 1 below.\n- Does not consider limitations of the fundamental contributions of the approach. See question 3 below.\n\nQUESTIONS:\n- What metrics or evidence did the authors use to determine that [12] is state-of-the-art in performance fuzzing?\n- Can the authors provide citations or empirical evidence to back up their claims about related work? Many performance-fuzzing papers are not cited.\n- What kind of performance bugs can the performance constraints capture? For example, can they express stateful performance issues? What other limitations are there? See, e.g., “SPIDER: Fuzzing for Stateful Performance Issues in the ONOS Software-Defined Network Controller” for an example of a test generator that does handle stateful performance issues.\n\nLIMITATIONS:\nNo. See question 3 above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework, WEDGE, for generating additional inputs for performance tests based on the performance profiles of already available inputs. The framework leverages LLM for synthesizing performance constraints from slow and fast execution profiles from already available inputs. These constraints are then integrated with a fuzzer to generate performance-stressing test inputs. Experimental results show that WEDGE generated tests lead programs to execute, on average, 84.5%\nmore CPU instructions than the SOTA approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper targets an challenging problem of evaluating performance for efficient LLM code generation. Overall, the paper is a well written and the proposed approach is easy to follow. \n\n- Leveraging LLM to synthesize performance constraints and use that as an intermediate step to generate more targeted performance-stressing input is a novel approach. Experimental results show utility of the proposed approach in improving performance of LLM generated code.\n\n\nWeakness\n\n- The improvements are measured using CPU instructions which may not directly translate to overall performance improvement of a program (for example, if the program bottleneck is memory usage). It would be interesting to how the results compare to execution time based performance numbers. \n\n- The proposed method relies on availability of slow and fast inputs but the baseline used for utility experiments (EFFIE-LEARNER and PIE) do not rely on such constraints. Although the author clearly expressed this limitation, I think this dependency significantly reduces practical utility of the proposed approach.\n\nQUESTIONS:\n1. Utility experiments with EFFI-LEARNER use top-5 slowest inputs from less than ten available tests. How does it compare to fuzzing without constraint generation?\n\n2. Given a performance constraint can be synthesized, can LLM improve the code given the performance constraints as prompt (without generating additional inputs)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents WEDGE, a novel framework that synthesizes performance-characterizing constraints to generate performance-stressing test inputs for code. By leveraging contrastive execution profiling and guiding fuzzing via LLM-inferred constraints, WEDGE exposes inefficiencies that size-based test generation methods often miss. The resulting test suite, PERFFORGE, stresses programs more effectively, leading to better optimization and evaluation outcomes when used with state-of-the-art tools like EFFI-LEARNER and PIE. The authors support their claims with robust empirical results and thoughtful ablation studies, making this work a compelling contribution to the field of LLM-based code evaluation and optimization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* One thing that I like is that WEDGE avoids naive input size scaling by synthesizing performance-characterizing constraints, like gcd(a, b) not dividing c in a nested loop problem. I feel that this enables precise targeting of inefficiencies beyond just large inputs, for instance, for the Diophantine example in Figure 2.\n* Contrastive profiling and fuzzing seems quite novel and effective.\n* Good evaluation results \n* Strong ablation study and the results justify the proposed solution.\n\nWeaknesses:\n* I don't like that each solution requires profiling, prompting and instrumentation. \n* The paper is somewhat narrowed down to a specific domain and the evaluation is focused on competitive programming-style problems.\n* The authors do not explore the generalizability to real-world systems or codebases is not explored.\n\nQUESTIONS:\n* What is the cost in time and tokens?\n* I'd suggest using an anonymous github https://anonymous.4open.science/ instead.\n* Given the iterative prompt-refine loop and profiling overhead, can you quantify the average cost (in tokens/runtime) per test case?\n* Do different constraints generated for the same problem typically yield different slow paths? Or is there convergence to a few common patterns?\n\nLIMITATIONS:\n* The paper is somewhat narrowed down to a specific domain and the evaluation is focused on competitive programming-style problems.\n* The authors do not explore the generalizability to real-world systems or codebases is not explored.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes WEDGE, a framework for generating performance-stressing tests given the program under test.

WEDGE uses a combination of LLMs and fuzzers to generate such tests. It first collects some contrasting pairs of test inputs that are similar on the surface but differs in instruction count to run. Then it utilizes LLMs to analyze the performance critical regions in the code and generate performance-characterizing constraints at the beginning of these regions. After the constraints are inserted, a fuzzer is used to conduct coverage-guided fuzzing, which needs to get through these constraints to improve coverage, thus satisfying these constraints.

Experiments show that WEDGE-generated tests can greatly slow the programs down. Also in terms of downstream tasks, WEDGE tests can be a better feedback signal for performance-improving refinements.

STRENGTHS AND WEAKNESSES:
### Strengths:

1. The combination of LLMs and fuzzing is interesting and intuitive. LLMs, with their knowledge about algorithms and data structures and their worst-case scenarios, are used to identify the performance-critical parts and generate constraints on performance-stressing program states. While fuzzers are used to actually create inputs that meet these constraints.

2. Very clear advantage over baselines and significant improvements in downstream tasks. The method could have impacts on test-time scaling of compute for coding LLMs as better signals can improve iterative refinement. It may also serve as better rewards for model training.

QUESTIONS:
1. How generalizable is this method to more complicated projects compared to single programs?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04