Title: Execution Guided Line-by-Line Code Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Program Induction

Problem Solving

Reasoning

Natural Language Processing
Abstract: We present a novel approach to neural code generation that incorporates real-time execution signals into the language model generation process. While large language models (LLMs) have demonstrated impressive code generation capabilities, they typically do not utilize execution feedback during inference, a critical signal that human programmers regularly leverage. Our method, Execution-Guided Classifier-Free Guidance EG-CFG, dynamically incorporates execution signals as the model generates code, providing line-by-line feedback that guides the generation process toward executable solutions.
EG-CFG employs a multi-stage process: first, we conduct beam search to sample candidate program completions for each line; second, we extract execution signals by executing these candidates against test cases; and finally, we incorporate these signals into the prompt during generation. By maintaining consistent signals across tokens within the same line and refreshing signals at line boundaries, our approach provides coherent guidance while preserving syntactic structure. Moreover, the method naturally supports native parallelism at the task level in which multiple agents operate in parallel, exploring diverse reasoning paths and collectively generating a broad set of candidate solutions.
Our experiments across diverse coding tasks demonstrate that EG-CFG significantly improves code generation performance compared to standard approaches, achieving state-of-the-art results across various levels of complexity, from foundational problems to challenging competitive programming and data science tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an interesting program synthesis approach called **EG-CFG**, which provides the language model with real-time, line-by-line execution feedback of current variables. During inference-time decoding, EG-CFG generates the next token by aggregating the distributions from two sources: the original prompt without inline execution feedback and a dynamic prompt enriched with inline execution feedback. The method is evaluated on a programming by examples with language instructions benchmark: the MBPP benchmark. The EG-CFG method achieves state-of-the-art accuracy performance (96.6%) on MBPP, significantly outperforming other baselines using the latest proprietary LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- **Quality:** The paper introduces a novel program synthesis approach with inline program execution feedback, mimicking how humans write and debug Python code using breakpoints. This strategy effectively increases the number of tasks successfully solved.\n- **Significance:** The proposed method achieves new state-of-the-art performance on the MBPP benchmark (96.6%), and outperforms existing approaches across both 671B and 1.3B model sizes.\n- **Clarity:** The paper is well-written and easy to follow. The method is clearly explained through both natural language and mathematical formulations, and the prompt example provides a helpful illustration.\n\n**Weakness**\n\n- **Significance:** The method is only evaluated on a single programming domain, MBPP. It would strengthen the paper to include results on more diverse and challenging datasets to demonstrate broader applicability.\n- **Quality:** Program synthesis methods often benefit from increased sampling due to the ability to verify outputs against test cases. It would be helpful to include a plot showing how the method's accuracy performance scales with greater inference-time compute (e.g., more samples).\n\nQUESTIONS:\n- How does the computational cost (e.g. API-call tokens) of the EG-CFG method compare with other baselines? If the costs differ significantly, the comparison may not be fair.\n- The MBPP dataset contains relatively simple tasks with few variables. How does the method perform when applied to problems with more variables and inputs? This could increase the complexity of inline execution feedback and potentially result in an overly long prompt.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new decoding method for LMs to perform code completion tasks. Unlike prior work, which mainly focuses on having LMs loop between generating code and understanding execution logs at a multi-turn level, this method proposes incorporating execution into the decoding procedure, with LMs being provided execution signals at intermediate points in the code generation process. The authors describe the iterative procedure to extract executable code from intermediate generations, incorporate execution into the in-progress inference output, and use CFG to allow for more dynamic sampling during the decoding process. Evaluating on MBPP, the authors show superior performance over LMs equipped with more traditional execution-feedback techniques, and the ablations back up several of the design decisions made for the decoding pipeline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* I appreciate the novelty of this work, and I think it is fundamentally quite interesting. Incorporating execution feedback into the decoding process of an LM seems quite neat, and prior works have not really explored this space at all, so it is very neat to see this idea carried out with positive results.\n* I liked going through Section 3, and I thought it was written in a very clear and presentable way, where it was very straightforward how each step flowed into the next.\n* The results are strong, indicating that this approach works well for MBPP, and more generally, code completion tasks that are written in Python. I think it would be helpful to make the ablations section a bit longer and potentially explain why performance drops can be attributed to certain omissions (e.g. no beam search, CFG), but otherwise the ablations as is very succinctly demonstrate the benefit of each step of the pipeline.\n\nWeaknesses:\n* My interpretation is that this approach works exclusively for Python code right now - is this the right understanding? I think this is because of the “Executable Extraction” step, which seems to suggest that the extraction relies heavily on the `ast` library. I’m sure this may be extensible to other languages using the `tree-sitter` library or some non-Python AST construction library, but I feel this adaptation might be non-trivial?\n* Since the testbed is mainly function completions (e.g. MBPP, HumanEval), what happens if the code completions are not so “self-contained”? I think this method is quite cool, but I’m not sure how adaptable it is to in-the-wild code completion, as it feels like there’s several criteria that must be satisfied (unit tests, executable env., Python, target completion is for a function) in order for this to work.\n* Can you do evaluations with this method on other code completion benchmarks? E.g. HumanEval, LiveCodeBench, BigCodeBench? It’s impressive this method leads to a +14% performance boost (based on Table 1), but I’m curious if this carries over for other code completion settings.\n\nQUESTIONS:\n- Line 158 - p_inst is p_0 and p_task combined? Line 170 - where does p_sol fit into this whole process? This is referring to what is eventually regarded as the solution generated by the LM, which is then evaluated for correctness?\n- Figure 2 - what is the highlighted “...” character referring to?\n- Is there a reason you didn’t compare against agent scaffolds such as SWE-agent, InterCode, CodeAct, or OpenHands? It feels like those works explore the inference-from-execution-across-turns approach that maybe interesting to compare against. At this point, a lot of the code completions benchmarks (e.g. MBPP, HumanEval) feel very saturated, as a strong model like the GPT o* series or Claude 3.7/4 could zero-shot get 90%+ on the benchmark. I feel like the performance gains could be more meaningful if evaluated on a more challenging setting.\n- What is “minimal trace” in Table 3?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method for generating solutions to the MBPP (mostly Basic Python Programs) dataset by forming incomplete candidate programs consisting of a small number of lines, then iteratively: executing them (after some minor syntactic repair to make them executable) while tracing the internal state, and prompting the model to further continue the programs by appending more lines to the end. Compared to other iterative generation approaches, which largely rely on debugging and editing already-completed programs, the approach focuses on executing unfinished programs and using the execution information to further extend them. They compare against several other methods and report state-of-the-art results on the MBPP dataset.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper reports SoTA results on a dataset that many others have attempted.\n- The method tries out an interesting combination of classifier-free guidance with iterative generation of candidate programs while providing execution feedback.\n\n## Weaknesses\n- I found the procedure in the paper a bit hard to understand, for example:\n  - Some parts (like equation 3) discuss steps as tokens during decoding; but in other parts, there is a different notion of a step which occurs due to gradual extension of canddiate solutions, which isn't as clearly described.\n  - There is an outer loop for extending solutions but it is not explicitly and clearly stated how it works, e.g. how many times it would run and what are the termination conditions.\n  - The details of the execution traces is not clearly stated.\n- The experiments are only on MBPP, which is now a few years old and superceded by other similar datasets such as LiveCodeBench. - - As I understand it, MBPP has no hidden test cases, which reduces the realism of the benchmark; the procedure outlined might have overfit to that aspect of the benchmark, considering aspects like \"we return the first correct solution identified\".\n- Because of the lack of hidden test cases, a method that tries all possible programs (given infinite time) would get 100% accuracy on MBPP. It wasn't clear to me whether the comparison to baseline methods is fair in the sense that this method might have used significantly more computation than the alternatives, or significantly more trials. Only wall-clock time with \"full parallelization\" is reported as a comparison.\n\nI would have trouble recommending the paper for acceptance until at least the clarity-related concerns are resolved, and I would suggest trying the approach on other datasets than MBPP as well.\n\n(REVISION AFTER REBUTTAL): While the clarity-related concerns are hard to evaluate, I believe the authors have significantly addressed my concerns about generalizability and inference budget.\n\nQUESTIONS:\nQuestions:\n- How was beam search performed with lines of code as the units for the length of the samples, since a fixed number of lines of code doens't translate into a fixed number of tokens?\n- In equation 10, what happens if some value of $\\gamma$ causes $\\log M_{CFG}(\\cdots)$ to become a positive number? I think this can happen unless $0 \\le \\gamma \\le 1$.\n- When does the loop end if a correct solution is not found (i.e. how long do you try before you give up)?\n- What happens to the initial solution $w^0_{pre}, \\cdots, w_\n- What is the relationship between $p_{sol}$ and $p_{pre}$?\n\nI found it hard to follow exactly the procedure used and would have appreciated 1) more examples of the prompts used with the models, 2) a diagram explaining the loop, 3) the overall procedure expressed as pseudocode.\n\nLine 168: \"last few example inside the template\" should be \"last few-shot example inside the template\"?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

comments: SUMMARY:
This paper introduces a novel code generation method called Execution-Guided Classifier-Free Guidance. The core idea is to incorporate real-time execution feedback during the inference process of a language model, mimicking how human programmers iteratively run partial code to guide development.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well-writing and easy to follow.
2. The proposed method is inference-based without introducing the additional training cost.
3. Simple, single-agent setup makes the method easy to understand and reproduce.
4. The performance for the evaluation on MBPP is good.

Weaknesses:
1. The evaluation is not diversity enough. It may be possible that the proposed method's generalization ability is not so clear. Is it possible to extend to some more challenging benchmark like LivecodeBench, and other coding tasks like DS-1000 (data science) and Bird (SQL)?
2. Relies on test cases for feedback, which may limit applicability to real-world tasks lacking such supervision. Or would the methods highly rely on the high quality and reliable test cases? At least, the paper should discuss more about corresponding impact and possible limitations.
3. Introduces significant computational overhead due to repeated execution and beam search. Would it be possible to discuss the inference time and token number compared with other baselines?

QUESTIONS:
Refer to the weaknesses please.

LIMITATIONS:
Refer to the weaknesses please.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-05